Near Real Time IoT data exploration using Azure Time Series Insights

Azure Time Series Insights (TSI) is a service in Azure which can be used to store, query and visualize time series data from IoT sensors. There are implementations where an Azure Stream Analytics job would process and then port telemetry values of one or more sensors from Azure IoT hub to a document store (something like Azure CosmosDB) for further visualization. With Azure Time Series Insights architectural implementation is simplified as Azure TSI does most of these heavy lifting for us. Azure Stream Analytics job can always be integrated with IoT Hub in case application demands any other analysis or offline reporting .You can read more details about Azure TSI here.

“Azure TSI has both GA and Preview version.The one which we would be testing is preview version”

This article is divided into three section:

  • Creating resources for Azure Time Series Insights.
  • Ingest telemetry from NodeMCU to Azure IoT hub as needed by
    Azure Time Series Insights.
  • Reading telemetry from Azure Time Series Insights using C#.

Creating resources for Azure Time Series Insights

We would first start by creating an Azure IoT Hub which would be our data source and place it in a new resource group for better maintainability.

Once our IoT Hub is created we can proceed with creating Devices in that. Let’s keep things simple for testing and select the default options.

Next, we proceed with creating a Consumer Group. Consumer groups are used by applications to pull data from Azure IoT Hub. Consumer groups enable multiple consuming applications to have a separate view of the event stream, and to read the stream independently. Lets create an additional consumer group called “tsilearningcosumergroup” so that TSI can pull data using this consumer group.

Now, let’s start creating Azure TSI resources. Doing a search for time series in All services pops us these options

Once we have our data source set we can then proceed with creating Azure Time Series Insights resources. First what we would create is Time Series Insights environment. Please select this option and give the required parameters. For the preview model lets select PAYG pricing plan.

One of the important field to choose is TimeSeriesID which would act as partition key for your data. As with all database choosing partition key is the key for any application design. I would suggest to pick up a field which would be used to quickly identify the record. In my case deviceid suited well.

Azure Time Series Insights needs a storage account. You would see later that it saves these data in a file with .parquet extension in the blob container created under this storage account.

Next is to create Azure TSI event source. Technically speaking it has more to do with mapping our environment.Here we would give a name for our event source and select the source type which is IoT hub . This can be an EventHub as well. Further, we specify that we are going with an existing IoT hub which was created before and select the hub name and set the required access policy as well.

We discussed briefly on consumer groups at the starting of this article. While creating event-source we need to select our Consumer group .

Next is to set access permission to our Azure TSI environment. By default creator of environment would be having Reader and Contributor access. By setting data access policies we give an external application access to read data from our Azure TSI environment. We would see that in detail later while we work on reading Azure TSI data from C# code.

A quick recap of all resources

Ingest telemetry from NodeMCU to Azure IoT for Azure Time Series Insights

Azure TSI has two requirements when it comes to format and required field. The data shape has to be JSON since Time Series Insights supports only JSON data.Messages should have Time Series ID property specified at the time of provisioning which in our case was deviceid.Its good to pass timestamp in message but that isn’t mandatory.

A template script which can be used with ESP8266 is uploaded here . The only change in this script compared to the previous one is to use hardcoded values for temperature and humidity instead of reading from sensors since we are more interested in checking how to send data from ESP8266 so that it can be consumed by Azure TSI. Let’s go ahead and flash our MCU.

Assuming that ArduinoIDE has all libraries needed (which are included in header) and the correct board is selected we would see messages are being sent to Azure IoT Hub.

Please note the format of message which is JSON.I normally use JSONLint to validate the format of JSON message being sent and its valid so that I don’t spent time later troubleshooting on why data is missing.

At this point we can go to azure portal and select the TSI environment that was created.Azure TSI comes with an explorer tool which can be to explore our time series data.

Screenshot above shows url of Time Series Insights explorer.If we browse this url we can see the timeseries data.Please note our DeviceId is mapped to instances here.

So we have seen how to set Azure TSI environment and how we can ingest and view the data in Azure TSI.Next we would see how can we expose timeseries data using API which TSI provides and consume the same with a C# application and define aggregation on the same.

Reading telemetry from Azure Time Series Insights using C#.

To consume telemetry data from Azure TSI first we must set up an Azure Active Directory application and assign the data access policies in the Time Series Insights environment.The steps are fairly documented here .

We would be designing an ASP.NET Core Web API which would make call to Azure TSI and get the latest data about our sensors .Lets begin by installing “Microsoft.IdentityModel.Clients.ActiveDirectory” nuget package in our project.We would also assign few values to our variables which would hold secrets and configurations.Though not the best way to maintain your secrets and configurations values but for a demo it should be fine.We need EnvironmentFQDN which we can from our newly created enviroment from Azure Portal.Please look out for Data Access FQDN in Azure TSI environment.We would need our ActiveDirectory tenant along with ApplicationId and ClientSecret of the application which was registered in Azure Active Directory.

The code is uploaded here. The controller class (ValuesController )does most of work for us.Again this doesn’t follow any strict coding guidelines and practices it just explains you the basics of TSI.We pass the Telemetry details or the metrics for which we need data in an array.In real world scenario DeviceTwins can be used for this purpose.Then we make a GET call to fetch the data with few parameters.Please pay attention to the format of DateTime variable and timezone.It should be in UTC in this format “yyyy-MM-ddTHH:mm:ssZ”.The main part here is the method where we prepare request body of our call PrepareInput .We make a JSON and returns that as JObject which is passed as request-body in our POST.Yes its a POST to get the results.

TimeSeries Insights provides multiple API’s .You can refer to the documentation here. In this particular example we are using Aggregate Series API which enables querying data captured from events by aggregating records.

So we have seen how we can query captured data using programming language of our choice.These are plain https call so we don’t have to do anything fancy here.Azure TSI provides some additional functionality via API’s where we can provide reference data to be used for our dataset and we can create some interesting data models.We would definitely cover those in future.Keep Experimenting !