In this article of Ingesting and Processing Streaming and IoT Data for Real-Time Analytics, we are going to explore how to get your IoT events captured in a data stream into a database of your choosing. Processing real-time IoT data streams with Azure Stream Analytics is a thing of beauty.

Scenario
Softclap Technologies, which is a company in the vehicle tracking and automation space, has completely automated its vehicle tracking processes. Their vehicles are equipped with sensors that are capable of emitting streams of data in real time. In this scenario, a Data Analyst Engineer wants to have real-time insights from the sensor data to look for patterns and take actions on them. You can use Stream Analytics Query Language (SAQL) over the sensor data to find interesting patterns from the incoming stream of data.

Let us look at the pre-requisites;

You can find that setup in a recent post connect an IoT Device to Azure IoT Hub.

With the above requirements in place, go ahead to follow the remainder steps to get Azure Stream Analytics to stream your IoT events to your choice Database.

What we are building today;

Step 1: Create Stream Analytics

Step 2: Create SQL Database Server

Step 3: Configure Networking to Allow Azure services and resources to access this SQL Database Server

Step 4: Create a SQL database

Step 5: Create Firewall Rules – this helps you access the Database

Step 6: Create Azure Stream Analytics to allow you to perform near real-time analytics on streaming data. Create a job right from your database.

Step 7: Select IoT Hub as Input
Stream Analytics jobs enable you Ingest streaming data into your SQL table. Set your input and output, then author your query to transform your data.

You can create a new consumer group but in this setup, I have had to use the existing consumer group $Default.

IoT Hubs limit the number of readers within one consumer group (to 5). We recommend using a separate group for each job. Leaving this field empty will use the ‘$Default’ consumer group.

Step 8: Select Output
Since you are streaming the telemetry data to your database, select the credentials used for the output table where you can query your data from later on.

The new table will automatically be created in your database after you initially start your Stream Analytics job

Now you have completed the configuration for Input and Output from the IoT Hub Telemetry to the Database Table.

Step 9: Telemetry Stream Shows Sample Events from the IoT Device

Step 10: Click Test Query

Since the objective is really to record the events in our database table, there is a need to create a table matching the schema of your test query results.

PS: Using the click to create table has not worked well for me in the past. The fields were completely out of sync. I will therefore select view create table SQL script and then connect to the database locally or from Azure Query Editor to create the tables. Let’s dive in.

Step 11: Open SQL Database Query Editor

Now that this step has completed successfully, head back to Stream Analytics and click on Start Stream Analytics Job. Starting the stream analytics job ensured that the input iot device telemetry is captured in the database predefined table which can be queried later on.

Authenticate to Database SQL Server where Output Table is stored.

Step 12: Click Start to begin writing stream data into Database table.

Back to the Query Editor and below are the results.

And so there we have it, a successful stream of IoT events from a remote IoT device sending live telemetry ingested in our stream analytics and captured in our database table.

Click here to learn more about other ways of ingesting data in Azure Stream Analytics.

IoT a matter of fact has become a common place in all spheres of human interaction. They are in our refrigerators, cars, gardens, submarines, space probes and robots, they are just everywhere and for a good reason mainly. Before we get super excited and you must be, let us start with the recommended requisites.

Clone Repository for Needed Code | This has been provided by Microsoft

git clone --recursive https://github.com/azure-rtos/getting-started.git

Prepare Your Build Environment

To install the tools:

From File Explorer, navigate to the following path in the repo and run the setup script named get-toolchain.bat:

getting-started\tools\get-toolchain.bat

After the installation, open a new console window to recognize the configuration changes made by the setup script. Use this console to complete the remaining programming tasks in the quickstart. You can use Windows CMD, PowerShell, or Git Bash for Windows.

Run the following code to confirm that CMake version 3.14 or later is installed.

cmake --version

Now that your build environment seem to be correctly setup, go through the next steps to get your local environment setup.

Install Azure IoT Explorer on Your Computer
This part requires a utility called the Azure IoT Explorer which must be installed on your computer. In this demonstration, I have installed the Windows version of the program on my operating system.

Create Azure IoT Hub
This part requires the creation of Azure IoT Hub which could be done using the CLI or Web UI. I will do this using the Web UI but provide the commands for the same in CLI. Follow along;

Successfully Created Azure IoT Hub

Add Device to Azure IoT Hub

Get Connection String

$ az iot hub connection-string  show --hub-name mxchip-device-iot-hub.azure-devices.net

Copy the connection string without the surrounding quotation characters.

{
  "connectionString": "HostName=mxchip-device-iot-hub.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=UPEgplrCL+zQyabcdefgHiJkWqEXc2vOqulTAQ1k="
}
[ ~ ]$ 

Add Hubs on IoT Explorer using the connection string

Before continuing to the next section, confirm that you’ve copied the following values:

hostName
deviceId
primaryKey

I made a note of the following elements;

HostName: mxchip-device-iot-hub.azure-devices.net
Device ID : mxchipaz366
Primary Key : KUTkSnC6Sn0vVieeabcdefghijkllU9ko0XCOwKy4= 

Configure Connection on Local Repo
Open the following file in a text editor:

getting-started\MXChip\AZ3166\app\azure_config.h

Comment out the following line near the top of the file as shown:

// #define ENABLE_DPS

Set the Wi-Fi constants to the following values from your local environment.

WIFI_SSID	{Your Wi-Fi SSID}
WIFI_PASSWORD	{Your Wi-Fi password}
WIFI_MODE	{One of the enumerated Wi-Fi mode values in the file}

Set the Azure IoT device information constants to the values that you saved after you created Azure resources.

IOT_HUB_HOSTNAME	{Your Iot hub hostName value}
IOT_DPS_REGISTRATION_ID	{Your Device ID value}
IOT_DEVICE_SAS_KEY	{Your Primary key value}

Build the image
In your console or in File Explorer, run the script rebuild.bat at the following path to build the image:

getting-started\MXChip\AZ3166\tools\rebuild.bat

After the build completes, confirm that the binary file was created in the following path:

getting-started\MXChip\AZ3166\build\app\mxchip_azure_iot.bin

Follow the steps here.

Launch Termite and check connectivity
My device is on COM5. You can check for that in Command Prompt by typing mode.

Successful Connection to Azure IoT Hub

Check Telemetry on Azure IoT Explorer

Check Telemetry on Termite

Simulate Device Telemetry

Simply copy paste the following command to Azure Cloud Shell. It will start simulating device as it’s sending messages to IoT Hub. You can click ‘Start’ button from the Telemetry page to start monitoring the events.

az iot device simulate --device-id mxchipdevkitaz3166 --login "HostName=**cloudmxchipiot-01.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=slKRd09jokVHPXNjDabcdeEfgHizDhmq8="

View Telemetry Received from IoT Device

az iot hub monitor-events --output table --device-id mxchipaz366 --hub-name mxchip-device-iot-hub

Starting event monitor, filtering on device: mxchipaz366, use ctrl-c to stop...
event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    magnetometerX: -445.5
    magnetometerY: 531
    magnetometerZ: 496.5

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    accelerometerX: -377.04
    accelerometerY: -917.31
    accelerometerZ: -130.66

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    gyroscopeX: -770
    gyroscopeY: -420
    gyroscopeZ: 770

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    humidity: 60.61
    pressure: 1014.05
    temperature: 19.88

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    magnetometerX: -408
    magnetometerY: 504
    magnetometerZ: 495

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    accelerometerX: -380.33
    accelerometerY: -915.85
    accelerometerZ: -129.93

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    gyroscopeX: -1190
    gyroscopeY: 630
    gyroscopeZ: 2800

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    humidity: 60.2
    pressure: 1014.04
    temperature: 20.2

event:
  component: ''
  interface: dtmi:azurertos:devkit:gsgmxchip;2
  module: ''
  origin: mxchipaz366
  payload:
    magnetometerX: -417
    magnetometerY: 531
    magnetometerZ: 486

Communicate with your IoT Device
Run the az iot hub invoke-device-method command, and specify the method name and payload. For this method, setting method-payload to true turns on the LED, and setting it to false turns it off.

az iot hub invoke-device-method --device-id mxchipaz366 --method-name setLedState --method-payload true --hub-name mxchip-device-iot-hub

az iot hub invoke-device-method --device-id mxchipaz366 --method-name setLedState --method-payload true --hub-name mxchip-device-iot-hub
{
  "payload": {},
  "status": 200
}

There are advanced aspects to provisioning IoT devices and the following guide helps you do just that.
Create a new IoT Hub Device Provisioning Service
https://learn.microsoft.com/en-gb/azure/iot-dps/quick-setup-auto-provision#create-a-new-iot-hub-device-provisioning-service

When you encounter this error;

"ERROR: azure_iot_nx_client_dps_entry"

then it is likely you did not comment out #define ENABLE_DPS.

So here is the deal, I had a great affinity for Arduino until I fell in love with Python. My life has never been the same again. Aaahhh, do I need to ditch my Ardunio control boards to use a Raspberry Pi instead? How about all the interesting projects I made back then in Arduino? Could I not learn to code them in Python and just use my Python scripts to control them?

Things you’ll need;

• Personal Computer Linux | MAC | Windows with Arduino IDE.
• Arduino Board (I have used the Elegoo Uno Version here)
• USB cable for Arduino.
• Breadboard
• Jumper Wires
• LED Light
• Resistor

Step 1: Upload Standard Firmata on your Arduino board (Firmata is a serial communication protocol that can control the Arduino’s GPIO)
Files > Examples > Firmata > StandardFirmata (You will only need to upload this once).

What is Firmata

Firmata is an intermediate protocol that connects an embedded system to a host computer, and the protocol channel uses a serial port by default. The Arduino platform is the standard reference implementation for Firmata. The Arduino IDE comes with the support for Firmata.

This could work perfectly with Odyssey-X86 with its onboard Arduino Core meaning that you can control the Arduino Core simply using Firmata protocol with different programming languages too!

The Firmata library implements the Firmata protocol for communicating with software on the host computer. This allows you to write custom firmware without having to create your own protocol and objects for the programming environment that you are using.

To use this library
#include <Firmata.h>

Step 2:
Install PyFirmata on Ubuntu

pip3 install pyfirmata

Step 3:

Build your circuit. Roll your sleeves and get your jumper wires, resistor and LED together in a very simple circuit like the one below;

Schematic: How to wire up your circuit

Step 4: Write Your Python Blink LED Program

from pyfirmata import Arduino
import time

if __name__ == '__main__':
    board = Arduino('/dev/ttyACM1')
    print("Communication Successfully Initiated ")

    while True:
        board.digital[13].write(1)
        time.sleep(0.5)
        board.digital[13].write(0)
        time.sleep(0.5)

Full Code: Blink LED Python Program


from pyfirmata import Arduino, util
import time

while True:
board.digital[9].write(1)
time.sleep(0.5)
board.digital[9].write(0)
time.sleep(0.5)


Step 5 Hit Run and see it in action

Troubleshooting
Your USB connection to your Arduino board is disconnected if you see this sudden error;

raise SerialException('write failed: {}'.format(e))
serial.serialutil.SerialException: write failed: [Errno 5] Input/output error

Here you are, a step by step guide to get Python to control your GPIOs on Arduino. Hope this little project helps you on your journey to great electronics?

See more python projects like this fading led light project.