This is a follow up to my previous posts about using Qt MQTT to connect to the cloud. MQTT is a prominent standard for telemetry, especially in the IoT scenario.
We are often approached by Qt customers and users on how to connect to a variety of cloud providers, preferably keeping the requirements list short.
With this post I would like to provide some more information on how to create a connection by just using Qt, without any third-party dependency. For this comparison we have chosen the following cloud providers:
The ultimate summary can be viewed in this table
The source code to locally test the results is available here.
However, if you are interested in this topic, I recommend preparing a pitcher of coffee and continue reading…
And if you are want to jump to a specific topic, use these shortcuts:
Preface
Getting connected
Standard derivation (limitations)
Available (custom) topics
Communication routes
Other / references
Additional notes
How can I test this myself?
Closing words
Preface / Setting expectations
Before getting into the details I would like to emphasize some details.
First, the focus is on getting devices connected to the cloud. Being able to send and receive messages is the prime target. This post will not talk about services, features, or costs by the cloud providers themselves once messages are in the cloud.
Furthermore, the idea is to only use Qt and/or Qt MQTT to establish a connection. Most, if not all, vendors provide SDKs for either devices or monitoring (web and native) applications. However, using these SDKs extends the amount of additional dependencies, leading to higher requirements for storage and memory.
The order in which the providers are being evaluated in this article is based on public usage according to this article.
Getting connected
The very first steps for sending messages are to create a solution for each vendor and then establish a TCP connection.
Amazon IoT Core
We assume that you have created an AWS account and an IoT Core service from your AWS console in the browser.
The dashboard of this service looks like this:
The create button open a wizard which will help setting up the first device.
The only required information is the name of the device. All other items can be left empty.
The service allows to automatically create a certificate to be used for a connection later.
Store the certificates (including the root CA) and keep them available to be used in an application.
For now, no policy is required. But we will get into this at a later stage.
The last missing piece to start implementing an example is the hostname to connect to. AWS provides a list of endpoints here. Please note, that for MQTT you must use the account-specific prefix. Also, you can find the information on the settings page of the AWS IOT dashboard.
Using Qt MQTT, a connection is then established with those few lines:
const QString host = QStringLiteral("<your-endpoint>.amazonaws.com");
const QString rootCA = QStringLiteral("root-CA.crt");
const QString local = QStringLiteral("<device>.cert.pem");
const QString key = QStringLiteral("<device>.private.key");
QMqttClient client;
client.setKeepAlive(10000);
client.setHostname(host);
client.setPort(8883);
client.setClientId("basicPubSub");
QSslConfiguration conf;
conf.setCaCertificates(QSslCertificate::fromPath(rootCA));
conf.setLocalCertificateChain(QSslCertificate::fromPath(local));
QSslKey sslkey(readKey(key), QSsl::Rsa);
conf.setPrivateKey(sslkey);
client.connectToHostEncrypted(conf);A couple of details are important for a successful connection:
- The keepalive value needs to be within a certain threshold. 10 seconds seem to be a good indicator.
- Port 8883 is the standardized port for encrypted MQTT connections.
- The ClientID must be basicPubSub. This is a valid ID auto-generated during the creation of the IoT Core instance.
Microsoft Azure IoT Hub
First, an account for the Azure Portal needs to be created. From the dashboard you need to create a new “Iot Hub” resource.
The dashboard can be overwhelming initially, as Microsoft puts many cloud services and features on the fore-front. As the focus is on getting a first device connected, the simplest way is to go to Shared access policies and create a new access policy with all rights enabled.
This is highly discouraged in a production environment for security reasons.
Selecting the freshly-created policy we can copy the connection string.
Following, we will use the Azure Device Explorer application, which can be downloaded here. This application suits perfectly for testing purposes. After launch, enter the connection string from above into the connection test edit and click update.
The management tab allows for creating new test devices, specifying either an authentication via X509 or Security Keys. Security keys are the preselected standard method, which we aim at as well.
Lastly, the Device Explorer allows us to create a SAS token, which will be needed to configure the MQTT client. A token has the following shape:
HostName=<yourIoTHub>.azure-devices.net;DeviceId=<yourDeviceName>;SharedAccessSignature=SharedAccessSignature sr==<yourIoTHub>.azure-devices.net%2F…..
We only need this part for authentication:
SharedAccessSignature sr==<yourIoTHub>.azure-devices.net%2F…..
The Azure IoT Hub uses TLS for the connection as well. To achieve the root CA, you can clone the Azure IoT C SDK located here or obtain the DigiCert Baltimore Root Certificate manually. Neither the web interface nor the Device Explorer provides it.
To establish a connection from a Qt application using Qt MQTT the code looks like this
const QString iotHubName = QStringLiteral("<yourIoTHub>");
const QString iotHubHostName = iotHubName + QStringLiteral(".azure-devices.net");
const QString deviceId = QStringLiteral("<yourDeviceName>");
QMqttClient client;
client.setPort(8883);
client.setHostname(iotHubHostName);
client.setClientId(deviceId);
client.setUsername(iotHubHostName + QStringLiteral("/") + deviceId + QStringLiteral("/?api-version=2018-06-30"));
client.setPassword(QLatin1String("SharedAccessSignature sr=<yourIoTHub>.azure-devices.net%2Fdevices…"));
auto caCerts = QSslCertificate::fromData(QByteArray(certificates));
QSslConfiguration sslConf;
sslConf.setCaCertificates(caCerts);
client.connectToHostEncryped(sslConf);
Google Cloud IoT Core
Once you have created an account for the Google Cloud Platform, the web interface provides a wizard to get your first project running using Cloud IoT Core.
Once the project has been created, it might be hard to find your registry. A registry stores all information on devices, communication, rules, etc.
Similar to Microsoft Azure, all available services are placed on the dashboard. You will find the IoT Core item in the Big Data section on the left side.
After using the Google Cloud Platform for a while, you will find the search very useful to get to your target page.
From the registry itself you can now add new devices.
The interface asks you to provide the keys/certificates for your device. But it does not have a mean to create some from the service itself. Documentation exists on how to create these. And at production stage those steps will probably be automated in a different manner. However, for getting started these are additional steps required, which can become a hurdle.
Once your device is entered into the registry, you can start with the client side implementation.
Contrary to other providers, Google Cloud IoT Core does not use the device certificate while creating a connection. Instead, the private key is used for the password creation. The password itself needs to be generated as a JSON Web Token. While JSON Web Tokens are an open industry standard, this adds another dependency to your project. Something needs to be able to create these tokens. Google provides some sample code here, but adaptations to include it into an application are required.
The client ID for the MQTT connection is constructed of multiple parameters and has the following form:
projects/PROJECT_ID/locations/REGION/registries/REGISTRY_ID/devices/DEVICE_ID
From personal experience, be aware of case sensitivity. Everything but the Project ID keeps the same capitalization as you created your project, registry and device. However, the project ID will be stored in all lower-case.
Having considered all of this, the simplest implementation to establish a connection looks like this:
const QString rootCAPath = QStringLiteral("root_ca.pem");
const QString deviceKeyPath = QStringLiteral("rsa_private.pem");
const QString clientId = QStringLiteral("projects/PROJECT_ID/locations/REGION/registries/REGISTRY_ID/devices/DEVICE_ID");
const QString googleiotHostName = QStringLiteral("mqtt.googleapis.com");
const QString password = QByteArray(CreateJwt(deviceKeyPath, "<yourprojectID>", "RS256"););
QMqttClient client;
client.setKeepAlive(60);
client.setPort(8883);
client.setHostname(googleiotHostName);
client.setClientId(clientId);
client.setPassword(password);
QSslConfiguration sslConf;
sslConf.setCaCertificates(QSslCertificate::fromPath(rootCAPath));
client.connectToHostEncrypted(sslConf);Alibaba Cloud IoT Platform
The Alibaba Cloud IoT Platform is the only product which does come in multiple variants, a basic and a pro version. As of writing of this article this product structure seems to have changed. From what we can say it does not have an influence on the MQTT related items investigated here.
After creating an account for the Alibaba Cloud the web dashboard allows to create a new IoT Platform instance.
Following the instantiation, a wizard interface allows to create a product and a device.
From these we need a couple of details to establish a MQTT connection.
- Product Key
- Product Secret
- Device Name
- Device Secret
The implementation requires a couple of additional steps. To acquire all MQTT specific properties, the Client ID, username and password are created by concatenations and signing. This procedure is fully documented here. For convenience the documentation also includes example source code to handle this. If the concern is to not introduce external code, the instructions in the first link have to be followed.
To connect a QMqtt client instance, this is sufficient
iotx_dev_meta_info_t deviceInfo;
qstrcpy(deviceInfo.product_key, "<yourproductkey>");
qstrcpy(deviceInfo.product_secret, "<yourproductsecret>");
qstrcpy(deviceInfo.device_name, "<yourdeviceID>");
qstrcpy(deviceInfo.device_secret, "<yourdeviceSecret>");
iotx_sign_mqtt_t signInfo;
int32_t result = IOT_Sign_MQTT(IOTX_CLOUD_REGION_GERMANY, &deviceInfo, &signInfo);
QMqttClient client;
client.setKeepAlive(10000);
client.setHostname(QString::fromLocal8Bit(signInfo.hostname));
client.setPort(signInfo.port);
client.setClientId(QString::fromLocal8Bit(signInfo.clientid));
client.setUsername(QString::fromLocal8Bit(signInfo.username));
client.setPassword(QString::fromLocal8Bit(signInfo.password));
client.connectToHost();You might recognize that we are not using QMqttClient::connectToHostEncrypted() as for all other providers. The Alibaba Cloud IoT Platform is the only vendor, which uses a non-TLS connection by default. It is documented, that it is possible to use one and also to receive a rootCA. However, the fact that this is possible after all surprises.
Standard derivation (limitations)
So far, we have established a MQTT connection to each of the IoT vendors. Each uses a slightly different approach to identify and authenticate a device, but all of these services follow the MQTT 3.1.1 standard.
However, for the next steps developers need to be aware of certain limitations or variations to the standard. These will be discussed next.
None of the providers have built-in support for quality-of-service (QoS) level 2. To some extend that makes sense, as telemetry information do not require multiple steps to verify message delivery. Whether a message is processed and validated is not of interest in this scenario. A developer should be aware of this limitation though.
To refresh our memory on terminology, let us briefly recap retained and will messages.
Retained messages are stored on the server side for future subscribers to receive the last information available on a topic. Will messages embedded to the connection request and will only be propagated in case of an unexpected disconnect from the client.
Amazon IoT Core
The client ID is used to identify a device. If a second device uses the same ID during a connection attempt, then the first device will be disconnected without any notice. The second device will connect successfully. If your application code contains some sort of automatic reconnect, this can cause all devices with the same client ID to be unavailable.
Retained messages are not supported by AWS and trying to send a retained message will cause the connection to be closed.
AWS IoT Core supports will messages within the given allowed topics.
A full description of standard deviations can be viewed here.
Microsoft Azure IoT Hub
The client ID is used to identify a device. The behavior of two devices with the same ID is the same as for Amazon IoT Core.
Retained messages are not supported on the IoT Hub. However, the documentation states that the Hub will internally append a flag to let the backend know that the messages was intended as retained.
Will messages are allowed and supported, given the topic restrictions which will be discussed below.
A full description of standard deviations can be viewed here.
Google Cloud IoT Core
This provider uses the client ID and the password to successfully identify a device.
Messages flagged as retain seem to lose this option during delivery. According to the debug logs they are forwarded as regular messages. We have not found any documentation about whether it might behave similar to the Azure IoT Hub, which forwards this request to its internal message queue.
Will messages do not seem to be supported. While it is possible to store a will message in the connect statement, it does get ignored in case of irregular disconnect.
Alibaba Cloud IoT Platform
The triplet of client ID, username and password are used to identify a device within a product.
Both, the retain flag as well as will messages, are getting ignored from the server side. A message with retain specified is forwarded as a regular message and lost after delivery. Will messages are ignored and not stored anywhere during a connection.
Available (custom) Topics
MQTT uses a topic hierarchy to create a fine-grained context for messages. Topics are similar to a directory structure, starting from generic to device specific. One example of a topic hierarchy would be
Sensors/Europe/Germany/Berlin/device_xyz/temperature
Each IoT provider handles topics differently, so developers need to be very careful on this section.
Amazon IoT Core
First, one needs to check which topics can be used by default. From the dashboard, browse to Secure-> Policies and select the default created policy. It should look like this
AWS IoT Core specifies policies in JSON format, and you will find some of the previous details specified in this document. For instance, the available client IDs are specified in the Connect resource. It also allows to declare which topics are valid for publication, subscribing and receiving. It is possible to have multiple policies in place and devices need to have a policy attached. That way, it allows for a fine-grained security model where certain types groups have different access rights.
Note that the topic description also allows wildcards. Those should not be confused with the wildcards in the MQTT standard. Meaning, you must use * instead of # to enable all subtopics.
Once you have created a topic hierarchy based on your needs, the code itself is simple
client.publish(QStringLiteral("topic_1"), "{\"message\":\"Somecontent\"}", 1);
client.subscribe(QStringLiteral("topic_1"), 1);Microsoft Azure IoT Hub
The IoT Hub merely acts as an interface to connect existing MQTT solutions to the Hub. A user is not allowed to specify any custom topic, nor is it possible to introduce a topic hierarchy.
A message can only be published in the following shape:
const QString topic = QStringLiteral("devices/") + deviceId + QStringLiteral("/messages/events/");
client.publish(topic, "{id=123}", 1);
For subscriptions similar limitations exist
client.subscribe(QStringLiteral("devices/") + deviceId + QStringLiteral("/messages/devicebound/#"), 1);
The wildcard for the subscription is used for additional information that the IoT Hub might add to a message. This can be a message ID for instance. To combine multiple properties the subtopic itself is url-encoded. An example message send from the IoT Hub has this topic included
devices/TestDevice01/messages/devicebound/%24.mid=7493c5cc-d783-4ecd-8129-d3c87590b544&%24.to=%2Fdevices%2FTestDevice01%2Fmessages%2FdeviceBound&iothub-ack=full
Google Cloud IoT Core
By default, a MQTT client should use this topic for publication
/devices/<deviceID>/events
But it is also possible to add additional topics using the Google Cloud Shell or other APIs.
In this case a topic customCross has been created. Those additional topics are reflected as subtopics on the MQTT side, though, meaning to publish a message to this topic, it would be
/devices/<deviceID>/events/customCross
For subscriptions custom topics are not available and there are only two available topics a client can subscribe to
/devices/<deviceID>/commands/# /devices/<deviceID>/config/
Config messages are retained messages from the cloud. Those will be send every time a client connects to keep the device in sync.
Alibaba Cloud IoT Platform
Topics can easily be managed in the Topic Categories tab of the product dashboard.
Each topic can be configured to receive, send or bidirectional communication. Furthermore, a couple of additional topics are generated by default to help creating a scalable structure.
Note that the topic always contains the device ID. This has implications on communication routes as mentioned below.
Communication routes
Communication in the IoT context can be split into three different categories
- Device to Cloud (D2C)
- Cloud to Device (C2D)
- Device to Device (D2D)
The first category is the most common one. Devices provide information about their state, sensor data or any other kind of information. Talking in the other direction happens in the case of providing behavior instructions, managing debug levels or any generic instruction.
Regarding device-to-device communication, we need to be a bit more verbose on the definition inside this context. A typical example can be taken from the home automation. Given a certain light intensity, the sensor propagates the information and the blinds automatically react to this by going down (Something which never seems to work properly in office spaces ;) ). Here, all logic is handled on the devices and no cloud intelligence is needed. Also, no additional rules or filters need to be created in the cloud instance itself. Surely, all tested providers can instantiate a method running in the cloud and then forwarding a command to another device. But that process is not part of this investigation.
Amazon IoT Core
In the previous section we already covered the D2C and C2D cases. Once a topic hierarchy has been specified a client can publish to these topics, and also subscribe to one.
To verify that the C2D connection works, select the Test tab on the left side of the dashboard. The browser will show a minimal interface, which allows to send a message with a specified topic.
Also, the device-to-device case is handled nicely by subscribing and publishing to a topic as specified in the policy.
Microsoft Azure IoT Hub
It is possible to send messages from a device to the cloud and vice-versa. However, a user is not free to choose a topic.
For sending the Device Explorer is a good utility, especially for testing the property bag feature.
Device to device communication as in our definition is not possible using Azure IoT Hub.
During the creation of this post, this article popped up. It talks about this exact use case using the Azure SDKs instead of plain MQTT. The approach there is to locate the Service SDK on the recipient device. So for bidirectional communication this would be needed on all devices, with the advantage of not routing through any server.
Google Cloud IoT Core
Sending messages from a device to the cloud is possible, allowing further granularity with subtopics for publication. Messages are received on two available topics as discussed in above section.
As the custom topics still include the device ID, it is not possible to use a Google Cloud IoT Core instance as standard broker to propagate messages between devices (D2D).
The dashboard for a device allows to send a command, as well as a configuration from the cloud interface to the device itself.
Alibaba Cloud IoT Platform
Publishing and Subscribing can be done in a flexible manner using the IoT Platform. (Sub-)Topics can be generated to provide more structure.
To test sending a message from the cloud to a device, the Topic List in the device dashboard includes a dialog.
Device to device communication is also possible. Topics for these cannot be freely specified, they must reside exactly one level below
/broadcast/<yourProductName>/
. The topic on this sub-level can be chosen freely.
Other / References
Amazon IoT Core
- MQTT specifics for AWS
- Available default topics
- Design Guide for Topic hierarchies, also a very good reference for non AWS related designs
Microsoft Azure IoT Hub
Google Cloud IoT Core
Alibaba Cloud IoT Platform
Additional notes
MQTT version 5 seems to be too young for the biggest providers to adopt to. This is very unfortunate, given that the latest standard adds in a couple of features specifically useful in the IoT world. Shared subscriptions would allow for automatic balancing of tasks, the new authentication command allows for higher flexibility registering devices, connection and message properties enable cloud connectivity to be more performant and easier to restrict/configure, etc. But at this point in time, we will have to wait for its adoption.
Again, I want to emphasize that we have not looked into any of the features above IoT solutions provide to handle messages once received. This is a part of a completely different study and we would be very interested in hearing from your results in that field.
Additionally, we have not included RPC utilization of the providers. Some have hard coded topics to handle RPC like Google differentiating between commands and configuration. Alibaba even uses default topics to handle firmware update notifications via MQTT. TrendMicro has released a study on security related concerns in MQTT and RPC has a prominent spot in there, a must read for anyone setting up a MQTT architecture from scratch.
How can I test this myself?
I’ve created a sample application, which allows to connect to any of the above cloud vendors when required details are available. The interface itself is rather simple:
You can find the source code here on GitHub.
Closing words
For any of the broader IoT and cloud providers it is possible to connect a telemetry-based application using MQTT (and Qt MQTT). Each has different variations on connection details, also to which extend the standard is fully available for developers.
Personally, I look forward to the adoption of MQTT version 5. The AUTH command allows for better integration of authentication methods, other features like topic aliases and properties bring in further use-cases for the IoT world. Additionally, shared-subscription are beneficial to create a data-worker relationship between devices. This last point however might step onto the toes of cloud vendors, as the purpose of them is to handle the load inside the cloud.
I would like to close this post with questions to you.
- What is your experience with those cloud solutions?
- Is there anything in the list, I might have missed?
- Should other vendors or companies be included as well?
Looking forward to your feedback…
