As companies continue to fully roll out their IoT projects, management of the various components of the solution becomes a critical part of their operations. The flexibility of Azure IoT Hub to enable customers to start small, paying only for the amount of IoT Hub capacity needed at any point along the device deployment curve, helps drive predictability in the cost of an IoT solution.
However, the potentially irregular rate of device and message growth in an IoT solution does add a unique challenge for operations. When the number of messages ingested from devices in a given day exceeds the limit of the chosen IoT Hub capacity, the IoT Hub will begin to reject messages until either the IoT Hub is scaled-up, or the time rolls over into the next day (UTC time). Wouldn’t it be nice to have IoT Hub just automatically scale up to a higher capacity when a certain threshold of messages is met, before this limit is reached?
While at this point, IoT Hub does not have this capability built into the service, we have published a sample solution for monitoring and automatically scaling your IoT Hub based on reaching a specific threshold of messages. The sample, published on the Azure-Samples site, leverages the Azure Durable Functions framework and the IoT Hub Management Client to continually monitor the consumption of your IoT Hub message quota and, when needed, programmatically scale up your IoT Hub capacity.
Azure Durable Functions
To orchestrate our IoT Hub scaling solution, we leverage the Singleton Orchestrator pattern of the Azure Durable Functions framework. The key benefit of this pattern is the ability to ensure that exactly one instance of the scaling solution for a given IoT Hub is running at a time. That frees us from having to worry about the possible race conditions of multiple instances of our scaling function running concurrently. The pattern really consists of three functions that operate our solution:
- IotHubScaleInit - this function is executed on a regular timer (by default, once per hour). This function checks to see if an instance of the Orchestrator function is running and, if not, starts one.
- IotHubScaleOrchestrator - this function implements the Orchestrator for the solution. It’s role in the pattern is to manage the execution of the worker function
- IotHubScaleWorker - this is the function that performs the actions of checking to see if the IoTHub needs to be scales and, if so, scaling it.
We start with a timer-initiated IoTHubScaleInit function that runs occasionally (in the sample, once an hour) and checks to see if an instance of the orchestrator is already running and, if not, starts one. The relevant code, from the IoTHubScaleInit function is shown below, some code removed for brevity.
const string IotHubScaleOrchestratorInstanceId = "iothubscaleorchestrator_1";
var existingInstance = await starter.GetStatusAsync(IotHubScaleOrchestratorInstanceId);
if (existingInstance == null)
{
await starter.StartNewAsync(IotHubScaleOrchestratorName,IotHubScaleOrchestratorInstanceId, input: null);
}
The key to this function is the constant instance ID. By default, when you launch an orchestrator, the system will generate a unique instance ID. In our case, by specifying an ID, we can check and see if that instance is already running with the GetStatusAsync function.
The IotHubScaleOrchestrator function, as the name implies, orchestrates the execution of the solution. It recovers from failures in execution, and also allows dehydration of the code while waiting on the next execution. But most importantly, it allows us to make sure we kick off another instance of the scaling function after the existing one finishes. This is the critical part of making sure we never have more than one instance executing at a given time. The key parts of this function are:
await context.CallActivityAsync(IotHubScaleWorkerName); DateTime wakeupTime = context.CurrentUtcDateTime.Add(TimeSpan.FromMinutes(JobFrequencyMinutes)); await context.CreateTimer(wakeupTime, CancellationToken.None); context.ContinueAsNew(null);
After calling and waiting on the worker function, we create a timer via the Durable Functions framework. The ContinueAsNew method of the context object then tells the framework to end this instance and schedule another one to fire up when the timer expires. The framework takes care of the rest.
The remainder of the solution is the IotHubScaleWorker function, which performs the actual work of checking the status of the IoT Hub usage and, if necessary, scaling it.
IoT Hub Management Client
The IoT Hub Management Client enables you to interact with the control plane of the IoT Hub service, including creating, deleting, and managing the configuration of your IoT Hubs. Within the worker function, the client does all of the heavy lifting of interacting with the IoT Hub service.
For example, the following two snippets from the code get the current configuration details of the IoT Hub, the most important of which for our purposes is the current SKU (S1, S2, or S3) and the current number of units. The second line gets the current operational metrics of the hub. The primary one of interest is the TotalMessages metric which gives the current number of messages the IoT Hub has ingested that day.
IotHubDescription desc = client.IotHubResource.Get(ResourceGroupName, IotHubName); IPage<iothubquotametricinfo> mi = client.IotHubResource.GetQuotaMetrics(ResourceGroupName, IotHubName);
Once we have that information, we determine via a couple of helper functions included in the sample, if we need to scale the IoT Hub by comparing the current message count with a defined threshold for that SKU/unit combination. If we need to scale, we simply update the SKU and units within the IoTHubDescription object we obtained above, and leverage the CreateOrUpdate management function to update the configuration of our IoT Hub. This performs the scale up of the IoT Hub with no interruption to existing devices or clients.
desc.Sku.Name = newSkuName; desc.Sku.Capacity = newSkuUnits; client.IotHubResource.CreateOrUpdate(ResourceGroupName, IotHubName, desc);
Scaling Down
With the trajectory of most IoT projects being growth, and for simplicity, we focused this sample on scaling-up IoT Hubs. However, there are certainly valid scenarios where an IoT Hub may need to be automatically scaled down to lower costs when previous message volumes drop. In the sample documentation, we offer some suggestions for modifying the solution for scaling down IoT Hubs when necessary.
Give the sample a try and sleep better tonight knowing you have one fewer operational tasks on your plate!
A few notes about the sample
- The sample only works for the Standard tiers of IoT Hub. The Free tier of IoT Hub can’t be scaled, so it’s not applicable. Also note that you cannot convert directly from the Free tier of IoT Hub to a Standard tier.
- The sample provides one straightforward implementation of a scaling algorithm, but with the supplied source code, you can customize it to meet your unique scaling needs.
- For the sake of your IoT budget, due consideration should be given to automatically scaling IoT Hub as you reach the higher service levels, such as S3, as each unit increase adds both significant capacity as well as cost.