Lost messages when IoT Hub quota exceeded

When the quota of an IoT Hub that an edge device belongs to is exceeded messages are lost somewhere in transport.

Expected Behavior

One would expect edgeHub to behave the same as if it lost connection to the IoT Hub and to persist not-yet-delivered messages until they are delivered or the TTL of the message is exceeded.

Current Behavior

When the IoT Hub quota is exceeded edgeHub appears to behave as if the messages have been successfully delivered (although I cannot verify if the problem is with edgeHub or IoT Hub). When the quota is increased a large percentage of messages generated while the quota was exceeded never reach Azure.

Steps to Reproduce

1. Deploy a device linked to an IoT Hub with 1 unit of S1 provisioned.
2. Set up a route from the IoT Hub to store messages to verify successful/unsuccessful delivery.
3. Make sure the device generates enough messages/day to exceed the quota.
4. Make the edgeHub TTL value suitably large (1 day works).
5. Exceed the IoT Hub quota.
6. Provision an additional unit for the IoT Hub and wait for the device’s edgeHub to catch-up to real time.
7. Check the storage endpoint and verify messages were lost.

Context (Environment)

Device (Host) Operating System

Ubuntu Server 18.04

Architecture

amd64

Container Operating System

Linux containers

Runtime Versions

iotedged

iotedge 1.0.6.1 (3fa6cbef8b7fc3c55a49a622735eb1021b8a5963)

Edge Agent

mcr.microsoft.com/azureiotedge-agent:1.0

Edge Hub

mcr.microsoft.com/azureiotedge-hub:1.0

Docker

Client: Version: 3.0.3 API version: 1.40 Go version: go1.11.4 Git commit: 48bd4c6d Built: Wed Jan 23 16:17:56 2019 OS/Arch: linux/amd64 Experimental: false

Server: Engine: Version: 3.0.4 API version: 1.40 (minimum version 1.12) Go version: go1.11.4 Git commit: 8ecd530 Built: Fri Jan 25 01:45:38 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.2.2 GitCommit: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce runc: Version: 1.0.0-rc6+dev GitCommit: 96ec2177ae841256168fcf76954f7177af9446eb docker-init: Version: 0.18.0 GitCommit: fec3683

Logs

edgeHub logs: https://gist.github.com/jackt-moran/8de29ba80f3e1b2c158467e9362ef812

Please let me know if there are additional logs that would be helpful. Thanks.

[varunpuranik]

The fix for this has been checked in to master. It will be part of our 1.0.8 release.

[arsing]

1.0.8 is released.