[BUG] Unable to authenticate from Function App's iothub function binding with user-assigned MSI: InvalidIssuer: Token issuer is invalid

Library name and version

Microsoft.Azure.WebJobs.Extensions.EventHubs: 5.0.0

Describe the bug

A Function App containing an IoT hub trigger binding configured with am eventhub connection via a user-assigned MSI is unable to authenticate with the EventHub, the SDK throws an exception Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid.

Expected behavior

The Function App is able to authenticate successfully with the event hub.

Actual behavior

The following exception is thrown by the Function App runtime, indicating that the function has failed to start (indeed it is never invoked successfully).

Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid

2022-03-10T13:52:17.487 [Information] Starting JobHost 2022-03-10T13:52:17.490 [Information] Starting Host (HostId=func-d-10007318-cdbfx-hdz7, InstanceId=e42ad721-4353-4d59-8d42-ca9638defca1, Version=4.1.3.17473, ProcessId=7160, AppDomainId=1, InDebugMode=True, InDiagnosticMode=False, FunctionsExtensionVersion=~4) 2022-03-10T13:52:17.504 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:java 2022-03-10T13:52:17.506 [Information] FUNCTIONS_WORKER_RUNTIME set to node. Skipping WorkerConfig for language:powershell 2022-03-10T13:52:17.508 [Information] Loading functions metadata 2022-03-10T13:52:17.533 [Information] 1 functions loaded 2022-03-10T13:52:17.559 [Information] Generating 1 job function(s) 2022-03-10T13:52:17.599 [Information] Found the following functions: Host.Functions.iothub-events-func 2022-03-10T13:52:18.295 [Information] Initializing function HTTP routes No HTTP routes mapped 2022-03-10T13:52:18.301 [Information] Host initialized (804ms) 2022-03-10T13:52:18.343 [Information] HttpOptions { “DynamicThrottlesEnabled”: true, “EnableChunkedRequestBinding”: false, “MaxConcurrentRequests”: 100, “MaxOutstandingRequests”: 200, “RoutePrefix”: “api” } 2022-03-10T13:52:18.418 [Information] Worker process started and initialized. 2022-03-10T13:52:22.613 [Information] Host lock lease acquired by instance ID ‘ffb36bd0b4e8606c3d02b60790248fe9’. 2022-03-10T13:52:30.511 [Error] The listener for function ‘Functions.iothub-events-func’ was unable to start. Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function ‘Functions.iothub-events-func’ was unable to start. —> System.AggregateException : One or more errors occurred. (InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0)) —> InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor1.StartProcessingInternalAsync[TPartition](Boolean async,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor1.StartProcessingAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Processor.EventProcessorHost.StartProcessingAsync(IEventProcessorFactory processorFactory,BlobsCheckpointStore checkpointStore,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.EventHubs.Listeners.EventHubListener.StartAsync(CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs : 68 —> (Inner Exception #0) Azure.Messaging.EventHubs.EventHubsException : InvalidIssuer: Token issuer is invalid. TrackingId:ec70fbfe-4f50-4621-a6e1-c58a67070b49, SystemTracker:NoSystemTracker, Timestamp:2022-03-10T13:52:30 (iot-d-10007318-0) at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response,String eventHubName) at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor1.ValidateEventHubsConnectionAsync[TPartition](CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Messaging.EventHubs.Primitives.EventProcessor`1.ValidateStartupAsync[TPartition](Boolean async,CancellationToken cancellationToken)<—

End of inner exception

It is unclear how I can further diagnose the issue, as I lack access to the Azure Functions runtime. I verified that the app settings are indeed set from the scm.azurewebsites.net page.

Reproduction Steps

Deploy a Function App with an IoT hub trigger using a user-managed MSI for authentication. When Azure Functions tries to start the function, you will find the InvalidIssuer exception in the function logs.

Environment

• IoT hub trigger binding configured with eventhub connection via a user-assigned MSI.
• The function app runs on a Windows Consumption Function App,
• The following App settings are set to authenticate via a user-managed MSI, in accordance with the docs on Identity-based connections:

iotHubTriggerAppSettingsPrefix__fullyQualifiedNamespace = iothub-ns-iot-d-1000-17752060-5d3adbd147.servicebus.windows.net
iotHubTriggerAppSettingsPrefix__credential = managedidentity
iotHubTriggerAppSettingsPrefix__clientId = 7c4129d6-a91b-4d8f-b2b7-0c4d54734082

• The user-assigned MSI has been granted a role assignment on the resource group containing the IoT hub for the Azure Event Hubs Data Receiver role.

• The function’s Eventhub binding is configured as follows (function.json; the function is written in javascript):

{
  "bindings": [
    {
      "type": "eventHubTrigger",
      "name": "IoTHubMessages",
      "direction": "in",
      "connection": "iotHubTriggerAppSettingsPrefix",
      "eventHubName": "%eventHubName%",
      "cardinality": "many",
      "consumerGroup": "$Default"
    },

• The function is using the latest extension bundle publicly available: Azure Functions Extensions Bundles Preview release 4.0.0.

• Windows Consumption plan Function app details:

2022-03-10T13:52:32.614 [Information] Host Status: {
  "id": "func-d-10007318-cdbfx-hdz7",
  "state": "Running",
  "version": "4.1.3.17473",
  "versionDetails": "4.1.3+3ed9ce8ebeef5b156badaf203a016a56e819a852",
  "platformVersion": "97.0.7.624",
  "instanceId": "ffb36bd0b4e8606c3d02b60790248fe9ad51747a9ac80fb7cc625589efab50b4",
  "computerName": "10-30-14-75",
  "processUptime": 17854,
  "functionAppContentEditingState": "Unknown",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle.Preview",
    "version": "4.0.0"
  }
}

edit:

• The function app has two user-managed MSIs assigned: one for authenticating with the IoT hub eventhub and a second one for authenticating with a key vault (containing the storage account connection string, amongst others). There is no system-managed MSI assigned to the function app

[fvdnabee]

That is indeed unfortunate, might I then suggest to update the IoT hub function docs with this limitation and to just include a link to the docs on the eventhub binding for identity-based connections.

Are there any plans for supporting Azure AD Auth on the Event Hub Compatible Endpoint? Could be an interesting differentiator for the Standard SKU.

[jlorich]

This is unfortunately not a bug, but incorrect Azure Functions documentation. If you take a look at the Permissions for IoT Hub Service APIs section you can see that Azure AD Auth for the Event Hub Compatible Endpoint is not supported. Only key-based access is available.

If you’d like to use Managed Identities end to end, the recommendation is to use IoT Hub Routing to send data to an event hub you control (outbound routing can be authenticated with Managed Identity). And then read from that event hub using MI.

We should update the functions doc to show the correct capabilities: @ggailey777 @cachai2 @dksimpson @craigshoemaker