[BUG] ServiceBus.Core.MessageSender.OnSendAsync throws "InvalidCastException: Unable to cast object of type 'Microsoft.Azure.Amqp.Framing.Accepted' to type 'Microsoft.Azure.Amqp.Transaction.Declared'."

Describe the bug

ServiceBus.Core.MessageSender.OnSendAsync is occasionally throwing the exception “System.InvalidCastException: Unable to cast object of type ‘Microsoft.Azure.Amqp.Framing.Accepted’ to type ‘Microsoft.Azure.Amqp.Transaction.Declared’.”

Expected behavior

No InvalidCastException is thrown. If this is an error due to how I’m using the library, something more descriptive than an AMQP cast exception is thrown so that I can fix how I’m calling the library/service.

Actual behavior (include Exception or Stack Trace)

Here’s an exception stack trace excerpt. Everything after the “…” is Polly retry code or my service’s code:

System.InvalidCastException: Unable to cast object of type 'Microsoft.Azure.Amqp.Framing.Accepted' to type 'Microsoft.Azure.Amqp.Transaction.Declared'.
   at Microsoft.Azure.ServiceBus.Core.MessageSender.OnSendAsync(IList`1 messageList)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.Core.MessageSender.SendAsync(IList`1 messageList)
   at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
...

To Reproduce

Unknown. This happens occasionally when our service tries to add items to a Service Bus queue. I can provide more details, but I would appreciate some guidance on what you’re looking for.

Environment:

• Name and version of the Library package used:
  • Microsoft.Azure.ServiceBus 4.1.3, the most recent version of this package as of 2020-09-02.
• Hosting platform or OS and .NET runtime version:
  • .NET Core version 3.1.0 (commit 157910edee) with runtimes:
    • Microsoft.AspNetCore.App 3.1.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
    • Microsoft.NETCore.App 3.1.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  • Running on Azure Kubernetes Service with
    • Kubernetes version 1.14.8
    • Linux kernel version “4.15.0-1089-azure #<span/>99~16.04.1-Ubuntu SMP Fri Jun 5 15:30:32 UTC 2020 x86_64 GNU/Linux”
• IDE and version : N/A (exception is encountered when running under Azure Kubernetes Service, not an IDE)

[yvgopal]

We couldn’t yet reproduce it on our end. There is something going wrong with AMQP library, in a particular corner case. I just couldn’t reproduce it so far and don’t see any particular code path that may have caused this. We will keep this open, and continue to investigate as a low priority item. In the meantime, if any of you can consistently reproduce it and identify a reproducible pattern, we will be happy to fix it.

[jsquire]

@JoshLove-msft : From the most recent report, it looks as if this can be reproduced on Azure.Messaging.ServiceBus. Would you please take a look and transfer to the AMQP repo if it turns out to be in the AMQP lib?