BadTimeout exception during publish due to sequenceNumber mismatch

Describe the bug We are using publisher (2.8.2) in standalone mode. After updating the pn.json (same data items, same server, new DataSetWriterId - new test in our E2E test pipeline) we sometimes observe that publishing stops or is no longer possible because a timeout exception is constantly thrown (due to sequenceNumber mismatch?). Restarting of the publisher fixes the problem.

To Reproduce Steps to reproduce the behavior:

1. replace pn.json (same content different DataSetWriterIds)
2. publishing stops

Expected behavior Publishing is possible without publisher restart - no Timeout exceptions.

Unfortunately, the error is not reproducible. Happens only from time to time.

Could you explain the behavior? What can be done about it? Is it a bug in publisher or in the OPC UA stack?

Corresponding logs: 2022-08-10T20:59:28.093566790Z [2022-08-10 20:59:28.093 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing done for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' 2022-08-10T20:59:28.106508260Z [2022-08-10 20:59:28.106 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:29.318226166Z [2022-08-10 20:59:29.317 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:29.318324167Z [2022-08-10 20:59:29.317 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0' - sequence# 6-08/10/2022 20:59:29 2022-08-10T20:59:29.318330967Z [2022-08-10 20:59:29.317 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' has unexpected sequenceNumber 6 vs expected 1, isKeepAlive True, publishTime 08/10/2022 20:59:29 2022-08-10T20:59:29.318385268Z [2022-08-10 20:59:29.318 ERR Microsoft.Azure.IIoT.OpcUa.Protocol.Services.StackLogger] (Error) PUBLISH #11 - Unhandled error during Publish. 2022-08-10T20:59:29.318389968Z BadTimeout 'BadTimeout' 2022-08-10T20:59:29.318393368Z 2022-08-10T20:59:29.318396868Z Opc.Ua.ServiceResultException: BadTimeout 2022-08-10T20:59:29.318400668Z at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header) 2022-08-10T20:59:29.318405868Z at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos) 2022-08-10T20:59:29.318409968Z at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result) 2022-08-10T20:59:31.368463479Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:31.368683082Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000' - sequence# 3-08/10/2022 20:59:31 2022-08-10T20:59:31.368886985Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' with sequenceNumber 3, isKeepAlive True, publishTime 08/10/2022 20:59:31 2022-08-10T20:59:31.369025086Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] Found 1 notifications with null value or not good status code for 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' subscription. 2022-08-10T20:59:31.369122188Z [2022-08-10 20:59:31.369 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Message trigger for opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 received message with sequenceNumber 6 2022-08-10T20:59:32.357676861Z [2022-08-10 20:59:32.357 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Sink block in engine opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 triggered with 1 messages 2022-08-10T20:59:35.309186709Z [2022-08-10 20:59:35.308 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW64a4168558ac472baf88d9113c93fe6c': server current state: Running 2022-08-10T20:59:36.371601376Z [2022-08-10 20:59:36.371 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:36.371922681Z [2022-08-10 20:59:36.371 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000' - sequence# 3-08/10/2022 20:59:36 2022-08-10T20:59:36.372202084Z [2022-08-10 20:59:36.371 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' has unexpected sequenceNumber 3 vs expected 4, isKeepAlive False, publishTime 08/10/2022 20:59:36 2022-08-10T20:59:36.372344186Z [2022-08-10 20:59:36.372 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] Found 1 notifications with null value or not good status code for 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' subscription. 2022-08-10T20:59:36.372387287Z [2022-08-10 20:59:36.372 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Message trigger for opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 received message with sequenceNumber 7 2022-08-10T20:59:37.363153326Z [2022-08-10 20:59:37.362 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Sink block in engine opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 triggered with 1 messages 2022-08-10T20:59:38.089600939Z [2022-08-10 20:59:38.089 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Runner Keepalive reset due to checkAlive 2022-08-10T20:59:38.089914843Z [2022-08-10 20:59:38.089 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW64a4168558ac472baf88d9113c93fe6c' 2022-08-10T20:59:38.089927244Z [2022-08-10 20:59:38.089 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' 2022-08-10T20:59:38.090469751Z [2022-08-10 20:59:38.090 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing done for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' 2022-08-10T20:59:38.090482351Z [2022-08-10 20:59:38.090 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing done for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW64a4168558ac472baf88d9113c93fe6c' 2022-08-10T20:59:38.106047653Z [2022-08-10 20:59:38.105 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:39.090222806Z [2022-08-10 20:59:39.090 DBG Microsoft.Azure.IIoT.Agent.Framework.Agent.Worker+JobProcess] Sending job processor heartbeat... 2022-08-10T20:59:39.090907515Z [2022-08-10 20:59:39.090 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.StandaloneJobOrchestrator] Worker update with Keep instruction for job opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7. 2022-08-10T20:59:39.324631644Z [2022-08-10 20:59:39.323 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:39.324653144Z [2022-08-10 20:59:39.323 ERR Microsoft.Azure.IIoT.OpcUa.Protocol.Services.StackLogger] (Error) PUBLISH #15 - Unhandled error during Publish. 2022-08-10T20:59:39.324657344Z BadTimeout 'BadTimeout' 2022-08-10T20:59:39.324660244Z 2022-08-10T20:59:39.324663844Z Opc.Ua.ServiceResultException: BadTimeout 2022-08-10T20:59:39.324666844Z at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header) 2022-08-10T20:59:39.324671844Z at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos) 2022-08-10T20:59:39.324674844Z at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result) 2022-08-10T20:59:39.324678844Z [2022-08-10 20:59:39.323 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0' - sequence# 6-08/10/2022 20:59:39 2022-08-10T20:59:39.324682344Z [2022-08-10 20:59:39.323 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' has unexpected sequenceNumber 6 vs expected 7, isKeepAlive True, publishTime 08/10/2022 20:59:39 2022-08-10T20:59:40.324582602Z [2022-08-10 20:59:40.324 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:40.325115709Z [2022-08-10 20:59:40.324 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] Event for subscription: SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0, sequence#: 6 isKeepAlive: False, publishTime: 08/10/2022 20:59:40 2022-08-10T20:59:40.325123109Z [2022-08-10 20:59:40.324 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Message trigger for opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 received message with sequenceNumber 4

[Mat-pa]

Closing due to inactivity. Please reactivate if the issue exists.

@monikavar please reactivate

[Mat-pa]

We are also experiencing a strange behaviour with unexpected sequenceNumber

[2022-11-04 17:44:46.863 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription '1000_ns=2;s=[redacted identifier]'/'opc.tcp:/[IP:Port, redacted]_1AB1XZ1' has unexpected sequenceNumber 52 vs expected 53, isKeepAlive False, publishTime 11/04/2022 17:45:21

We are on version 2.8.3 and this also occurs after docker restart OPCPublisher