BadTimeout exception during publish due to sequenceNumber mismatch
See original GitHub issueDescribe the bug We are using publisher (2.8.2) in standalone mode. After updating the pn.json (same data items, same server, new DataSetWriterId - new test in our E2E test pipeline) we sometimes observe that publishing stops or is no longer possible because a timeout exception is constantly thrown (due to sequenceNumber mismatch?). Restarting of the publisher fixes the problem.
To Reproduce Steps to reproduce the behavior:
- replace pn.json (same content different DataSetWriterIds)
- publishing stops
Expected behavior Publishing is possible without publisher restart - no Timeout exceptions.
Unfortunately, the error is not reproducible. Happens only from time to time.
Could you explain the behavior? What can be done about it? Is it a bug in publisher or in the OPC UA stack?
Corresponding logs:
2022-08-10T20:59:28.093566790Z [2022-08-10 20:59:28.093 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing done for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' 2022-08-10T20:59:28.106508260Z [2022-08-10 20:59:28.106 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:29.318226166Z [2022-08-10 20:59:29.317 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:29.318324167Z [2022-08-10 20:59:29.317 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0' - sequence# 6-08/10/2022 20:59:29 2022-08-10T20:59:29.318330967Z [2022-08-10 20:59:29.317 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' has unexpected sequenceNumber 6 vs expected 1, isKeepAlive True, publishTime 08/10/2022 20:59:29 2022-08-10T20:59:29.318385268Z [2022-08-10 20:59:29.318 ERR Microsoft.Azure.IIoT.OpcUa.Protocol.Services.StackLogger] (Error) PUBLISH #11 - Unhandled error during Publish. 2022-08-10T20:59:29.318389968Z BadTimeout 'BadTimeout' 2022-08-10T20:59:29.318393368Z 2022-08-10T20:59:29.318396868Z Opc.Ua.ServiceResultException: BadTimeout 2022-08-10T20:59:29.318400668Z at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header) 2022-08-10T20:59:29.318405868Z at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos) 2022-08-10T20:59:29.318409968Z at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result) 2022-08-10T20:59:31.368463479Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:31.368683082Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000' - sequence# 3-08/10/2022 20:59:31 2022-08-10T20:59:31.368886985Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' with sequenceNumber 3, isKeepAlive True, publishTime 08/10/2022 20:59:31 2022-08-10T20:59:31.369025086Z [2022-08-10 20:59:31.368 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] Found 1 notifications with null value or not good status code for 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' subscription. 2022-08-10T20:59:31.369122188Z [2022-08-10 20:59:31.369 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Message trigger for opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 received message with sequenceNumber 6 2022-08-10T20:59:32.357676861Z [2022-08-10 20:59:32.357 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Sink block in engine opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 triggered with 1 messages 2022-08-10T20:59:35.309186709Z [2022-08-10 20:59:35.308 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW64a4168558ac472baf88d9113c93fe6c': server current state: Running 2022-08-10T20:59:36.371601376Z [2022-08-10 20:59:36.371 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:36.371922681Z [2022-08-10 20:59:36.371 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000' - sequence# 3-08/10/2022 20:59:36 2022-08-10T20:59:36.372202084Z [2022-08-10 20:59:36.371 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' has unexpected sequenceNumber 3 vs expected 4, isKeepAlive False, publishTime 08/10/2022 20:59:36 2022-08-10T20:59:36.372344186Z [2022-08-10 20:59:36.372 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] Found 1 notifications with null value or not good status code for 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_1000'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' subscription. 2022-08-10T20:59:36.372387287Z [2022-08-10 20:59:36.372 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Message trigger for opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 received message with sequenceNumber 7 2022-08-10T20:59:37.363153326Z [2022-08-10 20:59:37.362 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Sink block in engine opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 triggered with 1 messages 2022-08-10T20:59:38.089600939Z [2022-08-10 20:59:38.089 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Runner Keepalive reset due to checkAlive 2022-08-10T20:59:38.089914843Z [2022-08-10 20:59:38.089 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW64a4168558ac472baf88d9113c93fe6c' 2022-08-10T20:59:38.089927244Z [2022-08-10 20:59:38.089 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' 2022-08-10T20:59:38.090469751Z [2022-08-10 20:59:38.090 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing done for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' 2022-08-10T20:59:38.090482351Z [2022-08-10 20:59:38.090 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Refreshing done for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW64a4168558ac472baf88d9113c93fe6c' 2022-08-10T20:59:38.106047653Z [2022-08-10 20:59:38.105 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:39.090222806Z [2022-08-10 20:59:39.090 DBG Microsoft.Azure.IIoT.Agent.Framework.Agent.Worker+JobProcess] Sending job processor heartbeat... 2022-08-10T20:59:39.090907515Z [2022-08-10 20:59:39.090 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.StandaloneJobOrchestrator] Worker update with Keep instruction for job opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7. 2022-08-10T20:59:39.324631644Z [2022-08-10 20:59:39.323 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:39.324653144Z [2022-08-10 20:59:39.323 ERR Microsoft.Azure.IIoT.OpcUa.Protocol.Services.StackLogger] (Error) PUBLISH #15 - Unhandled error during Publish. 2022-08-10T20:59:39.324657344Z BadTimeout 'BadTimeout' 2022-08-10T20:59:39.324660244Z 2022-08-10T20:59:39.324663844Z Opc.Ua.ServiceResultException: BadTimeout 2022-08-10T20:59:39.324666844Z at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header) 2022-08-10T20:59:39.324671844Z at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos) 2022-08-10T20:59:39.324674844Z at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result) 2022-08-10T20:59:39.324678844Z [2022-08-10 20:59:39.323 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Notification for session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7', subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0' - sequence# 6-08/10/2022 20:59:39 2022-08-10T20:59:39.324682344Z [2022-08-10 20:59:39.323 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription 'SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0'/'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7' has unexpected sequenceNumber 6 vs expected 7, isKeepAlive True, publishTime 08/10/2022 20:59:39 2022-08-10T20:59:40.324582602Z [2022-08-10 20:59:40.324 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.DefaultSessionManager] Keepalive received from session 'opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7': server current state: Running 2022-08-10T20:59:40.325115709Z [2022-08-10 20:59:40.324 DBG Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] Event for subscription: SingleWriterW3ee5539f421f4972b9a446430f79a0b7_0, sequence#: 6 isKeepAlive: False, publishTime: 08/10/2022 20:59:40 2022-08-10T20:59:40.325123109Z [2022-08-10 20:59:40.324 DBG Microsoft.Azure.IIoT.OpcUa.Edge.Publisher.Engine.DataFlowProcessingEngine] Message trigger for opc.tcp://localhost:21878/_8E33560C_SingleWriterW3ee5539f421f4972b9a446430f79a0b7 received message with sequenceNumber 4
Issue Analytics
- State:
- Created a year ago
- Comments:9 (7 by maintainers)
@monikavar please reactivate
We are also experiencing a strange behaviour with unexpected sequenceNumber
[2022-11-04 17:44:46.863 WRN Microsoft.Azure.IIoT.OpcUa.Protocol.Services.SubscriptionServices+SubscriptionWrapper] DataChange for subscription '1000_ns=2;s=[redacted identifier]'/'opc.tcp:/[IP:Port, redacted]_1AB1XZ1' has unexpected sequenceNumber 52 vs expected 53, isKeepAlive False, publishTime 11/04/2022 17:45:21
We are on version 2.8.3 and this also occurs after docker restart OPCPublisher