[Service Bus] TypeError: Cannot read property '_process' of undefined

Following up for @azure/service-bus v7 package issues from https://github.com/Azure/azure-sdk-for-js/issues/9792#issuecomment-746480634 by @otociulis and @NoelAbrahams Quoting @otociulis:

we’re using “@azure/service-bus”: “^7.0.0”, (latest at the moment of writing) and we are running into same issue when running our microservice utilizing service bus in the Azure Cloud - usually this happens after longer periods of inactivity:

 /home/azure-user/render-service/node_modules/rhea/lib/connection.js:731
             this.local_channel_map[k]._process();
                                       ^
 TypeError: Cannot read property '_process' of undefined
     at Connection._process (/home/azure-user/render-service/node_modules/rhea/lib/connection.js:731:39)
     at processTicksAndRejections (internal/process/task_queues.js:79:11)

Typescript version: 4.0.5 Node.js version: 12.18.4 Rhea version (from package-lock.json): 1.0.24

[chradek]

Reopening as we’ve finally been able to reproduce this issue in one of our testing environments and I was able to track down what is happening.

So, the error that occurs is TypeError: Cannot read property 'process' of undefined. This is being thrown from rhea’s Connection.prototype._process method when iterating over the local session channels:

for (var k in this.local_channel_map) {
  this.local_channel_map[k]._process(); // <-- This is where the error is thrown
}

This looks like it should be completely safe at first glance. If we have 3 sessions on the connection, we’ll iterate over them from the local_channel_map and call _process() on each one. If for some reason a session is deleted from this.local_channel_map, it isn’t a big deal because the for..in is smart enough to skip deleted members.

However, there is also Connection.prototype.remove_all_sessions which we end up calling in our disconnected logic. It completely replaces this.local_channel_map:

Connection.prototype.remove_all_sessions = function() {
  this.remote_channel_map = {};
  this.local_channel_map = {}; // <-- Goodbye old object!
};

I did notice this when I investigated this error in the past and suspected the issue had to do with remove_all_sessions somehow, but couldn’t figure out a code path where it actually would. This method would need to be invoked somehow while iterating over the local_channel_map inside the Connection.prototype._process method. This would lead to the error being thrown because the this.local_channel_map in the for statement would be a different object than this.local_channel_map in the for body. That didn’t seem possible because rhea doesn’t ever call remove_all_sessions, only we do in our SDKs in response to the disconnected event. I figured surely the disconnected event would need to wait for the for..in loop to complete before it could be emitted since that loop is synchronous and thus blocking.

Oh, but there’s always a twist.

Just a quick reminder, this._local_channel_map contains Session objects. When _process is called on a session, that in turn calls _process on each of the session’s links. If a link need to write something, such as a flow frame, rhea ends up calling transport.write on the connection, which takes any pending frames that need to be written and sends them over the socket.

This is where things get interesting. If the socket was already closed for some reason, we get a writeAfterEnd error, which the rhea connection immediately handles and emits the disconnected event. Note that this event is emited synchronously, so the call stack still shows this is all triggered from that session._process that was invoked. Finally, our SDK’s disconnected event handler removes all the sessions not by deleting them individually (which would be safe), but by replacing the local_channel_map object via remove_all_sessions.

In our test that is failing, we’re calling connection.idle, which will cause the socket to end. I suspect that the users who see this would also have encountered the idle timeout at just the right moment for this sequence of events to occur, but really anything that causes the socket to be put in a closed state at the right time could lead to this.

So, how do we fix it?

The most straightforward way would be by updating rhea. There are 2 options:

1. Update remove_all_sessions so that it deletes all the keys in the channel maps explicitly rather than overwriting the maps with new empty objects.
2. Update _process to hold a reference to this._local_channel_map and use the same reference in both the for..in statement and the body of the for loop.

My vote would actually be to do both for total safety. The 1 issue we’re going to run into is rhea has had a major version bump, so we’ll either need to hotfix rhea 1.x, or absorb the breaking change present in rhea 2.x.

[chradek]

Unfortunately I haven’t been able to reproduce this issue. We’re going to close this issue due to inactivity, but if someone encounters it again please open a new issue and if possible provide logs by setting the DEBUG environment variable to DEBUG=azure*,rhea*,-rhea:raw,-rhea:message,-azure:core-amqp:datatransformer