Messages during workspace startup are not displayed in order

Describe the bug

When multiple messages are sent to the Che server in a short period (e.g. from the plugin broker), they can be displayed out of order in the workspace loading screen. I don’t know if this is an issue on the che-server side or in che-go-jsonrpc.

This behavior requires us a workaround on the plugin-broker side: https://github.com/eclipse/che-plugin-broker/commit/9d4f58b6e38a1cb61ff15108e0683e41d5d1f950

Steps to reproduce

1. Replace artifacts broker in a current Che deployment with amisevsk/che-plugin-artifacts-broker:eventbus:

   CHE_WORKSPACE_PLUGIN__BROKER_ARTIFACTS_IMAGE=amisevsk/che-plugin-artifacts-broker:eventbus
   

   This image is slightly modified from master: https://github.com/eclipse/che-plugin-broker/compare/master...amisevsk:event-bus-test
2. Start a workspace, note logs like

   Starting plugin artifacts broker
   Test ordering: 04 - ouaqjowzahunexxoy
   Test ordering: 01 - vbazpcssuignanngbljf
   Test ordering: 03 - ksmmbkxfwbbtylaszo
   Test ordering: 02 - xnpkrzvpfzojlljnkvj
   Test ordering: 05 - lnkijrcbirxgeglq
   Test ordering: 06 - eimhjbqytyqyycg
   Test ordering: 09 - lfvpftxwydvy
   Test ordering: 08 - vparthcqhuxln
   Test ordering: 07 - sznpfkgklzzylw
   Test ordering: 10 - awfltjfnsrj
   Test ordering: 11 - kbcbgkbnlz
   Test ordering: 14 - adzmnka
   Test ordering: 12 - zkzidgodq
   Test ordering: 13 - ohzpgvym
   Test ordering: 15 - jvqwfi
   Test ordering: 17 - tdei
   Test ordering: 16 - vuqld
   Test ordering: 18 - ltl
   Test ordering: 19 - nn
   Test ordering: 20 - e
   Cleaning /plugins dir
   

3. Check artifacts broker logs (oc logs -f <workspace-pod> -c che-plugin-artifacts-broker-eventbus) and note that calls are logged in order.

Expected behavior

Messages from plugin broker are displayed in the same order they are sent.

[sleshchenko]

won’t be fixed

[sleshchenko]

@benoitf I just remember some conversation where I’ve been told that we can not guarantee messages order via WebSocket. Now, I’ve checked and there is no reason for WebSocket to change order messages during delivery.

So, the behavior we observe may be a result of Java Threads or as Angel mentioned - our jsonRPC library implementation since we have something like: Browser <-WebSocket_1-> RemoteSubscriptionManager <-> EventService <-> BrokerService <-WebSocket_2-> PluginBroker

So, on browser we see WebSocket_1, to check whether it’s Che Server or che-json-rpc go the library, it’s needed to check WebSocket_2 content.