Client errors if client=2.12.0 and scheduler=2.13.0 instead of mismatch warning

If client and scheduler versions are mismatched such that client=2.12.0 and scheduler=2.13.0, the client will error out on instantiation with assert msg[0]["op"] == "stream-start" instead of dask’s helpful version mismatch warning. The scheduler’s logs will show an error in versions.pyL125 with TypeError: tuple indices must be integers or slices, not str. Of course, aligning versions fixes this issue, but the usual helpful warning doesn’t appear. Looks to be related to https://github.com/dask/distributed/pull/3567/files

[jrbourbeau]

Thanks @grantgustafson for raising an issue and @consideRatio for investigating. I was able to reproduce the TypeError using client=2.12.0 and scheduler=2.13.0. Changing the value returned from get_system_info from a list of tuples to a dictionary in #3567 seems to have introduced the error. This was an oversight on my part. Reverting the change (i.e. get_system_info returns a list of tuples instead of a dictionary) resolves the issue moving forward.

[consideRatio]

I’ve stranded in my debugging efforts and hope that someone can continue from here, @jrbourbeau can you understand what caused this issue?

It may be that the we got into some bad state by upgrading our scheduler / workers and that we only need to document that when upgrading to 2.13.0 the workers need to fully shut down, then update the scheduler, then add workers back.