Restarted host is not picking up newly-deployed assemblies

Spawning this from #1690, as it is a separate issue.

While investigating #1690, I noticed that a customer was seeing ~20 simultaneous requests to shutdown the host, yet the process never recycled. The newly deployed assembly was never loaded so the site had to be restarted to pick up the changes.

Details on finding site name:

| where PreciseTimeStamp between (datetime(2018-03-08 11:00) .. datetime(2018-03-08 12:10))
| where Summary contains "c67d3f04-1134-4b2d-a4c3-266878710784"

And then for seeing the full timeframe of this occurrence:

| where PreciseTimeStamp between (datetime(2018-03-08 11:00) .. datetime(2018-03-08 12:10))
| where AppName == "{sitenamefoundabove}"
| project PreciseTimeStamp, Pid, Details, Summary, FunctionName  

You’ll see that an environment restart was requested around 11:05, yet the process id never changed. That didn’t happen until the next request around 12:05, at which point the host shuts down and the process id changes.

@simonness, FYI

[paulbatum]

@petro2050 Thanks for this additional info. It is possible for multiple workers to be assigned, even when load is low. This sometimes happens when we are running a platform upgrade. We start running your app on another machine so that we can unload it from the existing machine, and we do this with overlap so there is no downtime. I think what was happening here is as you said, an older version of the app running in parallel with the new version.

The fact that even after waiting 10 minutes, the old version was still running and you had to do a restart is suspicious. It sounds like something did not work right. But this is different to the problem I was worried about that can happen very occasionally with deployments that are not based on a package, where the app picks up only some of the file changes.

[andrewdmoreno]

As an update to my last comment, this seems to be happening pretty consistently on our end (function deployment requires a manual restart in order for new code to run).

Just a heads up that we will probably be submitting a formal support request for this today to see if we can get some traction as perhaps it’s something we are doing wrong since no one else appears to be having this issue.