Workflow run is blocked at the automatic update for the self-hosted runner application

In this ticket, the user has installed a self-hosted macOS runner and also set the proxy configuration. when he uses this runner to run the workflow, the workflow run is blocked at the automatic update for the self-hosted runner application. The following is the debug logs shared by the user:

[2020-04-09 12:48:42Z INFO RSAFileKeyManager] Loading RSA key parameters from file /Volumes/beyourself/dev_tools/github_action/actions-runner/.credentials_rsaparams
[2020-04-09 12:48:42Z INFO MessageListener] Message '8' received from session 'd4def077-beb1-4999-a88c-4a5333940ce5'.
[2020-04-09 12:48:43Z INFO Runner] Refresh message received, kick-off selfupdate background process.
[2020-04-09 12:48:45Z INFO SelfUpdater] Version '2.168.0' of 'agent' package available in server.
[2020-04-09 12:48:45Z INFO SelfUpdater] Current running runner version is 2.165.2
[2020-04-09 12:48:45Z INFO SelfUpdater] An update is available.
[2020-04-09 12:48:45Z INFO Terminal] WRITE LINE: Runner update in progress, do not shutdown runner.
[2020-04-09 12:48:46Z INFO Terminal] WRITE LINE: Downloading 2.168.0 runner
[2020-04-09 12:48:46Z INFO HostContext] Well known directory 'Bin': '/Volumes/beyourself/dev_tools/github_action/actions-runner/bin'
[2020-04-09 12:48:46Z INFO HostContext] Well known directory 'Root': '/Volumes/beyourself/dev_tools/github_action/actions-runner'
[2020-04-09 12:48:46Z INFO HostContext] Well known directory 'Work': '/Volumes/beyourself/dev_tools/github_action/actions-runner/_work'
[2020-04-09 12:48:47Z INFO SelfUpdater] Attempt 1: save latest runner into /Volumes/beyourself/dev_tools/github_action/actions-runner/_work/_update/runner1.tar.gz.
[2020-04-09 12:48:47Z INFO SelfUpdater] Download runner: begin download
[2020-04-09 13:03:47Z WARN SelfUpdater] Runner download has timed out after 900 seconds
[2020-04-09 13:03:47Z WARN SelfUpdater] Failed to get package '/Volumes/beyourself/dev_tools/github_action/actions-runner/_work/_update/runner1.tar.gz' from 'https://github.com/actions/runner/releases/download/v2.168.0/actions-runner-osx-x64-2.168.0.tar.gz'. Exception System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
---> System.Net.Sockets.SocketException (89): Operation canceled
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
at GitHub.Runner.Listener.SelfUpdater.DownloadLatestRunner(CancellationToken token)
[2020-04-09 13:03:47Z INFO SelfUpdater] Attempt 2: save latest runner into /Volumes/beyourself/dev_tools/github_action/actions-runner/_work/_update/runner1.tar.gz.
[2020-04-09 13:03:47Z INFO SelfUpdater] Download runner: begin download

Looks like, there are some problems cause the download of the latest runner application package to be failed, and then it goes to try the download again and again.

[krissmarteragent]

I’m not the original reporter that BrightRan opened this issue for, but i’m having similar issues when my self-hosted agents try to auto-update. This has happened on 2 successive version updates, -> 2.267.1 & -> 2.273.0 both times from the previous versions. What I’ve noticed is that the auto-update feature seems to be pulling down the new version and creating 2 new folders, in the last instance bin.2.273.0 & externals.2.273.0, but the origin bin & external folders from the previous version remain. From there my jobs just queue up. If I restart the runners with systemctl or init script, then restart my jobs, the runners just go back into an auto-update cycle & the new jobs queue up, as if waiting for the auto-update to complete.

seems to me there’s a final step either missing or unable to run in the auto-update process where the old bin & external folders are not replaced by the new versions, whether through some symlinking or straight up overwriting

[thboop]

Going to close this out since its been a while since its been reported, if we see this again we can reopen.