Self-hosted Runner stuck in " Not configured" state

Describe the bug After a restart self-hosted Runner randomly get stuck in “Not configured’” state. Possibly because of a failure to register with GH.

To Reproduce Error affects Runners seemingly at random. After the Runner tries to autoupdate and restarts it may or may not get stuck. We are running ~30 runners for the org.

Expected behavior Runner restarts and registers with GH

Runner Version and Platform

2.278.0 on Linux

What’s not working?

Starting Runner listener with startup type: service Started listener process An error occurred: Not configured Runner listener exited with error code 2 Runner listener exit with retryable error, re-launch runner in 5 seconds indefinitely

Runner and Worker’s Diagnostic Logs

[2021-04-23 17:20:03Z INFO HostContext] No proxy settings were found based on environmental variables (http_proxy/https_proxy/HTTP_PROXY/HTTPS_PROXY) [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Root’: ‘/runner’ [2021-04-23 17:20:03Z INFO HostContext] Well known config file ‘Credentials’: ‘/runner/.credentials’ [2021-04-23 17:20:03Z INFO Listener] Runner is built for Linux (X64) - linux-x64. [2021-04-23 17:20:03Z INFO Listener] RuntimeInformation: Linux 4.14.198-152.320.amzn2.x86_64 #1 SMP Wed Sep 23 23:57:28 UTC 2020. [2021-04-23 17:20:03Z INFO Listener] Version: 2.277.1 [2021-04-23 17:20:03Z INFO Listener] Commit: [2021-04-23 17:20:03Z INFO Listener] Culture: [2021-04-23 17:20:03Z INFO Listener] UI Culture: [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Root’: ‘/runner’ [2021-04-23 17:20:03Z INFO Listener] Validating directory permissions for: ‘/runner’ [2021-04-23 17:20:03Z INFO CommandLineParser] Parse [2021-04-23 17:20:03Z INFO CommandLineParser] Parsing 15 args [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: False [2021-04-23 17:20:03Z INFO CommandLineParser] Adding Command: configure [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: unattended [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: replace [2021-04-23 17:20:03Z INFO CommandLineParser] Adding flag: unattended [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: name [2021-04-23 17:20:03Z INFO CommandLineParser] Adding flag: replace [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] Adding option ‘name’: ‘runners-hi-cpu-q6jhz-f45th’ [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: url [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] Adding option ‘url’: ‘https://github.com/xxx-software’ [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: token [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] Adding option ‘token’: ‘’ [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: runnergroup [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] Adding option ‘runnergroup’: ‘’ [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: labels [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] Adding option ‘labels’: ‘runners-hi-cpu’ [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] arg: work [2021-04-23 17:20:03Z INFO CommandLineParser] parsing argument [2021-04-23 17:20:03Z INFO CommandLineParser] HasArgs: True [2021-04-23 17:20:03Z INFO CommandLineParser] Adding option ‘work’: '/runner/work’ [2021-04-23 17:20:03Z INFO Listener] Arguments parsed [2021-04-23 17:20:03Z INFO Runner] ExecuteCommand [2021-04-23 17:20:03Z INFO ConfigurationStore] currentAssemblyLocation: /runner/bin/Runner.Listener.dll [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO ConfigurationStore] binPath: /runner/bin [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Root’: ‘/runner’ [2021-04-23 17:20:03Z INFO ConfigurationStore] RootFolder: /runner [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Root’: ‘/runner’ [2021-04-23 17:20:03Z INFO HostContext] Well known config file ‘Runner’: ‘/runner/.runner’ [2021-04-23 17:20:03Z INFO ConfigurationStore] ConfigFilePath: /runner/.runner [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Root’: ‘/runner’ [2021-04-23 17:20:03Z INFO HostContext] Well known config file ‘Credentials’: ‘/runner/.credentials’ [2021-04-23 17:20:03Z INFO ConfigurationStore] CredFilePath: /runner/.credentials [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Root’: ‘/runner’ [2021-04-23 17:20:03Z INFO HostContext] Well known config file ‘MigratedCredentials’: ‘/runner/.credentials_migrated’ [2021-04-23 17:20:03Z INFO ConfigurationStore] MigratedCredFilePath: /runner/.credentials_migrated [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Bin’: ‘/runner/bin’ [2021-04-23 17:20:03Z INFO HostContext] Well known directory ‘Root’: ‘/runner’ [2021-04-23 17:20:03Z INFO HostContext] Well known config file ‘Service’: ‘/runner/.service’ [2021-04-23 17:20:03Z INFO ConfigurationStore] ServiceConfigFilePath: /runner/.service [2021-04-23 17:20:03Z INFO CommandSettings] Flag ‘help’: ‘False’ [2021-04-23 17:20:03Z INFO CommandSettings] Flag ‘version’: ‘False’ [2021-04-23 17:20:03Z INFO CommandSettings] Flag ‘commit’: ‘False’ [2021-04-23 17:20:03Z INFO CommandSettings] Flag ‘check’: ‘False’ [2021-04-23 17:20:03Z INFO CommandSettings] Command ‘configure’: ‘True’ [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: -------------------------------------------------------------------------------- [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | ____ _ _ _ _ _ _ _ _ | [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | / () || | | | | | / \ | |() ___ _ __ ___ | [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | | | | | __| || | | | | ’ \ / _ \ / | | |/ _ | ’ / | | [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | | || | | || _ | || | |) | / ___ \ (| || | () | | | _ \ | [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | _||_|| ||_,|./ /_/ __|_||_/|| ||_/ | [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | | [2021-04-23 17:20:03Z INFO Terminal] WRITE: | [2021-04-23 17:20:03Z INFO Terminal] WRITE: Self-hosted runner registration [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: | | [2021-04-23 17:20:03Z INFO Terminal] WRITE LINE: -------------------------------------------------------------------------------- [2021-04-23 17:20:03Z INFO ConfigurationManager] ConfigureAsync [2021-04-23 17:20:03Z INFO ConfigurationStore] IsConfigured() [2021-04-23 17:20:03Z INFO ConfigurationStore] IsConfigured: False [2021-04-23 17:20:03Z INFO ConfigurationManager] Is configured: False [2021-04-23 17:20:03Z INFO CommandSettings] Arg ‘url’: ‘https://github.com/xxx-software’ [2021-04-23 17:20:03Z INFO CommandSettings] Remove url from Arg dictionary. [2021-04-23 17:20:03Z INFO CommandSettings] Arg ‘token’: '’ [2021-04-23 17:20:03Z INFO CommandSettings] Remove token from Arg dictionary. [2021-04-23 17:20:13Z ERR Runner] System.Net.Http.HttpRequestException: Error while copying content to a stream. —> System.IO.IOException: The response ended prematurely. at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken) at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) — End of inner exception stack trace — at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at GitHub.Runner.Listener.Configuration.ConfigurationManager.GetTenantCredential(String githubUrl, String githubToken, String runnerEvent) at GitHub.Runner.Listener.Configuration.ConfigurationManager.ConfigureAsync(CommandSettings command) at GitHub.Runner.Listener.Runner.ExecuteCommand(CommandSettings command) [2021-04-23 17:20:13Z ERR Runner] ##################################################### [2021-04-23 17:20:13Z ERR Runner] System.IO.IOException: The response ended prematurely. at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken) at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) [2021-04-23 17:20:13Z ERR Terminal] WRITE ERROR: Error while copying content to a stream. [2021-04-23 17:20:13Z INFO Listener] Runner execution has finished with return code 1

[Puneeth-n]

@valeramaniukIHM Thanks for the reply. I was already on 0.18.2 of the controller. I figured this happens when we run out of API limits, Runners cannot join GitHub. I switched to purely webhook based scaling in https://github.com/summerwind/actions-runner-controller and that seems to have fixed the issue.

[valeramaniukIHM]

@Puneeth-n yep, updating to the latest version of https://github.com/summerwind/actions-runner-controller did the trick. Now the controller knows how to identify and restart stuck runners. Thank you!