Self-hosted runner fails with error 401 when talking to TFS
See original GitHub issueDescribe the bug I have set up some self-hosted runners that serve a couple of repos under my personal user. Those work alright. Now when I try to set up runners against the parent repos, under an organization I admin, the first runner I test is unable to initialize correctly against the GHA backend.
The runner is a Windows runner. The config.cmd
handshake seems to go well initially, but then I get a TFS-related error.
The self-hosted runner runs using a GitHub service account that is Owner both of the key repository as well as the organization. I have created a PAT for the service account, and given it all scopes available.
(The runners that run against personal repos use PATs for my personal account.)
Somehow, this results in the GitHub Actions UI claiming that the runner is “Idle” - but the runner isn’t actually accepting any jobs.
I believe this begun happening within the last 7 days.
I have no idea how to troubleshoot this further. Any hints?
Complete Runner_ log from the first time that the agent boots up:
[2020-07-13 21:36:06Z INFO HostContext] No proxy settings were found based on environmental variables (http_proxy/https_proxy/HTTP_PROXY/HTTPS_PROXY)
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Credentials': 'C:\A\.credentials'
[2020-07-13 21:36:06Z INFO Listener] Runner is built for Windows (X64) - win-x64.
[2020-07-13 21:36:06Z INFO Listener] RuntimeInformation: Microsoft Windows 10.0.17763.
[2020-07-13 21:36:06Z INFO Listener] Version: 2.263.0
[2020-07-13 21:36:06Z INFO Listener] Commit: 1c35cda326d2c239210601e768b348f0c7e5e3e2
[2020-07-13 21:36:06Z INFO Listener] Culture: en-US
[2020-07-13 21:36:06Z INFO Listener] UI Culture: en-US
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO Listener] Validating directory permissions for: 'C:\A'
[2020-07-13 21:36:06Z INFO CommandLineParser] Parse
[2020-07-13 21:36:06Z INFO CommandLineParser] Parsing 11 args
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: False
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding Command: configure
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: unattended
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: replace
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding flag: unattended
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: url
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding flag: replace
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'url': 'https://github.com/falldamagestudio/UE4-GHA-Engine'
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: token
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'token': '***'
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: name
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'name': 'engine_build_agent'
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: labels
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'labels': 'engine_build_agent'
[2020-07-13 21:36:06Z INFO Listener] Arguments parsed
[2020-07-13 21:36:06Z INFO Runner] ExecuteCommand
[2020-07-13 21:36:06Z INFO ConfigurationStore] currentAssemblyLocation: C:\A\bin\Runner.Listener.dll
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO ConfigurationStore] binPath: C:\A\bin
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO ConfigurationStore] RootFolder: C:\A
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Runner': 'C:\A\.runner'
[2020-07-13 21:36:06Z INFO ConfigurationStore] ConfigFilePath: C:\A\.runner
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Credentials': 'C:\A\.credentials'
[2020-07-13 21:36:06Z INFO ConfigurationStore] CredFilePath: C:\A\.credentials
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'MigratedCredentials': 'C:\A\.credentials_migrated'
[2020-07-13 21:36:06Z INFO ConfigurationStore] MigratedCredFilePath: C:\A\.credentials_migrated
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Service': 'C:\A\.service'
[2020-07-13 21:36:06Z INFO ConfigurationStore] ServiceConfigFilePath: C:\A\.service
[2020-07-13 21:36:06Z INFO CommandSettings] Flag 'help': 'False'
[2020-07-13 21:36:06Z INFO CommandSettings] Flag 'version': 'False'
[2020-07-13 21:36:06Z INFO CommandSettings] Flag 'commit': 'False'
[2020-07-13 21:36:06Z INFO CommandSettings] Command 'configure': 'True'
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: --------------------------------------------------------------------------------
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: | ____ _ _ _ _ _ _ _ _ |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: | / ___(_) |_| | | |_ _| |__ / \ ___| |_(_) ___ _ __ ___ |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: | | | _| | __| |_| | | | | '_ \ / _ \ / __| __| |/ _ \| '_ \/ __| |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: | | |_| | | |_| _ | |_| | |_) | / ___ \ (__| |_| | (_) | | | \__ \ |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: | \____|_|\__|_| |_|\__,_|_.__/ /_/ \_\___|\__|_|\___/|_| |_|___/ |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: | |
[2020-07-13 21:36:06Z INFO Terminal] WRITE: |
[2020-07-13 21:36:06Z INFO Terminal] WRITE: Self-hosted runner registration
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: | |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: --------------------------------------------------------------------------------
[2020-07-13 21:36:06Z INFO ConfigurationManager] ConfigureAsync
[2020-07-13 21:36:06Z INFO ConfigurationStore] IsConfigured()
[2020-07-13 21:36:06Z INFO ConfigurationStore] IsConfigured: False
[2020-07-13 21:36:06Z INFO ConfigurationManager] Is configured: False
[2020-07-13 21:36:06Z INFO CommandSettings] Arg 'url': 'https://github.com/falldamagestudio/UE4-GHA-Engine'
[2020-07-13 21:36:06Z INFO CommandSettings] Remove url from Arg dictionary.
[2020-07-13 21:36:06Z INFO CommandSettings] Arg 'token': '***'
[2020-07-13 21:36:06Z INFO CommandSettings] Remove token from Arg dictionary.
[2020-07-13 21:36:08Z INFO ConfigurationManager] Http response code: OK from 'POST https://api.github.com/actions/runner-registration'
[2020-07-13 21:36:08Z INFO ConfigurationManager] cred retrieved via GitHub auth
[2020-07-13 21:36:08Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[2020-07-13 21:36:08Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:08Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:08Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:08Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:08Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:10Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:10Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:10Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:10Z INFO ConfigurationManager] Test Connection complete.
[2020-07-13 21:36:10Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:10Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:10Z INFO HostContext] Well known config file 'RSACredentials': 'C:\A\.credentials_rsaparams'
[2020-07-13 21:36:10Z INFO RSAEncryptedFileKeyManager] Creating new RSA key using 2048-bit key length
[2020-07-13 21:36:10Z INFO RSAEncryptedFileKeyManager] Successfully saved RSA key parameters to file C:\A\.credentials_rsaparams
[2020-07-13 21:36:10Z INFO ConfigurationManager] Found a private pool with id {1} and name {2} 1, Default
[2020-07-13 21:36:10Z INFO CommandSettings] Arg 'name': 'engine_build_agent'
[2020-07-13 21:36:10Z INFO CommandSettings] Remove name from Arg dictionary.
[2020-07-13 21:36:10Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:10Z INFO CommandSettings] Arg 'labels': 'engine_build_agent'
[2020-07-13 21:36:10Z INFO CommandSettings] Remove labels from Arg dictionary.
[2020-07-13 21:36:10Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:11Z INFO ConfigurationManager] Agent server url resolve by server: 'https://pipelines.actions.githubusercontent.com/serviceHosts/cf20d806-2b42-4d9d-a239-83eff0807664/'.
[2020-07-13 21:36:11Z INFO ConfigurationStore] Saving OAuth credential @ C:\A\.credentials
[2020-07-13 21:36:11Z INFO ConfigurationStore] Credentials Saved.
[2020-07-13 21:36:11Z INFO ConfigurationStore] HasCredentials()
[2020-07-13 21:36:11Z INFO ConfigurationStore] stored True
[2020-07-13 21:36:11Z INFO CredentialManager] GetCredentialProvider
[2020-07-13 21:36:11Z INFO CredentialManager] Creating type OAuth
[2020-07-13 21:36:11Z INFO CredentialManager] Creating credential type: OAuth
[2020-07-13 21:36:11Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file C:\A\.credentials_rsaparams
[2020-07-13 21:36:11Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[2020-07-13 21:36:11Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:11Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:11Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:11Z WARN GitHubActionsService] Authentication failed with status code 401.
WWW-Authenticate: Bearer
Strict-Transport-Security: max-age=2592000
X-TFS-ProcessId: bbde113b-9a9d-4721-aa00-98abb06dc849
ActivityId: 3961f0b8-a449-4b0d-8770-5eac33ffa9b3
X-TFS-Session: fd9a3090-eac9-4fd2-afff-438bf7e92485
X-VSS-E2EID: 00bf5d0f-626a-4827-a516-756ea3cde3ff
X-VSS-SenderDeploymentId: a07ab14e-025a-39c3-8d53-788cd7ce207f
X-TFS-ServiceError: The+user+%27System%3aPublicAccess%3baaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa%27+is+not+authorized+to+access+this+resource.
X-VSS-S2STargetService: 0000005A-0000-8888-8000-000000000000/visualstudio.com
X-MSEdge-Ref: Ref A: C4DB781BF4594A2D8A3044784C3AB8DF Ref B: LON21EDGE0210 Ref C: 2020-07-13T21:36:11Z
Date: Mon, 13 Jul 2020 21:36:10 GMT
[2020-07-13 21:36:11Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file C:\A\.credentials_rsaparams
[2020-07-13 21:36:11Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file C:\A\.credentials_rsaparams
[2020-07-13 21:36:11Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2020-07-13 21:36:11Z INFO CommandSettings] Arg 'work': ''
[2020-07-13 21:36:11Z INFO CommandSettings] Flag 'unattended': 'True'
[2020-07-13 21:36:11Z INFO PromptManager] ReadValue
[2020-07-13 21:36:11Z INFO ConfigurationStore] Saving runner settings.
[2020-07-13 21:36:11Z INFO ConfigurationStore] Settings Saved.
[2020-07-13 21:36:11Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:11Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:11Z INFO CommandSettings] Flag 'runasservice': 'False'
[2020-07-13 21:36:11Z INFO CommandSettings] Flag 'unattended': 'True'
[2020-07-13 21:36:11Z INFO PromptManager] ReadValue
During subsequent service start-ups, the runner will find the .runner
configuration, load that - but again, receive TFS error 401 a second later.
Runner Version and Platform
Runner: 2.263.0 OS: Microsoft Windows 10.0.17763
Issue Analytics
- State:
- Created 3 years ago
- Reactions:4
- Comments:9
I found a solution/workaround: this happens because your Runner is older version than what is the latest. So github starts blocking requests with older version that is sent in the http useragent.
So i cloned github runn repo. then ran the following commadn from /src folder
it will produce files in /_layout folder. Replace the runner agent binaries on ur server with these
so we basically just manually upgraded the self-hosted agent by recompiling the source
I’m seeing this with a brand new setup - no runners yet, brand new raspberry pi: Getting the exact same log output, although the local runner outputs Connected to GitHub … Listening for Jobs. However, on the GitHub side, the web UI shows no runners configured.