Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Self-hosted runner fails with error 401 when talking to TFS

See original GitHub issue

Describe the bug I have set up some self-hosted runners that serve a couple of repos under my personal user. Those work alright. Now when I try to set up runners against the parent repos, under an organization I admin, the first runner I test is unable to initialize correctly against the GHA backend.

The runner is a Windows runner. The config.cmd handshake seems to go well initially, but then I get a TFS-related error.

The self-hosted runner runs using a GitHub service account that is Owner both of the key repository as well as the organization. I have created a PAT for the service account, and given it all scopes available.

(The runners that run against personal repos use PATs for my personal account.)

Somehow, this results in the GitHub Actions UI claiming that the runner is “Idle” - but the runner isn’t actually accepting any jobs.

I believe this begun happening within the last 7 days.

I have no idea how to troubleshoot this further. Any hints?

Complete Runner_ log from the first time that the agent boots up:

[2020-07-13 21:36:06Z INFO HostContext] No proxy settings were found based on environmental variables (http_proxy/https_proxy/HTTP_PROXY/HTTPS_PROXY)
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Credentials': 'C:\A\.credentials'
[2020-07-13 21:36:06Z INFO Listener] Runner is built for Windows (X64) - win-x64.
[2020-07-13 21:36:06Z INFO Listener] RuntimeInformation: Microsoft Windows 10.0.17763.
[2020-07-13 21:36:06Z INFO Listener] Version: 2.263.0
[2020-07-13 21:36:06Z INFO Listener] Commit: 1c35cda326d2c239210601e768b348f0c7e5e3e2
[2020-07-13 21:36:06Z INFO Listener] Culture: en-US
[2020-07-13 21:36:06Z INFO Listener] UI Culture: en-US
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO Listener] Validating directory permissions for: 'C:\A'
[2020-07-13 21:36:06Z INFO CommandLineParser] Parse
[2020-07-13 21:36:06Z INFO CommandLineParser] Parsing 11 args
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: False
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding Command: configure
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: unattended
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: replace
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding flag: unattended
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: url
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding flag: replace
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'url': 'https://github.com/falldamagestudio/UE4-GHA-Engine'
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: token
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'token': '***'
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: name
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'name': 'engine_build_agent'
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] arg: labels
[2020-07-13 21:36:06Z INFO CommandLineParser] parsing argument
[2020-07-13 21:36:06Z INFO CommandLineParser] HasArgs: True
[2020-07-13 21:36:06Z INFO CommandLineParser] Adding option 'labels': 'engine_build_agent'
[2020-07-13 21:36:06Z INFO Listener] Arguments parsed
[2020-07-13 21:36:06Z INFO Runner] ExecuteCommand
[2020-07-13 21:36:06Z INFO ConfigurationStore] currentAssemblyLocation: C:\A\bin\Runner.Listener.dll
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO ConfigurationStore] binPath: C:\A\bin
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO ConfigurationStore] RootFolder: C:\A
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Runner': 'C:\A\.runner'
[2020-07-13 21:36:06Z INFO ConfigurationStore] ConfigFilePath: C:\A\.runner
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Credentials': 'C:\A\.credentials'
[2020-07-13 21:36:06Z INFO ConfigurationStore] CredFilePath: C:\A\.credentials
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'MigratedCredentials': 'C:\A\.credentials_migrated'
[2020-07-13 21:36:06Z INFO ConfigurationStore] MigratedCredFilePath: C:\A\.credentials_migrated
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:06Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:06Z INFO HostContext] Well known config file 'Service': 'C:\A\.service'
[2020-07-13 21:36:06Z INFO ConfigurationStore] ServiceConfigFilePath: C:\A\.service
[2020-07-13 21:36:06Z INFO CommandSettings] Flag 'help': 'False'
[2020-07-13 21:36:06Z INFO CommandSettings] Flag 'version': 'False'
[2020-07-13 21:36:06Z INFO CommandSettings] Flag 'commit': 'False'
[2020-07-13 21:36:06Z INFO CommandSettings] Command 'configure': 'True'
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: --------------------------------------------------------------------------------
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |        ____ _ _   _   _       _          _        _   _                      |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |                                                                              |
[2020-07-13 21:36:06Z INFO Terminal] WRITE: |
[2020-07-13 21:36:06Z INFO Terminal] WRITE: Self-hosted runner registration
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE:                         |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: |                                                                              |
[2020-07-13 21:36:06Z INFO Terminal] WRITE LINE: --------------------------------------------------------------------------------
[2020-07-13 21:36:06Z INFO ConfigurationManager] ConfigureAsync
[2020-07-13 21:36:06Z INFO ConfigurationStore] IsConfigured()
[2020-07-13 21:36:06Z INFO ConfigurationStore] IsConfigured: False
[2020-07-13 21:36:06Z INFO ConfigurationManager] Is configured: False
[2020-07-13 21:36:06Z INFO CommandSettings] Arg 'url': 'https://github.com/falldamagestudio/UE4-GHA-Engine'
[2020-07-13 21:36:06Z INFO CommandSettings] Remove url from Arg dictionary.
[2020-07-13 21:36:06Z INFO CommandSettings] Arg 'token': '***'
[2020-07-13 21:36:06Z INFO CommandSettings] Remove token from Arg dictionary.
[2020-07-13 21:36:08Z INFO ConfigurationManager] Http response code: OK from 'POST https://api.github.com/actions/runner-registration'
[2020-07-13 21:36:08Z INFO ConfigurationManager] cred retrieved via GitHub auth
[2020-07-13 21:36:08Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[2020-07-13 21:36:08Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:08Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:08Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:08Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:08Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:09Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:10Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:10Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:10Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:10Z INFO ConfigurationManager] Test Connection complete.
[2020-07-13 21:36:10Z INFO HostContext] Well known directory 'Bin': 'C:\A\bin'
[2020-07-13 21:36:10Z INFO HostContext] Well known directory 'Root': 'C:\A'
[2020-07-13 21:36:10Z INFO HostContext] Well known config file 'RSACredentials': 'C:\A\.credentials_rsaparams'
[2020-07-13 21:36:10Z INFO RSAEncryptedFileKeyManager] Creating new RSA key using 2048-bit key length
[2020-07-13 21:36:10Z INFO RSAEncryptedFileKeyManager] Successfully saved RSA key parameters to file C:\A\.credentials_rsaparams
[2020-07-13 21:36:10Z INFO ConfigurationManager] Found a private pool with id {1} and name {2} 1, Default
[2020-07-13 21:36:10Z INFO CommandSettings] Arg 'name': 'engine_build_agent'
[2020-07-13 21:36:10Z INFO CommandSettings] Remove name from Arg dictionary.
[2020-07-13 21:36:10Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:10Z INFO CommandSettings] Arg 'labels': 'engine_build_agent'
[2020-07-13 21:36:10Z INFO CommandSettings] Remove labels from Arg dictionary.
[2020-07-13 21:36:10Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:11Z INFO ConfigurationManager] Agent server url resolve by server: 'https://pipelines.actions.githubusercontent.com/serviceHosts/cf20d806-2b42-4d9d-a239-83eff0807664/'.
[2020-07-13 21:36:11Z INFO ConfigurationStore] Saving OAuth credential @ C:\A\.credentials
[2020-07-13 21:36:11Z INFO ConfigurationStore] Credentials Saved.
[2020-07-13 21:36:11Z INFO ConfigurationStore] HasCredentials()
[2020-07-13 21:36:11Z INFO ConfigurationStore] stored True
[2020-07-13 21:36:11Z INFO CredentialManager] GetCredentialProvider
[2020-07-13 21:36:11Z INFO CredentialManager] Creating type OAuth
[2020-07-13 21:36:11Z INFO CredentialManager] Creating credential type: OAuth
[2020-07-13 21:36:11Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file C:\A\.credentials_rsaparams
[2020-07-13 21:36:11Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[2020-07-13 21:36:11Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:11Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2020-07-13 21:36:11Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:11Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2020-07-13 21:36:11Z WARN GitHubActionsService] Authentication failed with status code 401.
WWW-Authenticate: Bearer
Strict-Transport-Security: max-age=2592000
X-TFS-ProcessId: bbde113b-9a9d-4721-aa00-98abb06dc849
ActivityId: 3961f0b8-a449-4b0d-8770-5eac33ffa9b3
X-TFS-Session: fd9a3090-eac9-4fd2-afff-438bf7e92485
X-VSS-E2EID: 00bf5d0f-626a-4827-a516-756ea3cde3ff
X-VSS-SenderDeploymentId: a07ab14e-025a-39c3-8d53-788cd7ce207f
X-TFS-ServiceError: The+user+%27System%3aPublicAccess%3baaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa%27+is+not+authorized+to+access+this+resource.
X-VSS-S2STargetService: 0000005A-0000-8888-8000-000000000000/visualstudio.com
X-MSEdge-Ref: Ref A: C4DB781BF4594A2D8A3044784C3AB8DF Ref B: LON21EDGE0210 Ref C: 2020-07-13T21:36:11Z
Date: Mon, 13 Jul 2020 21:36:10 GMT

[2020-07-13 21:36:11Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file C:\A\.credentials_rsaparams
[2020-07-13 21:36:11Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file C:\A\.credentials_rsaparams
[2020-07-13 21:36:11Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2020-07-13 21:36:11Z INFO CommandSettings] Arg 'work': ''
[2020-07-13 21:36:11Z INFO CommandSettings] Flag 'unattended': 'True'
[2020-07-13 21:36:11Z INFO PromptManager] ReadValue
[2020-07-13 21:36:11Z INFO ConfigurationStore] Saving runner settings.
[2020-07-13 21:36:11Z INFO ConfigurationStore] Settings Saved.
[2020-07-13 21:36:11Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:11Z INFO Terminal] WRITE LINE:
[2020-07-13 21:36:11Z INFO CommandSettings] Flag 'runasservice': 'False'
[2020-07-13 21:36:11Z INFO CommandSettings] Flag 'unattended': 'True'
[2020-07-13 21:36:11Z INFO PromptManager] ReadValue

During subsequent service start-ups, the runner will find the .runner configuration, load that - but again, receive TFS error 401 a second later.

Runner Version and Platform

Runner: 2.263.0 OS: Microsoft Windows 10.0.17763

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:4
  • Comments:9

github_iconTop GitHub Comments

3reactions
San-Jeevancommented, Jan 21, 2022

I found a solution/workaround: this happens because your Runner is older version than what is the latest. So github starts blocking requests with older version that is sent in the http useragent.

So i cloned github runn repo. then ran the following commadn from /src folder

.\dev.cmd layout

it will produce files in /_layout folder. Replace the runner agent binaries on ur server with these

so we basically just manually upgraded the self-hosted agent by recompiling the source

3reactions
dogweathercommented, Sep 15, 2020

I’m seeing this with a brand new setup - no runners yet, brand new raspberry pi: Getting the exact same log output, although the local runner outputs Connected to GitHub … Listening for Jobs. However, on the GitHub side, the web UI shows no runners configured.

Read more comments on GitHub >

github_iconTop Results From Across the Web

401 (Unauthorized) nuget pipeline restore from ...
I select feeds in my nuget.config and it almost works but I get an error authenticating to a feed that is in our...
Read more >
git-tfs returns "(401) unauthorized" error when cloning a ...
I've added collection name to the server's url and it worked. The correct command was: git tfs clone ...
Read more >
Run the agent with a self-signed certificate - Azure Pipelines
Learn how to run the build and release agent with a self-signed certificate for Azure Pipelines and Team Foundation Server (TFS)
Read more >
Unable to consume TFS 2015 API. Getting 401 unauthrozed ...
Your TFS indicates (see WWW-Authenticate Header) that it supports Bearer, Negotiate and NTLM; but not Basic. I would try: Remove client.DefaultRequestHeaders.
Read more >
Troubleshoot self-hosted runner
The following are errors you could encounter using container runner. Container fails to start due to disk space. The task remains in the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Disable process cleanup

Next page

Auto updater wants to update to a pre-release version