Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

AuthenticationError when k8 agent runs get_runs_in_queue

See original GitHub issue

Description

I’m running a k8 agent I get the following error. It happens when a processes is collected my Lazarus:

[2020-09-05 04:11:04,158] ERROR - agent | [{'path': ['get_runs_in_queue'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]

Full print out


 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2020-09-04 15:24:02,360] INFO - agent | Starting KubernetesAgent with labels []
[2020-09-04 15:24:02,361] INFO - agent | Agent documentation can be found at https://docs.prefect.io/orchestration/
[2020-09-04 15:24:02,361] INFO - agent | Agent connecting to the Prefect API at https://api.prefect.io
[2020-09-04 15:24:02,542] INFO - agent | Waiting for flow runs...
[2020-09-05 00:00:07,758] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 00:00:08,126] INFO - agent | Deploying flow run 2e60d4df-78a3-4512-a817-4b9f69216baa
[2020-09-05 00:10:36,858] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 00:10:37,067] INFO - agent | Deploying flow run c5d20308-2a14-4258-b556-ab9e6fddf461
[2020-09-05 00:21:06,450] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 00:21:06,683] INFO - agent | Deploying flow run 2e60d4df-78a3-4512-a817-4b9f69216baa
[2020-09-05 00:31:14,955] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 00:31:15,172] INFO - agent | Deploying flow run c5d20308-2a14-4258-b556-ab9e6fddf461
[2020-09-05 00:41:43,349] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 00:41:43,669] INFO - agent | Deploying flow run 2e60d4df-78a3-4512-a817-4b9f69216baa
[2020-09-05 00:52:11,450] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 00:52:11,684] INFO - agent | Deploying flow run c5d20308-2a14-4258-b556-ab9e6fddf461
[2020-09-05 01:02:19,367] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 01:02:19,605] INFO - agent | Deploying flow run 2e60d4df-78a3-4512-a817-4b9f69216baa
[2020-09-05 01:12:48,156] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 01:12:48,367] INFO - agent | Deploying flow run c5d20308-2a14-4258-b556-ab9e6fddf461
[2020-09-05 04:11:04,158] ERROR - agent | [{'path': ['get_runs_in_queue'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]
[2020-09-05 12:00:07,760] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 12:00:08,061] INFO - agent | Deploying flow run 40d59a2c-8aab-4707-b7e2-6e4832ec7827
[2020-09-05 12:10:26,260] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 12:10:26,561] INFO - agent | Deploying flow run 40d59a2c-8aab-4707-b7e2-6e4832ec7827
[2020-09-05 12:20:54,843] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 12:20:55,064] INFO - agent | Deploying flow run 40d59a2c-8aab-4707-b7e2-6e4832ec7827
[2020-09-05 12:31:23,954] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-05 12:31:24,175] INFO - agent | Deploying flow run 40d59a2c-8aab-4707-b7e2-6e4832ec7827
[2020-09-06 00:00:02,262] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-06 00:00:02,484] INFO - agent | Deploying flow run 2519dfa1-addb-432d-9c29-52f86cf9febc
[2020-09-06 00:01:10,565] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-06 00:01:10,769] INFO - agent | Deploying flow run d29c9b49-c2e4-4478-946c-456a85786f74
[2020-09-06 12:00:09,455] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-06 12:00:09,669] INFO - agent | Deploying flow run 50959947-fdc0-4c9e-a853-3b978628152d
[2020-09-07 00:00:02,362] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-07 00:00:02,567] INFO - agent | Deploying flow run 61059a94-f63c-4018-9ff2-410b0748705a
[2020-09-07 00:01:00,358] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-09-07 00:01:00,590] INFO - agent | Deploying flow run 8444271e-7598-4a6d-a730-85b391af29b7

Expected Behavior

I would not expect a AuthenticationError

Reproduction

Environment

prefect==0.13.5 `python 3.7 docker image

Issue Analytics

State:
Created 3 years ago
Comments:9 (4 by maintainers)

Top GitHub Comments

1reaction

jcristcommented, Sep 8, 2020

Thanks for the issue report @thomasfrederikhoeck, this appears to be a permissions bug in prefect cloud (the get_runs_in_queue query succeeds earlier, so it’s not a token issue). I’ve created an issue in our internal tracker to fix this, and will report back here once resolved.

0reactions

thomasfrederikhoeckcommented, Sep 17, 2020

No, I’m not calling it from my flow. The log appeared in the Cloud logs.

Top Results From Across the Web

Attempt to authenticate an agent fails with an Kerberos ... - IBM

An attempt to authenticate an IBM Rational Performance Tester (RPT) remote agent results in an authentication error. Symptom. Steps to reproduce: Run the...

Troubleshooting the GitLab agent for Kubernetes

This error occurs when you configured wss as kas-address on the agent side, but the agent server is not available at wss ....

Nessus Agent Status “Link status: authentication error”

When running the "nessuscli agent status" command the error “Link status: authentication error” is encountered.

why I got the error "authentication error" when running ...

When I executing datastax-agent with version 5.2. ... authentication error, please carefully check your Auth settings, retrying soon.

Kubernetes Components

An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod. The kubelet takes...