Bad Request when authenticating to ClearML server

Summary

I’m getting a bad request error (400) when trying to connect to a self hosted server via the python API. I’ve traced it down to the Content-Length header. If I set it to 0 or remove it the request is successful, but otherwise it is set to 2 and the request fails. I believe the root cause is this line:

https://github.com/allegroai/clearml/blob/fd2d6c6f5d46cad3e406e88eeb4d805455b5b3d8/clearml/backend_api/session/session.py#L710

Replacing the default argument with None like below ensures that the Content-Length header is set correctly and works for the request in question, but seems to break other things (I’ll keep looking).

data = {"expiration_sec": exp} if exp else None

Unfortunately the error is not repeatable on the ClearML community server (I’m not sure why!) so hard to provide a MWE.

What I did

Set the environment variables CLEARML_API_ACCESS_KEY, CLEARML_API_SECRET_KEY, CLEARML_API_HOST, CLEARML_WEB_HOST, CLEARML_FILES_HOST to connect to my self-hosted server (which is running v1.1.0 as 1.1.1 is not available in the GCP Images yet).

Attempt to connect via:

from clearml import Task
project = "ProjectName"
Task.get_tasks(project_name=project))

And get this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/task.py", line 847, in get_tasks
    return cls.__get_tasks(task_ids=task_ids, project_name=project_name, tags=tags,
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/task.py", line 3535, in __get_tasks
    for task in cls._query_tasks(project_name=project_name, task_name=task_name, **kwargs)]
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/task.py", line 3553, in _query_tasks
    cls._get_default_session(),
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/backend_interface/base.py", line 101, in _get_default_session
    InterfaceBase._default_session = Session(
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/backend_api/session/session.py", line 201, in __init__
    self.refresh_token()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/backend_api/session/token_manager.py", line 112, in refresh_token
    self._set_token(self._do_refresh_token(self.__token, exp=self.req_token_expiration_sec))
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/backend_api/session/session.py", line 739, in _do_refresh_token
    six.reraise(*sys.exc_info())
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/six.py", line 719, in reraise
    raise value
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/experiment-test-qPFpYaQg-py3.9/lib/python3.9/site-packages/clearml/backend_api/session/session.py", line 725, in _do_refresh_token
    raise LoginError(
clearml.backend_api.session.session.LoginError: Failed getting token (error 400 from https://api.cleaml.mydomain.com): Bad Request

Setup

clearml==1.1.4
$ python --version
Python 3.9.9

[mmiller-max]

it seems like a relatively big change, one I wouldn’t want to push right now…

Makes sense yep.

would you like to PR it?

Yes I’ll get one ready!

Thanks for your quick response with this, really appreciated 👍

[jkhenning]

Is it a change that can be tested and demonstrated to be OK and therefore would be likely to make it into a release, or do you think it’s too big a change?

Since this has to do with how load-balancers handle these requests and how resends are handled on the client side (by Python’s requests package etc.), it seems like a relatively big change, one I wouldn’t want to push right now…

Perhaps an env var is a nice solution to this too, at least as a development feature

That’s a really good idea! Since my main concern is production use and stability - having this kind of override that can be enabled in development (or even in production, if you so choose, under the current “disclaimer”) is a very good solution - would you like to PR it? 🙂