Artifact successfully uploaded but cannot be downloaded by client.

• Weights and Biases version: 0.10.0,0.9.7
• Python version: 3.8.5
• Operating System: Arch Linux x86_64 (5.8.8-arch1-1)

Description

Creating an artifact on one machine (wandb version 0.10.0) over an SSH session with VPN connection allowed the artifact to be uploaded to the cloud, and a training run could be started on that machine from the newly synced artifact. Trying to start a training run on a different machineusing the same artifact version failed (wandb version 0.10.0), as the artifact could not be fetched. All artifact files are present in the web client, and can be downloaded from the artifacts tab. In addition, the wandb CLI client cannot download the artifact, and downgrading the CLI client on the second machine to 0.9.7 does not fix the issue.

Details

Starting a training run with wandb version 0.10.0 yields the following stacktrace:

...
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/apis/public.py", line 2281, in download
    pool.map(partial(self._download_file, dirpath=dirpath), manifest.entries)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/apis/public.py", line 2314, in _download_file
    return self.get_path(name).download(dirpath)
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/apis/public.py", line 2236, in download
    cache_path = storage_policy.load_file(self, name, manifest.entries[name])
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/sdk/wandb_artifacts.py", line 409, in load_file
    response = self._session.get(
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/requests-2.24.0-py3.8.egg/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/requests-2.24.0-py3.8.egg/requests/sessions.py", line 516, in request
    prep = self.prepare_request(req)
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/requests-2.24.0-py3.8.egg/requests/sessions.py", line 449, in prepare_request
    p.prepare(
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/requests-2.24.0-py3.8.egg/requests/models.py", line 314, in prepare
    self.prepare_url(url, params)
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/requests-2.24.0-py3.8.egg/requests/models.py", line 388, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'None': No schema supplied. Perhaps you meant http://None?

After downgrading the CLI client to 0.9.7, a different, but perhaps more useful error occurs:

...
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/apis/public.py", line 2245, in download
    pool.map(partial(self._download_file, dirpath=dirpath), manifest.entries)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/apis/public.py", line 2281, in _download_file
    return artifact_entry.download(dirpath)
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/apis/public.py", line 2200, in download
    cache_path = storage_policy.load_file(self, name, manifest.entries[name])
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/wandb/artifacts.py", line 536, in load_file
    response.raise_for_status()
  File "~/.virtualenvs/python-latest/lib/python3.8/site-packages/requests-2.24.0-py3.8.egg/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://storage.googleapis.com/wandb-artifacts-prod/alexmirrington/wandb_artifacts/objs/4bde5***************************?Expires=1600263353&GoogleAccessId=gorilla-cloud-storage%40wandb-production.iam.gserviceaccount.com&Signature=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

Trying to download the artifact from the CLI yields similar stacktraces.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.91. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[annirudh]

This issue has been resolved in CLI 0.10.1. Upgrading to the latest CLI should allow you to download artifacts successfully.