Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[CLI]: Network error (ReadTimeout) when manually overriding `WANDB_TAGS`

See original GitHub issue

Describe the bug

The client errors out with Network error (ReadTimeout) when manually overriding WANDB_TAGS.

import os
import wandb
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
os.environ["WANDB_TAGS"] = "v1.0.0b2-15-g9ca363c,pr-308"
wandb.init(project="cleanrl")
for i in range(10):
    wandb.log({"x": data[i]})

(cleanrl-py3.9) ➜  cleanrl git:(autotag) ✗ python t.py
wandb: Currently logged in as: costa-huang. Use `wandb login --relogin` to force relogin
Problem at: /Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/t.py 8 <module>
wandb: ERROR Error communicating with wandb process
wandb: ERROR For more info see: https://docs.wandb.ai/library/init#init-start-error
Traceback (most recent call last):
  File "/Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/t.py", line 8, in <module>
    wandb.init(project="cleanrl")
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 1076, in init
    run = wi.init()
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 718, in init
    raise UsageError(error_message)
wandb.errors.UsageError: Error communicating with wandb process
For more info see: https://docs.wandb.ai/library/init#init-start-error
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: Network error (ReadTimeout), entering retry loop.

However, if I don’t override the environment variable through os.environ and run WANDB_TAG=v1.0.0b2-15-g9ca363c,pr-308 python t.py with the following file, it runs fine…

import os
import wandb
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# os.environ["WANDB_TAGS"] = "v1.0.0b2-15-g9ca363c,pr-308"
wandb.init(project="cleanrl")
for i in range(10):
    wandb.log({"x": data[i]})

Additional Files

No response

Environment

WandB version: 0.13.4

OS: ubuntu

Python version: 3.9

Versions of relevant libraries:

Additional Context

2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_setup.py:_flush():68] Configure stats pid to 2550
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_setup.py:_flush():68] Loading settings from /Users/costahuang/.config/wandb/settings
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_setup.py:_flush():68] Loading settings from /Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/wandb/settings
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_setup.py:_flush():68] Loading settings from environment variables: {'_require_service': 'True', 'run_tags': ['v1.0.0b2-15-g9ca363c', 'pr-308']}
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_setup.py:_flush():68] Inferring run settings from compute environment: {'program_relpath': 't.py', 'program': '/Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/t.py'}
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_init.py:_log_setup():476] Logging user logs to /Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/wandb/run-20221101_153653-3ih88ljl/logs/debug.log
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_init.py:_log_setup():477] Logging internal logs to /Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/wandb/run-20221101_153653-3ih88ljl/logs/debug-internal.log
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_init.py:init():516] calling init triggers
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_init.py:init():519] wandb.init called with sweep_config: {}
config: {}
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_init.py:init():569] starting backend
2022-11-01 15:36:53,349 INFO    MainThread:2550 [wandb_init.py:init():573] setting up manager
2022-11-01 15:36:53,352 INFO    MainThread:2550 [backend.py:_multiprocessing_setup():102] multiprocessing start_methods=spawn,fork,forkserver, using: spawn
2022-11-01 15:36:53,353 INFO    MainThread:2550 [wandb_init.py:init():580] backend started and connected
2022-11-01 15:36:53,355 INFO    MainThread:2550 [wandb_init.py:init():658] updated telemetry
2022-11-01 15:36:53,383 INFO    MainThread:2550 [wandb_init.py:init():693] communicating run to backend with 60 second timeout
2022-11-01 15:37:52,920 WARNING MainThread:2550 [wandb_init.py:init():1099] interrupted
Traceback (most recent call last):
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 1078, in init
    run = wi.init()
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 697, in init
    result = handle.wait(
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/lib/mailbox.py", line 261, in wait
    found = self._slot._get_and_clear(timeout=wait_timeout)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/lib/mailbox.py", line 118, in _get_and_clear
    if self._wait(timeout=timeout):
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/lib/mailbox.py", line 114, in _wait
    return self._event.wait(timeout=timeout)
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/threading.py", line 574, in wait
    signaled = self._cond.wait(timeout)
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/threading.py", line 316, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

2022-11-01 15:36:53,366 INFO    StreamThr :2557 [internal.py:wandb_internal():88] W&B internal server running at pid: 2557, started at: 2022-11-01 15:36:53.364536
2022-11-01 15:36:53,368 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: status
2022-11-01 15:36:53,369 DEBUG   SenderThread:2557 [sender.py:send_request():317] send_request: status
2022-11-01 15:36:53,370 DEBUG   SenderThread:2557 [sender.py:send():303] send: header
2022-11-01 15:36:53,370 INFO    WriterThread:2557 [datastore.py:open_for_write():75] open: /Users/costahuang/go/src/github.com/vwxyzjn/cleanrl/wandb/run-20221101_153653-3ih88ljl/run-3ih88ljl.wandb
2022-11-01 15:36:53,384 DEBUG   SenderThread:2557 [sender.py:send():303] send: run
2022-11-01 15:36:58,408 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:03,426 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:08,445 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:13,471 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:18,485 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:23,506 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:27,320 ERROR   SenderThread:2557 [retry.py:__call__():154] Retry attempt failed:
Traceback (most recent call last):
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/Users/costahuang/.pyenv/versions/3.9.9/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/lib/retry.py", line 113, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/sdk/internal/internal_api.py", line 213, in execute
    return self.client.execute(*args, **kwargs)  # type: ignore
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/transport/requests.py", line 38, in execute
    request = requests.post(self.url, **post_args)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/costahuang/Library/Caches/pypoetry/virtualenvs/cleanrl--brdzvln-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 578, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.wandb.ai', port=443): Read timed out. (read timeout=10)
2022-11-01 15:37:28,532 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:33,546 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:38,574 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:43,597 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:48,619 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:37:53,931 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: poll_exit
2022-11-01 15:37:58,957 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:38:03,980 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive
2022-11-01 15:38:08,994 DEBUG   HandlerThread:2557 [handler.py:handle_request():138] handle_request: keepalive

Issue Analytics

  • State:closed
  • Created a year ago
  • Reactions:1
  • Comments:10 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
exalate-issue-sync[bot]commented, Nov 1, 2022

Leslie commented: Thank you for your patience! Our engineers were able to push a fix for this. There’s still currently an issue regarding batch moving runs, but for the most part this issue has been resolved.

0reactions
vwxyzjncommented, Nov 2, 2022

Seems fixed on my end. Thanks!

Read more comments on GitHub >

github_iconTop Results From Across the Web

wandb: Network error (ReadTimeout)，How to solve it?
I use the command wandb sync $LOG_DIRECTORY , but I meet the error: wandb: Network error (ReadTimeout) . I have checked ping wandb.ai...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[Q]Whether the searched parameters support parameter nesting

Next page

[CLI]: Repord Editing API does not create a valid report