wandb hangs experiment (10 min+): Internal Server Error for url: https://api.wandb.ai/graphql

wandb --version && python --version && uname

• Weights and Biases version: 0.8.35
• Python version: 3.7
• Operating System: Linux

Description

For a few days, I noticed experiments hanging on wandb logging. Sometimes I even saw crashes.

So far, downgrading to 0.8.33 seems to help. Will report if the problem arises again.

What I Did

2020-05-05 09:25:36,056 ERROR   Thread-18 :22373 [internal.py:execute():113] 500 response executing GraphQL.
2020-05-05 09:25:36,057 ERROR   Thread-18 :22373 [internal.py:execute():114] {"error":"Error 1040: Too many connections"}

2020-05-05 09:25:36,058 ERROR   Thread-18 :22373 [retry.py:__call__():108] Retry attempt failed:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/wandb/retry.py", line 95, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/wandb/apis/internal.py", line 116, in execute
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/wandb/apis/internal.py", line 110, in execute
    return self.client.execute(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/gql/transport/requests.py", line 39, in execute
    request.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/graphql
2020-05-05 09:25:39,928 INFO    Thread-3  :22373 [run_manager.py:_on_file_modified():691] file/dir modified: <redacted>/run-20200505_072213-camelyon-16384-full-correct-loss/wandb-metadata.json
2020-05-05 09:25:40,883 ERROR   Thread-18 :22373 [internal.py:execute():113] 500 response executing GraphQL.
2020-05-05 09:25:40,884 ERROR   Thread-18 :22373 [internal.py:execute():114] {"error":"Error 1040: Too many connections"}

2020-05-05 09:25:50,866 ERROR   Thread-18 :22373 [internal.py:execute():113] 500 response executing GraphQL.
2020-05-05 09:25:50,867 ERROR   Thread-18 :22373 [internal.py:execute():114] {"error":"Error 1040: Too many connections"}

2020-05-05 09:25:51,143 WARNING Thread-7  :22373 [util.py:request_with_retry():614] requests_with_retry encountered retryable exception: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/files/hanspinckaers/camelyon/camelyon-16384-full-correct-loss/file_stream. args: ('https://api.wandb.ai/files/hanspinckaers/camelyon/camelyon-16384-full-correct-loss/file_stream',), kwargs: {'json': {'files': {'output.log':

[vanpelt]

@richardrl we had an outage last night that caused these errors. Everything should be functioning properly now.

[sadra-barikbin]

Hi, I’m experiencing this issue in Google Colab environment. To reproduce:

#bash
wandb login  --cloud "API_KEY"

then

#python
api = wandb.Api()
runs = api.runs(f'{entity}/{project_name}')
runs[0]

output:

Retry attempt failed:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/lib/retry.py", line 102, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/wandb/apis/public.py", line 205, in execute
    return self._client.execute(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/wandb/vendor/gql-0.2.0/wandb_gql/transport/requests.py", line 39, in execute
    request.raise_for_status()
  File "/usr/local/lib/python3.7/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/graphql
wandb: Network error (HTTPError), entering retry loop.```