Error when using wandb sweep & agent

• Weights and Biases version: 0.8.13
• Python version: 3.7.4
• Operating System: Linux

Description

I tried to use sweep, and write following code (search.yaml) the define hyperparameter search space

program: main.py
method: bayes
metric:
    name: performance
    goal: maximize
parameters:
    lr:
        distribution: log_uniform
        min: 0.001
        max: 0.3
    decay_epoch:
        min: 5
        max: 20
    decay_ratio:
        min: 0.3
        max: 0.1
    epochs:
        min: 20
        max: 40
    scheduler:
        value: StepLR
early_terminate:
    type: envelope

and executed sweep command for this file as follows:

$ wandb sweep search.yaml

The output of this command is:

wandb Creating sweep from: search.yaml
agnet Create sweep with ID: 1391qnn5
Sweep URL: https://app.wandb.ai/{my_username}/{my_project_name}/sweeps/1391qnn5

Finally, I executed agent as follows:

$ wandb agent 1391qnn5

But this failed and I can see error messages below:

Starting wandb agent 🕵️
2019-10-28 12:44:56,469 - wandb.wandb_agent - INFO - Running runs: []
wandb: ERROR Error while calling W&B API: invalid character 'N' looking for beginning of value
wandb: ERROR Error while calling W&B API: invalid character 'N' looking for beginning of value
wandb: ERROR Error while calling W&B API: invalid character 'N' looking for beginning of value
wandb: Network error (HTTPError), entering retry loop. See /path/to/my/workdir/wandb/debug.log for full traceback.

and wandb/debug.log contains:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/wandb/retry.py", line 95, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/wandb/apis/internal.py", line 109, in execute
    six.reraise(*sys.exc_info())
  File "/opt/conda/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/opt/conda/lib/python3.7/site-packages/wandb/apis/internal.py", line 103, in execute
    return self.client.execute(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/gql/client.py", line 50, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/gql/client.py", line 58, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/gql/transport/requests.py", line 38, in execute
    request.raise_for_status()
  File "/opt/conda/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.wandb.ai/graphql
2019-10-28 12:48:07,446 DEBUG   MainThread:400335 [connectionpool.py:_new_conn():813] Starting new HTTPS connection (1): api.wandb.ai:443
2019-10-28 12:48:08,021 DEBUG   MainThread:400335 [connectionpool.py:_make_request():393] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 500 136
2019-10-28 12:48:08,023 ERROR   MainThread:400335 [internal.py:execute():106] 500 response executing GraphQL.
2019-10-28 12:48:08,024 ERROR   MainThread:400335 [internal.py:execute():107] {"errors":[{"message":"invalid character 'N' looking for beginning of value","path":["agentHeartbeat"]}],"data":{"agentHeartbeat":null}}
2019-10-28 12:48:17,798 DEBUG   MainThread:400335 [connectionpool.py:_new_conn():813] Starting new HTTPS connection (1): api.wandb.ai:443
2019-10-28 12:48:18,432 DEBUG   MainThread:400335 [connectionpool.py:_make_request():393] https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 500 136
2019-10-28 12:48:18,434 ERROR   MainThread:400335 [internal.py:execute():106] 500 response executing GraphQL.
2019-10-28 12:48:18,435 ERROR   MainThread:400335 [internal.py:execute():107] {"errors":[{"message":"invalid character 'N' looking for beginning of value","path":["agentHeartbeat"]}],"data":{"agentHeartbeat":null}}

How can I fix this error?

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.71. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[dannygoldstein]

Hi Mason,

The way your log uniform parameters are currently specified, you are asking for values between e^0.0001 and e^1000 (see docs https://docs.wandb.ai/guides/sweeps/configuration#distribution). The latter causes an overflow on ourbackend which is rendered as inf, which causes a json parsing error.

We can file a bug for the overflow parsing, but I think for now you want to adjust your loguniform limits to be between ln(0.0001) and ln(1000)

On Thu, Feb 3, 2022 at 2:41 PM Mason Hargrave @.***> wrote:

I still have this issue and don’t exactly know what the fix is.

Python Version: 3.9.7 OS: macOS Big Sur Version 11.6 Weights and Biases Version: 0.12.10

`sweep_config = { “name” : “rbf-svc-num-feats”, “method” : “random”, “parameters” : { “C” : { “distribution”: “log_uniform”, “min”: 0.0001, “max”: 1000. }, “gamma” : { “min”: 0.0001, “max”: 1. }, “num_features” : { “min”: 1, “max”: 83 } } }

sweep_id = wandb.sweep(sweep_config)`

And the error is:

wandb: ERROR Error while calling W&B API: invalid character ‘I’ looking for beginning of value (<Response [500]>)

— Reply to this email directly, view it on GitHub https://github.com/wandb/client/issues/642#issuecomment-1029469015, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVEFYDU3DUTVE5QTPHRQ53UZMACPANCNFSM4JFZXNQQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>