Logs not syncing when wandb-history.jsonl is quite large.

When I run wandb sync wandb/my_run the history metrics are not uploaded to the wandboard. It just stays like that till I kill the command with Ctrl+C.

The folder wandb/my_run contains the following contents:

2.4K  config.yaml
402K  model.ckpt
9.6M  output.log
748K  wandb-events.jsonl
62M  wandb-history.jsonl
16B  wandb-metadata.json
61K  wandb-summary.json

The output I get after running wandb sync wandb/my_run:

wandb: Syncing wandb/my_run to:
wandb: exp_name https://app.wandb.ai/usr_name/project_name/runs/hash
wandb: Uploading history metrics

and after that it does not progress.

Does wandb sync have limits in upload sizes.

[vanpelt]

Great, I’ve got meeting all afternoon, but I’ll look at this tonight and get back to you.

[vanpelt]

Bummer, I’ve confirmed the “fix” I had for uploading large files was added in 0.9.5. The relevant code that’s causing issue is here:

https://github.com/wandb/client/blob/v0.9.7/wandb/apis/file_stream.py#L251

You could modify your local installation of wandb to see what’s going on there. Another thing to try would be just hard-coding the lines_per_request to something like 2 or 10. The root cause is likely trying to send too much data to the backend at once. Adding a print statement to the while loop would let you see how much progress it’s able to make. You can find the local path to your wandb install by opening a python console and running:

import wandb
print(wandb.__path__)