[CLI]: sync offline-run artifacts get duplicated on web-server

Describe the bug

When syncing an offline run to the web via the sync command, the artifacts get duplicated.

In my code I have a loop logging each time a table artifact:

wandb.log({'table': table = wandb.Table(...})

When I use an online run I get the wanted output of a table with versions (each representing a loop iteration) as follows: In this example, the loop performed the python code above 8 times creating versions 0-7.

The problem starts when I train it with WANDB_MODE=offline and then run the following command:

wandb sync offline-run-<run-id>

The first time it will work correctly and will upload all versions up to the current loop status (lets say versions 0-2), when the loop will continue, if I’ll run the command again, instead of uploading versions 3-7, it will upload 0-2 again and start the counting from version 3, ending up with versions 0-10, where versions 0-2 are the same as versions 3-5.

Additional Files

No response

Environment

WandB version: 0.12.21

OS: CentOS Stream 8

Python version: 3.9.12

Versions of relevant libraries:

Additional Context

No response

[thanos-wandb]

Hi @urielsinger, I tried to replicate your issue but I wasn’t able to run to the same issue as you did.

I used this command wandb sync --sync-all to sync my artifacts. Can you provide me with some more information why you used the wand sync offline-run-<run-id> statement relative to --sync-all?

[ramit-wandb]

Hey @yuvalkirstain,

Understood, we have already created an internal ticket to track this issue and resolve it, and I absolutely agree that this is something we should resolve. I’ll reach out to you once some movement has been made here.

Thanks, Ramit