wandb.init fails when a parent directory has '.git' file

Describe the bug wandb.init fails with the error below.

git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)

It seems to have trouble getting settings.program_relpath. Maybe it is related to recent OS upgrade to Ubuntu 20.04

To Reproduce

# wandb_test.py
import wandb
wandb.init('gpt-3')

python wandb_test.py

Expected behavior Runs normally. This actually runs without any error in other server with Ubuntu 18.04

Screenshots

╰─ python wandb_test.py
Traceback (most recent call last):
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 478, in init
    wi.setup(kwargs)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 68, in setup
    self._wl = wandb_setup._setup()
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 221, in _setup
    wl = _WandbSetup(settings=settings, _warn=_warn)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 213, in __init__
    _WandbSetup._instance = _WandbSetup__WandbSetup(settings=settings)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 94, in __init__
    self._settings_setup(settings, self._early_logger)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 119, in _settings_setup
    s._infer_run_settings_from_env()
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_settings.py", line 710, in _infer_run_settings_from_env
    program_relpath = _get_program_relpath_from_gitrepo(program)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_settings.py", line 137, in _get_program_relpath_from_gitrepo
    root = repo.root
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/lib/git.py", line 47, in root
    return self.repo.git.rev_parse("--show-toplevel")
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/git/cmd.py", line 542, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/git/cmd.py", line 1005, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/git/cmd.py", line 822, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git rev-parse --show-toplevel
  stderr: 'fatal: not a git repository: /../../.git/modules/PACKAGES/HOSTFIX'
wandb: ERROR Abnormal program exit
Traceback (most recent call last):
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 478, in init
    wi.setup(kwargs)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 68, in setup
    self._wl = wandb_setup._setup()
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 221, in _setup
    wl = _WandbSetup(settings=settings, _warn=_warn)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 213, in __init__
    _WandbSetup._instance = _WandbSetup__WandbSetup(settings=settings)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 94, in __init__
    self._settings_setup(settings, self._early_logger)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_setup.py", line 119, in _settings_setup
    s._infer_run_settings_from_env()
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_settings.py", line 710, in _infer_run_settings_from_env
    program_relpath = _get_program_relpath_from_gitrepo(program)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_settings.py", line 137, in _get_program_relpath_from_gitrepo
    root = repo.root
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/lib/git.py", line 47, in root
    return self.repo.git.rev_parse("--show-toplevel")
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/git/cmd.py", line 542, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/git/cmd.py", line 1005, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/git/cmd.py", line 822, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git rev-parse --show-toplevel
  stderr: 'fatal: not a git repository: /../../.git/modules/PACKAGES/HOSTFIX'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "wandb_test.py", line 5, in <module>
    wandb.init(project='gpt-3')
  File "/ssd-playpen/home/jmincho/anaconda3/envs/t5/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 515, in init
    six.raise_from(Exception("problem"), error_seen)
  File "<string>", line 3, in raise_from
Exception: problem

Operating System

Problem

• OS: Ubuntu 20.04.1 LTS
• Wandb Version 0.10.11
• git version: 2.25.1

Works normally in

• OS: Ubuntu 18.04.5 LTS
• Wandb Version 0.10.11
• git version: 2.17.1

Additional context I don’t have .git/ in the server. I can run wandb normally with the exactly same code in another server. I’m using a shared server in a university. Our server admin recently upgraded the server including its ubuntu version (18.04 -> 20.04) and I cannot run wandb since then.

[vanpelt]

Thank you for your infinite patience! The next release of wandb will gracefully handle invalid git repo’s so you won’t need to modify all these settings. Should be out in early September 🥳 .

[vanpelt]

Sorry @j-min I gave you the wrong setting before. The branch I’ve opend above will fix this once and for all, but the following should work. 5th times a charm? 🤞

wandb.init(project="gpt-seq2seq", settings=wandb.Settings(program_relpath="run_gpt.py", disable_git=True, save_code=False))