dvc import broken authentication

We seem to no longer be able to import a remote dvc resources from a repository that requires authentication. This worked at some point prior to 2.10. This worked previously.

The output of the import is as follows:

2022-06-14 12:01:24,323 ERROR: failed to import 'model.hdf5' from 'https://github.build.company.com/org/Artifacts.git'. - Failed to clone repo 'https://github.build.company.com/org/Artifacts.git' to '/var/tmp/pbs.1540626.scinfra2/tmpc9sse3s9dvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/backend/dulwich/__init__.py", line 196, in clone
    repo = clone_from()
  File "/usr/local/lib/python3.8/dist-packages/dulwich/porcelain.py", line 443, in clone
    return client.clone(
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 747, in clone
    result = self.fetch(path, target, progress=progress, depth=depth)
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 824, in fetch
    result = self.fetch_pack(
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2079, in fetch_pack
    refs, server_capabilities, url = self._discover_references(
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 1938, in _discover_references
    resp, read = self._http_request(url, headers)
  File "/usr/local/lib/python3.8/dist-packages/dulwich/client.py", line 2219, in _http_request
    raise HTTPUnauthorized(resp.getheader("WWW-Authenticate"), url)
dulwich.client.HTTPUnauthorized: No valid credentials provided

The above exception was the direct cause of the following exception:

There was some previous commit appears to fix a similar issue, which was part of the 2.11 release: issue #7670 however, updating to 2.11 did not fix the issue.

DVC version: 2.11.0 (pip)
---------------------------------
Platform: Python 3.8.0 on Linux-3.10.0-1160.66.1.el7.x86_64-x86_64-with-glibc2.27
Supports:
	webhdfs (fsspec = 2022.5.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2022.5.0, boto3 = 1.21.21)
Cache types: hardlink, symlink
Cache directory: nfs on LEB1MLNAS.hpc.company.com:/leb1mlnas_projects
Caches: local
Remotes: None
Workspace directory: nfs on LEB1MLNAS.hpc.company.com:/leb1mlnas_projects
Repo: dvc, git

[dtrifiro]

Solved in 2.25.0

[d-miketa]

@dtrifiro FWIW I tried installing dulwich from your feature branch using

pip install git+https://github.com/dtrifiro/dulwich.git@feature/credential-helper

and I’m unfortunately still running into the same issue:

❯ dvc pull data/dataset.dvc
Everything is up to date.                                                                                                                                       
ERROR: failed to pull data from the cloud - Failed to clone repo '[e-mail]:organisation/datasets.git' to '/var/folders/b1/d587c0_d6hb6ndr4mv24h_gc0000gq/T/tmpu75hs20xdvc-clone'

Downgrading to 2.9 fixes the issue. I’m on macOS.