s3 remote: handle invalid auth information properly

Bug Report

If S3 auth token is invalid, DVC throws unexpected error

Description

I have MFA (2FA) setup. When temporary token is expired, and I try to push/pull, I’m getting an unexpected error from DVC:

ERROR: unexpected error - [Errno 22] The provided token is malformed or otherwise invalid.: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.

Quick fix is to update the token with AWS STS tools.

Reproduce

Use MFA Wait token expire Try to push something to S3

Expected

It’s not an unexpected error. It should be handled properly with some useful message on what is wrong. The provided token is malformed or otherwise invalid. is not useful.

Environment information

Output of dvc doctor:

DVC version: 2.5.4+6c8673
---------------------------------
Platform: Python 3.8.9 on macOS-10.15.6-x86_64-i386-64bit
Supports:
	azure (adlfs = 2021.7.0, knack = 0.8.2, azure-identity = 1.5.0),
	gdrive (pydrive2 = 1.8.2),
	gs (gcsfs = 2021.7.0),
	hdfs (pyarrow = 4.0.0),
	webhdfs (hdfs = 2.5.8),
	http (requests = 2.25.1),
	https (requests = 2.25.1),
	s3 (s3fs = 2021.7.0, boto3 = 1.17.49),
	ssh (paramiko = 2.7.2),
	oss (ossfs = 2021.7.3),
	webdav (webdav4 = 0.8.2),
	webdavs (webdav4 = 0.8.2)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

Stack trace with `-v`

2021-07-15 08:34:10,784 ERROR: unexpected error - [Errno 22] The provided token is malformed or otherwise invalid.: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 246, in _call_s3
    out = await method(**additional_kwargs)
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ivan/Projects/dvc/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/Users/ivan/Projects/dvc/dvc/command/base.py", line 50, in do_run
    return self.run()
  File "/Users/ivan/Projects/dvc/dvc/command/data_sync.py", line 57, in run
    processed_files_count = self.repo.push(
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 51, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/ivan/Projects/dvc/dvc/repo/push.py", line 44, in push
    pushed += self.cloud.push(objs, jobs, remote=remote)
  File "/Users/ivan/Projects/dvc/dvc/data_cloud.py", line 79, in push
    return remote_obj.push(
  File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 57, in wrapper
    return f(obj, *args, **kwargs)
  File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 488, in push
    ret = self._process(
  File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 345, in _process
    dir_status, file_status, dir_contents = self._status(
  File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 189, in _status
    remote_exists.update(self._indexed_dir_hashes(dir_objs))
  File "/Users/ivan/Projects/dvc/dvc/remote/base.py", line 252, in _indexed_dir_hashes
    self.odb.list_hashes_exists(indexed_dirs)
  File "/Users/ivan/Projects/dvc/dvc/objects/db/base.py", line 419, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/ivan/Projects/dvc/dvc/objects/db/base.py", line 410, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/Users/ivan/Projects/dvc/dvc/fs/fsspec_wrapper.py", line 83, in exists
    return self.fs.exists(self._with_bucket(path_info))
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/fsspec/asyn.py", line 88, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/fsspec/asyn.py", line 69, in sync
    raise result[0]
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 804, in _exists
    await self._info(path, bucket, key, version_id=version_id)
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 1064, in _info
    out = await self._simple_info(path)
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 977, in _simple_info
    out = await self._call_s3(
  File "/Users/ivan/Projects/example-get-started/.env/lib/python3.8/site-packages/s3fs/core.py", line 265, in _call_s3
    raise translate_boto_error(err)
OSError: [Errno 22] The provided token is malformed or otherwise invalid.

[shcheklein]

Agreed, this is a generally well-known issue with non-dvc exceptions. Def could reconsider among other things.

Yep, to me it looks like a bug I think. Unexpected error to me means that something actually critical error (bug) happened.

Though we could just include a link for https://dvc.org/doc/user-guide/troubleshooting#troubleshooting

Good point. Do one generic “Auth failed <link>” vs being more specific in the message for me that would depend on how bad those libraries that we use. If it’s hard to get that information, then generic message with a link is enough I think.

instead of rephrasing in our code base like in your suggestion

It’s not what I was suggesting to be precise 😃 I was only stating that it’s not a very good message from the user perspective and can be improved. How? Ideally in the libraries that we use if they take that part of auth, to the extent that is possible. I also don’t like parsing exception texts with regexps and wrapping them up. I would though go even that path if it’s needed (depends on the availability of other options, etc, etc).

[yathomasi]

I also felt like we could have better error messages.

$ dvc checkout   
WARNING: No file hash info found for '../../iterative.ai/static/uploads'. It won't be created.                                                            
ERROR: Checkout failed for following targets:                                                                                                             ../../iterative.ai/static/uploads
Is your cache up to date?
<https://error.dvc.org/missing-files>

It’s again due to the AWS MFA session token expired and couldn’t checkout. Also on dvc pull:

It would be better if these messages had some hint of authentication error. I don’t see any on both of them.