Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

dvc push: unable to validate self-signed certificate from S3 service

See original GitHub issue

Bug Report

Description

Attempting dvc push to s3 remote which uses SSL termination. Getting certificate validation error since we are using self-signed certificates

Reproduce

  1. dvc init
  2. modify remote to s3 endpoint with https scheme
  3. copy certificate bundle from remote se to local directory
  4. update AWS_CA_BUNDLE environment variable to point to local certificate bundle
  5. dvc add
  6. dvc push

Expected

data sent to s3 remote

Environment information

MacOs Homebrew DVC installation

Output of dvc doctor: DVC version: 2.0.18 (brew)

Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs Cache types: reflink, hardlink, symlink Cache directory: apfs on /dev/disk1s1 Caches: local Remotes: s3 Workspace directory: apfs on /dev/disk1s1 Repo: dvc, git

$ dvc doctor

Additional Information (if any): I placed the output of dvc push -v below along with relevant environment variables

johnathan@Johnathans-MacBook-Pro repo % dvc push -v
2021-05-05 13:56:31,064 DEBUG: Check for update is enabled.
+------------------------------------------+
|                                          |
|     Update available 2.0.18 -> 2.1.0     |
|          Run `brew upgrade dvc`          |
|                                          |
+------------------------------------------+

2021-05-05 13:56:31,329 DEBUG: Preparing to upload data to 's3://internal-test-files'
2021-05-05 13:56:31,330 DEBUG: Preparing to collect status from s3://internal-test-files
2021-05-05 13:56:31,330 DEBUG: Collecting information from local cache...
2021-05-05 13:56:31,332 DEBUG: Collecting information from remote cache...                                                                                                            
2021-05-05 13:56:31,332 DEBUG: Querying 1 hashes via object_exists
2021-05-05 13:56:43,907 ERROR: unexpected error - SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 314, in send
    urllib_response = conn.urlopen(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/main.py", line 55, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/command/data_sync.py", line 56, in run
    processed_files_count = self.repo.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/repo/push.py", line 41, in push
    return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/data_cloud.py", line 66, in push
    return remote.push(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 56, in wrapper
    return f(obj, *args, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 466, in push
    ret = self._process(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 323, in _process
    dir_status, file_status, dir_contents = self._status(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 169, in _status
    remote_exists.update(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/remote/base.py", line 242, in _indexed_dir_hashes
    dir_exists.update(self.odb.list_hashes_exists(dir_md5s - dir_exists))
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 338, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/objects/db/base.py", line 329, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 233, in exists
    return self.isfile(path_info) or self.isdir(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 263, in isfile
    return path_info.path in self._list_paths(path_info)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/dvc/fs/s3.py", line 284, in _list_paths
    for obj_summary in obj_summaries:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/boto3/resources/collection.py", line 166, in pages
    for page in pages:
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 662, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/usr/local/Cellar/dvc/2.0.18/libexec/lib/python3.9/site-packages/botocore/httpsession.py", line 341, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://remote:443/internal-test-files?prefix=a5%2Fb8b049b6da9ad14c1a6b77ad0f49b1.dir&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)
------------------------------------------------------------
2021-05-05 13:56:44,478 DEBUG: Version info for developers:
DVC version: 2.0.18 (brew)
---------------------------------
Platform: Python 3.9.4 on macOS-10.15.7-x86_64-i386-64bit
Supports: azure, gdrive, gs, http, https, s3, ssh, oss, webdav, webdavs
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-05-05 13:56:44,479 DEBUG: Analytics is enabled.
2021-05-05 13:56:44,717 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'
2021-05-05 13:56:44,722 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/9m/xx24sxdx2fvb9s5q5rt1jxy00000gn/T/tmpo1qv_wjs']'

johnathan@Johnathans-MacBook-Pro repo % printenv
USER=johnathan
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
...
TERM=xterm-256color
AWS_CA_BUNDLE=</path/to/bundle.crt>
REQUESTS_CA_BUNDLE=REQUESTS_CA_BUNDLE=</path/to/bundle.crt>
_=/usr/bin/printenv
johnathan@Johnathans-MacBook-Pro repo % cat cat /Users/johnathan/.bash_profile             
cat: cat: No such file or directory
export PATH="/usr/local/bin:$PATH"
export PATH="/usr/local/opt/python@3.7/bin:$PATH"
export GODEBUG="x509ignoreCN=0"
export AWS_CA_BUNDLE="</path/to/bundle.crt>"
export REQUESTS_CA_BUNDLE="</path/to/bundle.crt>"
export SSL_CERT_FILE="</path/to/bundle.crt>"

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:18 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
skshetrycommented, Jul 8, 2021

@justusschock, it’ll take a few hours to get to the homebrew, for now, it’s only available through pip.

2reactions
skshetrycommented, Jul 8, 2021

@justusschock, we’ll be releasing 2.5.2 with the fix in a few moments.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting | Data Version Control - DVC
The most common cause is changes pushed to Git without the corresponding data being uploaded to the DVC remote. Make sure to dvc...
Read more >
Boto [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify ...
Probably your bucket name contains a dot, that's why ssl certificate verification fails. This is quite a frequent problem, see this github ...
Read more >
Troubleshooting SSL - GitLab Docs
SSL certificate problem: unable to get local issuer certificate. This error indicates the client cannot get the root CA. To fix this, you...
Read more >
How to fix error “[SSL: CERTIFICATE_ VERIFY_FAILED ... - Blog
fatal error: SSL validation failed for ... have seen that AWS hits a regional endpoint over SSL when you access any AWS service...
Read more >
Administrator's Guide for ArcSight Platform 21.1 - Micro Focus
The only warranties for Micro Focus products and services are set ... Configuring Logger as a Transformation Hub Consumer using SSL/TLS.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

License confirmation for `dvc get` and `dvc import`

Next page

`exp run`: `--set-params` forces YAML 1.2 (breaks code that uses 1.1)