Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Use Scaleway bucket for image reporting

See original GitHub issue

Hi, We have configured clearml to use a scaleway bucket for storage. The configuration is as follows since scalway is compatible with s3.

				api {
					...
				}
				sdk {
				   ...
					aws {
						s3 {
							region: "fr-par"
							key: "SCALEWAY-ACCESS-KEY"
							secret: "SCALEWAY-SECRET-KEY"
							use_credentials_chain: false

							credentials: [
								{
									 host: "s3.fr-par.scw.cloud:443"
									 bucket: "clearml-files"
									 key: "SCALEWAY-ACCESS-KEY"
									 secret: "SCALEWAY-SECRET-KEY"
									 secure: true
								}
							]
						}
						boto3 {
							pool_connections: 512
							max_multipart_concurrency: 16
						}
					}
					...
				}

For dataset the usage is as follows and it works fine. dataset.upload(output_url = "s3://s3.fr-par.scw.cloud:443/clearml-files/", verbose=True) However for the logger we get the following exception. task = Task.init(project_name="examples", task_name="New Scalar reporting") logger = task.get_logger() logger.set_default_upload_destination(uri="s3://s3.fr-par.scw.cloud:443/clearml-files/")

		2022-05-11 09:29:50,889 - clearml.storage - ERROR - Could not connect to the endpoint URL: "https://clearml-files.s3.fr-par.amazonaws.com/clearml.storage.helper-upload_test"
		Traceback (most recent call last):
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
			conn = connection.create_connection(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/util/connection.py", line 72, in create_connection
			for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
		  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
			for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
		socket.gaierror: [Errno -2] Name or service not known

		During handling of the above exception, another exception occurred:

		Traceback (most recent call last):
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/httpsession.py", line 439, in send
			urllib_response = conn.urlopen(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
			retries = retries.increment(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/util/retry.py", line 525, in increment
			raise six.reraise(type(error), error, _stacktrace)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
			raise value
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
			httplib_response = self._make_request(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
			self._validate_conn(conn)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
			conn.connect()
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
			conn = self._new_conn()
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
			raise NewConnectionError(
		urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f84d77884c0>: Failed to establish a new connection: [Errno -2] Name or service not known

		During handling of the above exception, another exception occurred:

		Traceback (most recent call last):
		  File "limit.py", line 46, in <module>
			main()
		  File "limit.py", line 32, in main
			logger.set_default_upload_destination(uri="s3://s3.fr-par.scw.cloud:443/clearml-files/")
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/clearml/logger.py", line 1052, in set_default_upload_destination
			uri = storage.verify_upload(folder_uri=uri)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/clearml/storage/helper.py", line 561, in verify_upload
			_Boto3Driver._test_bucket_config(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/clearml/storage/helper.py", line 1552, in _test_bucket_config
			bucket.put_object(Key=filename, Body=six.b(json.dumps(data)))
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/boto3/resources/factory.py", line 580, in do_action
			response = action(self, *args, **kwargs)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/boto3/resources/action.py", line 88, in __call__
			response = getattr(parent.meta.client, operation_name)(*args, **params)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/client.py", line 415, in _api_call
			return self._make_api_call(operation_name, kwargs)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/client.py", line 731, in _make_api_call
			http, parsed_response = self._make_request(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/client.py", line 751, in _make_request
			return self._endpoint.make_request(operation_model, request_dict)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/endpoint.py", line 107, in make_request
			return self._send_request(request_dict, operation_model)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/endpoint.py", line 183, in _send_request
			while self._needs_retry(attempts, operation_model, request_dict,
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/endpoint.py", line 305, in _needs_retry
			responses = self._event_emitter.emit(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/hooks.py", line 358, in emit
			return self._emitter.emit(aliased_event_name, **kwargs)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/hooks.py", line 229, in emit
			return self._emit(event_name, kwargs)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/hooks.py", line 212, in _emit
			response = handler(**kwargs)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/retryhandler.py", line 194, in __call__
			if self._checker(**checker_kwargs):
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/retryhandler.py", line 267, in __call__
			should_retry = self._should_retry(attempt_number, response,
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/retryhandler.py", line 294, in _should_retry
			return self._checker(attempt_number, response, caught_exception)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/retryhandler.py", line 333, in __call__
			checker_response = checker(attempt_number, response,
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/retryhandler.py", line 233, in __call__
			return self._check_caught_exception(
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/retryhandler.py", line 376, in _check_caught_exception
			raise caught_exception
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/endpoint.py", line 249, in _do_get_response
			http_response = self._send(request)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/endpoint.py", line 321, in _send
			return self.http_session.send(request)
		  File "/mnt/c/Users/zandolsi/projects/clearml-client/venv-1/lib/python3.8/site-packages/botocore/httpsession.py", line 468, in send
			raise EndpointConnectionError(endpoint_url=request.url, error=e)
		botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://clearml-files.s3.fr-par.amazonaws.com/clearml.storage.helper-upload_test"

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:11 (10 by maintainers)

github_iconTop GitHub Comments

2reactions
jkhenningcommented, May 17, 2022

I seem, so this is a bug in _test_bucket_config() specifically for this use-case. I wonder how come we didn’t encounter this before, as this test is used for every host (minio for example).

Thanks for the PR 🙏 - we’ll take a look 🙂

0reactions
AndolsiZiedcommented, May 17, 2022

I wonder how come we didn't encounter this before, as this test is used for every host (minio for example). I think because this is a special case: logger + bucket + host compatible with S3. I think this may be since you're calling Task.set_default_upload_destination() Using Task to set the destination url does not have effect. We should set the logger destination url.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to enable and use the bucket website feature - Scaleway
Click the name of the bucket you wish to host your website on. · Scroll down to the bottom of the Bucket Settings...
Read more >
Object Storage | Scaleway Documentation
It allows you to store any kind of objects (documents, images, videos, etc.) ... Metrics such as storage and bucket usage help you...
Read more >
How to upload files into a bucket | Scaleway Documentation
Click Object Storage in the Storage section of the side menu. · Click the bucket name into which you want to upload your...
Read more >
Object Storage - Quickstart | Scaleway Documentation
It allows you to store any kind of objects (documents, images, videos, etc.) ... Each bucket represents a space where objects are stored....
Read more >
Adding objects to a bucket with POST object - Scaleway
The POST object operation adds an object to a specified bucket using HTML forms. POST is an alternate form of PUT that enables...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Deadlocks happening with version `1.4.0`

Next page

Datasets cache folder not shared between users