Capella Bucket Download issue

Bucket Issue Hi there, I’m trying to download Capella’s Open data from: https://www.stacindex.org/catalogs/capella-space-open-data#/BrVjRL9LGRjg8ipuaKLhdHr1kD4sbvyd8QTYGwbMTTtkR4fzPd4yeiApxyrQkiq/LGziSG3zruu42mUQkBqrXkRPLdBZyfQ9WBqmakLW8v6Q2jtwGXLNgDPJyeVfFR6XkZHa8qL5E75MA6d8vzhnxgNLQVUuVFNrq?t=2 But I got a “NoSuchKey” error from the “bucket_name” = “data”

I tried unsuccesfull workarounds… If someone got an idea, it’s very welcome.

Thanks, and thanks a lot EOdag devs! Ari

Code To Reproduce

from eodag import setup_logging
setup_logging(verbose=3)

from eodag.api.core import EODataAccessGateway

# Create an EODAG custom STAC provider
dag = EODataAccessGateway()

# Add the custom STAC provider + output + login an password
catalog_path = "https://capella-open-data.s3.us-west-2.amazonaws.com/stac/capella-open-data-by-product-type/capella-open-data-slc/collection.json"
# catalog_path = 'https://capella-open-data.s3.us-west-2.amazonaws.com/stac/catalog.json'
base_uri = "https://capella-open-data.s3.us-west-2.amazonaws.com"
outputs_prefix = r"D:\Radar\data\EODAG"
aws_access_key_id = "change_me"
aws_secret_access_key = "change_me"


dag.update_providers_config("""
capella:
    search:
        type: StaticStacSearch
        api_endpoint: %s
    products:
        GENERIC_PRODUCT_TYPE:
            productType: '{productType}'
    download:
        type: AwsDownload
        base_uri: %s
        flatten_top_dirs: True
        outputs_prefix: %s
    auth:
        type: AwsAuth
        credentials:
            aws_access_key_id: %s
            aws_secret_access_key: %s
""" % (catalog_path, base_uri, outputs_prefix, aws_access_key_id, aws_secret_access_key ))

# # Set the custom STAC provider as preferred
dag.set_preferred_provider("capella")

# Query every product from inside the catalog
# all_products, _ = dag.search()

capella_prod, _ = dag.search(start="2021-12-01", end="2021-12-15", **{"sar:product_type": "SLC", "sensorMode": "spotlight"})  # , locations=dict(country="BRA"), start="2021-01-01", end="2020-05-30", items_per_page=50)

# [(key, asset["href"]) for key, asset in capella_prod[0].assets.items()]

paths = dag.download_all(capella_prod)

TroubleShoot

2022-10-07 16:47:56,830 eodag.config                     [INFO    ] (config           ) Loading user configuration from: C:\Users\ajeannin\.config\eodag\eodag.yml
2022-10-07 16:47:56,923 eodag.core                       [INFO    ] (core             ) usgs: provider needing auth for search has been pruned because no crendentials could be found
2022-10-07 16:47:56,923 eodag.core                       [INFO    ] (core             ) aws_eos: provider needing auth for search has been pruned because no crendentials could be found
2022-10-07 16:47:57,010 eodag.core                       [DEBUG   ] (core             ) Opening product types index in C:\Users\ajeannin\.config\eodag\.index
2022-10-07 16:47:57,020 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from C:\Users\ajeannin\.config\eodag\locations.yml
2022-10-07 16:47:57,022 eodag.config                     [INFO    ] (config           ) capella: unknown provider found in user conf, trying to use provided configuration
2022-10-07 16:47:57,110 eodag.core                       [INFO    ] (core             ) No product type could be guessed with provided arguments
2022-10-07 16:47:57,686 eodag.core                       [INFO    ] (core             ) Searching product type 'None' on provider: capella
2022-10-07 16:47:57,686 eodag.core                       [DEBUG   ] (core             ) Using plugin class for search: StaticStacSearch
2022-10-07 16:47:58,682 eodag.utils.stac_reader          [DEBUG   ] (stac_reader      ) Fetching 125 items
2022-10-07 16:47:58,683 eodag.utils.stac_reader          [DEBUG   ] (stac_reader      ) read_local_json is not the right STAC opener
2022-10-07 16:48:00,570 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting genric provider product type definition parameters for None
2022-10-07 16:48:00,571 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Building the query string that will be used for search
2022-10-07 16:48:00,571 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Retrieving queryable metadata from metadata_mapping
2022-10-07 16:48:00,609 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page URL could not be collected
2022-10-07 16:48:00,626 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page Query-object could not be collected
2022-10-07 16:48:00,643 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page merge could not be collected
2022-10-07 16:48:00,643 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Adapting 125 plugin results to eodag product representation
2022-10-07 16:48:02,877 eodag.plugins.crunch.filter_date [DEBUG   ] (filter_date      ) Start filtering by date
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_date [INFO    ] (filter_date      ) Finished filtering products. 4 resulting products
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_overlap [DEBUG   ] (filter_overlap   ) Start filtering for overlapping products
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_overlap [WARNING ] (filter_overlap   ) geometry not found in cruncher arguments, filtering disabled.
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [DEBUG   ] (filter_property  ) Start filtering for products matching operator.eq(product.properties['sar:product_type'], SLC)
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [INFO    ] (filter_property  ) Finished filtering products. 4 resulting products
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [DEBUG   ] (filter_property  ) Start filtering for products matching operator.eq(product.properties['sensorMode'], spotlight)
2022-10-07 16:48:02,893 eodag.plugins.crunch.filter_property [INFO    ] (filter_property  ) Finished filtering products. 2 resulting products
2022-10-07 16:48:02,898 eodag.core                       [INFO    ] (core             ) Found 2 result(s) on provider 'capella'
2022-10-07 16:48:02,898 eodag.core                       [INFO    ] (core             ) Downloading 2 products
Downloaded products:   0%|          | 0/2 [00:00<?, ?product/s]
0.00B [00:00, ?B/s]
CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646: 0.00B [00:00, ?B/s]2022-10-07 16:48:02,900 eodag.plugins.download.base      [INFO    ] (base             ) Download url: https://capella-open-data.s3.us-west-2.amazonaws.com/collections/capella-open-data-spotlight/items/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646
2022-10-07 16:48:03,847 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:03,847 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646.tif
2022-10-07 16:48:04,678 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:04,678 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646/CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646_extended.json
2022-10-07 16:48:05,496 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:05,496 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656_preview.tif
2022-10-07 16:48:06,301 eodag.plugins.download.aws       [WARNING ] (aws              ) Unexpected error: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
2022-10-07 16:48:06,301 eodag.plugins.download.aws       [WARNING ] (aws              ) Skipping data//2021/12/14/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656/CAPELLA_C03_SP_GEO_HH_20211214232633_20211214232656_thumb.png
2022-10-07 16:48:06,302 eodag.plugins.download.base      [ERROR   ] (base             ) Stopped because of credentials problems with provider capella
Traceback (most recent call last):
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\base.py", line 456, in download_all
    product.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\api\product\_product.py", line 288, in download
    fs_path = self.downloader.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\aws.py", line 339, in download
    raise AuthenticationError(", ".join(auth_error_messages))
eodag.utils.exceptions.AuthenticationError: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
Downloaded products:   0%|          | 0/2 [00:03<?, ?product/s]
Traceback (most recent call last):
  File "D:\PythonCode\radar_dev\radar_dev\utils.py", line 48, in <module>
    paths = dag.download_all(capella_prod)
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\api\core.py", line 1288, in download_all
    paths = download_plugin.download_all(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\aws.py", line 920, in download_all
    return super(AwsDownload, self).download_all(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\base.py", line 456, in download_all
    product.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\api\product\_product.py", line 288, in download
    fs_path = self.downloader.download(
  File "C:\Users\ajeannin\Anaconda3\envs\radar_dev\lib\site-packages\eodag\plugins\download\aws.py", line 339, in download
    raise AuthenticationError(", ".join(auth_error_messages))
eodag.utils.exceptions.AuthenticationError: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
CAPELLA_C03_SP_SLC_HH_20211214232643_20211214232646: 0.00B [00:03, ?B/s]

Process finished with exit code 1

Environment:

• Conda version: 22.9.0
• Python version: 3.7
• EODAG version: eodag version tells me

(radar_dev) λ eodag version                                                                                                                                                                                                                                                                          
eodag (Earth Observation Data Access Gateway): version 0.0.0

But I think I got the 2.5.0 version as it is the latest on Conda env (should I open a new issue ?)

[remi-braun]

Sure

[remi-braun]

I managed to make it work 🥳 I add the same error again and again :

with open(url) as f:
OSError: [Errno 22] Invalid argument: 'https://capella-open-data.s3.us-west-2.amazonaws.com/stac/capella-open-data-by-datetime/capella-open-data-2021/capella-open-data-2021-4/capella-open-data-2021-4-10/CAPELLA_C03_SP_SLC_HH_20210410140302_20210410140304/CAPELLA_C03_SP_SLC_HH_20210410140302_20210410140304.json'

It was trying to read a remote json with the read_local_json (in stac_reader). As expected it failed, but (I think) with an unhandled exeception: an OSError, instead of a FileNotFoundError.

This behavior breaks the function, which is supposed to work or return a STACOpenerError. Th updated version is:

    @staticmethod
    def read_local_json(url, as_json):
        """Read JSON local file"""
        try:
            with open(url) as f:
                if as_json:
                    return json.load(f)
                else:
                    return f.read()
        except (FileNotFoundError, OSError):
            logger.debug("read_local_json is not the right STAC opener")
            raise STACOpenerError