Inaccessible file path for data catalog fails silently

When trying to load a catalog yml file hosted S3, missing credentials will silently lead to an empty catalog. So the user executes,

In [1]: from intake import Catalog

In [2]: cat = Catalog("s3://foo/bar.yml")

In [3]: cat.yaml()
Out[3]: "sources:\n  bar:\n    args:\n      path: s3://foo/bar.yml\n    description: ''\n    driver: yaml_files_cat\n    metadata: {}\n"

and it’s not obvious why there are no sources.

For the case of s3 specifically, a quick check looks like:

def verify_credentials(cat_uri):
    """
    FIXME copied from data-layer-api
    The intake catalog will appear empty when trying to read a catalog yml
    file on s3 if the credentials are not supplied correctly.
    """
    if "s3:" not in cat_uri:
        return True

    fs = s3fs.S3FileSystem()
    cat_exists = fs.exists(cat_uri)
    if not cat_exists:
        err_msg = "Catalog file not found at {}. Check AWS credentials.".format(cat_uri)
        raise EnvironmentError(err_msg)

There are probably issues related to this as well if trying to open a local yml file in a directory that doesn’t exist.

[seibert]

I think there might be an unnecessary linkage between the handling of a catalog directory and a catalog file. A catalog directory might be initially empty and have new files appear during operation (like conda installing a data package while notebook is running). A catalog file, local or remote, should exist when the catalog object for a file is created, otherwise simple typos in filenames are very hard to catch.

[martindurant]

Indeed, that could be the way to do it, @stsievert , have different defaults for allow_empty for the two file Catalog classes, or raise an exception in YAMLFileCatalog._load if nothing is found (and not change anything elsewhere). The latter is easier to implement.