Checking for file existence.

As we use s3 for storage of generated assets, and only generate them again if not existing yet, we need to implement a check for existence.

How about a from smart_open import exists? It would be great if this library could also allow easy checking for existence, tailored for it’s multiple backends.

from smart_open import exists  # like os.exists
if not exists("s3://my_key:my_secret@my_server:my_port@my_bucket/some_file.mp4"):
  data = long_running_procedure("some_file.mp4")   # generate assets
  from smart_open import open
  with open("s3://my_key:my_secret@my_server:my_port@my_bucket/some_file.mp4", "wb") as f:
    f.write(data)
  # end with
  print("Generated from scratch.")
else:
  print("Already uploaded.")
# end if

[mpenkov]

You don’t need to download it. Smart_open streams its data. If you open a stream, but don’t actually read from it, nothing gets downloaded.

Try:

try:
    with smart_open.open('s3://bucket/key/does/not/exist.txt'):
        file_exists = True
except ValueError:
    file_exists = False

The ValueError gets raised here in case you want to do more precise checking.

[luckydonald]

Using boto3, which also is internally used by smart_open, we used the listing with Prefix=file_path to check for existence, as it doesn’t download the file, and we have the s3:ListBucket permission set anyway.

def exists(bucket_name, file_path):
    s3 = boto3.resource(
        's3',
        endpoint_url=f'{proto}://{host}:{port}',
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
    )
    # Bucket name we wanna use
    bucket = s3.Bucket(bucket_name)
    # list files matching a filter: the path
    objs = list(bucket.objects.filter(Prefix=file_path))
    return len(objs) > 0
# end def 

Source: Stackoverflow, also includes other solutions with probably don’t require s3:ListBucket.