S3FileSystem.exists throwing inaccurate FILENOTFOUND

#53

I am still having the same issue when I am trying to create dask dataframe form s3. I did make sure anon=False when I create the interface object. It is a sporadic problem. Not sure why. I have troubleshot my code with boto3 s3 client and s3 object. They both agreed the file is on S3 but s3fs exists is returning ‘Filenotfound’.

client  = boto3.client('s3')
def check(s3, bucket, key):
   try:
       r=s3.head_object(Bucket=bucket, Key=key)
   except ClientError as e:
       return int(e.response['Error']['Code']) != 404 #e.response['Error'] #
   return True

check(client, bucket, key))
#########################################################

resource = boto3.resource('s3')

s3resource.Object(bucket,key).get()
except botocore.exceptions.ClientError as e:
                         if e.response['Error']['Code'] == "404":
                               return False
                         else: raise
else:
        return True

[scuddalo]

We are running into this exact issue in our project too, @martindurant. We would greatly appreciate any updates/workarounds.

[abdullin]

@TomAugspurger we were checking for the existence of the file in a dense folder. While stepping through the codebase with the debugger, it leads to exists -> info -> ls, where the result of ls is exactly 1000 files. The folder itself contains 4226 files and the file in question is the last one (given the sorting by name).

The fact that ls returns only first 1000 files is a known issue. We are always using a custom paginator to list directory contents.