I would like to configure Sleet to use Minio as a local S3 compliant object store

Summary

I would like to configure Sleet to use Minio as a local S3 compliant object store but I cannot make it work. This issue may be caused by my ignorance. I will pull source and debug.

Steps to Reproduce

NOTE: Performed on Fedora 31. I manually built everything in an effort to exercise available Sleet and Minio documentation.

• Employed:
  • https://play.min.io:9000 as S3 service endpoint; accessKeyId=Q3AM3UQ867SPQQA43P2F;secretAccessKey=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
  • mcr.microsoft.com/dotnet/core/sdk:3.1-alpine3.11 Docker image/OCI image

NOTE: This is a public sandbox and publicly documented access key and secret key

1. Build a local dotNET Core 3.1 OCI container image (I use Podman)
2. Run dotNET Core container interactively
3. Install Sleet. In container, dotnet tool install -g sleet -v q and export PATH="$PATH:/root/.dotnet/tools"
4. Create Sleet config. In container, sleet createconfig --s3
5. Modify and save sleet.config. In container, vi sleet.config, change content as specified below, and save changes.

{                                                                                                                                                                                                                                                                               
  "username": "devops",                                                                                                                                                                                                                                                      
  "useremail": "[e-mail]",                                                                                                                                                                                                                                            
  "sources": [                                                                                                                                                                                                                                                                  
    {                                                                                                                                                                                                                                                                           
      "name": "myMinioS3Feed",                                                                                                                                                                                                                                                  
      "type": "s3",                                                                                                                                                                                                                                                             
      "path": "https://play.min.io:9000/minio/my-bucket-feed/",                                                                                                                                                                                                                        
      "bucketName": "my-bucket-feed",                                                                                                                                                                                                                                           
      "serviceURL": "https://play.min.io:9000",                                                                                                                                                                                                                                 
      "accessKeyId": "Q3AM3UQ867SPQQA43P2F",                                                                                                                                                                                                                                    
      "secretAccessKey": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"                                                                                                                                                                         
    }                                                                                                                                                                                                                                                                           
  ]                                                                                                                                                                                                                                                                             
}                                    

1. Set SLEET_DEBUG=1. In container, export SLEET_DEBUG=1
2. Initialize feed. In container, sleet init --config sleet.json --source myMinioS3Feed

Expected Results

The feed is successfully initialized.

Actual Results

Exception is thrown.

Amazon.S3.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records.
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Sleet.AmazonS3FileSystemAbstraction.FileExistsAsync(IAmazonS3 client, String bucketName, String key, CancellationToken token) in D:\a\1\s\src\SleetLib\FileSystem\AmazonS3FileSystemAbstraction.cs:line 64
   at Sleet.AmazonS3File.CopyFromSource(ILogger log, CancellationToken token) in D:\a\1\s\src\SleetLib\FileSystem\AmazonS3File.cs:line 40
   at Sleet.FileBase.EnsureFile(ILogger log, CancellationToken token) in D:\a\1\s\src\SleetLib\FileSystem\FileBase.cs:line 289
Failed to sync 'https://play.min.io:9000/my-bucket-feed/sleet.settings.json'. Retrying.

Conclusion

Again, I don’t know if this is my issue or if a code change must be made to allow Sleet to use Minio as an S3 object store. I also reproduced this issue using a local Minio OCI container. I am not a C# developer so I don’t feel confident that I can make a satisfactory code change if this is what is necessary. I will provide user documentation to Sleet and Minio projects if this configuration is possible.

[ryancraig]

Looks like the majority of the changes will be in SleetLib/FileSystem/FileSystemFactory.cs. I’m adding a new type of minio. There’s obviously alot of commonality between s3 and minio. This new type will ensure that I don’t break s3 for Amazon or any other S3 compliant object stores that the current logic handles. It can be refactored later.

[emgarten]

Thanks for the update @ryancraig, I’ll re-open this for tracking a fix.