Support setting a different endpoint for COPY TO / FROM for DNS-style S3 bucket adresses

Use case: I want to be able to export data from CrateDB using COPY TO / FROM with (AWS) S3-compatible technologies like Minio, so that I don’t rely on AWS for my storage setup.

Feature description: With AWS SDK the endpoint defaults to <bucket>.s3.amazonaws.com. In order to use my Minio S3 bucket, I am able to enter my own endpoint in the URI for COPY FROM / TO commands. The ability to use AWS S3 is not affected.

---

Origin: https://community.crate.io/t/export-table-using-copy-to-command-to-alternate-s3-endpoint/246/4

S3 Client Helper: https://github.com/crate/crate/blob/b8f5bfc1ae1d34b0426f2a01217fde663505225b/server/src/main/java/io/crate/external/S3ClientHelper.java

[emres]

Would be nice to rely on S3 compatible storage. Google Cloud Platform storage has S3 compatibility, there’s Digital Ocean, Linode, and Vultr as well that offer S3 compatible object storage as well.

As-is, I am using s3fs as a bandaid with file://.

In addition to the above, there are also other S3-compatible storage systems such as Nutanix Object Storage which I’m using in one of my clients (so far it works as expected with utilities such as s5cmd and Cyberduck.)

Therefore it’ll be very good if CrateDB provides enough flexibility with different S3-compatible object storage systems.

[jeeminso]

Closing this issue, all necessary changes are now merged.

The scope of this issue was to delegate the provided s3 endpoints to AWS SDK. There still could be problems between SDK and the s3 compatible storage, ex) #12094. So, please try this feature with the latest nightly build.

The latest nightly build can be found at: https://cdn.crate.io/downloads/releases/nightly/ The steps to install and run: https://crate.io/docs/crate/tutorials/en/latest/install.html#ad-hoc-unix-macos-windows Also have a few sample copy from/to statements, here and here.

Thank you!