ReactiveStreams API Adapter for S3 ByteStream

Describe the feature

Currently, the S3Client upload and get support ByteStream, when using it in a Reactive application, it is not so easy to convert a reactive streams type to ByteStream.

Is your Feature Request related to a problem?

No

Proposed Solution

I would like use the ReactiveStreams compatible Publisher<ByteBuffer>, esp, in Spring WebFlux/Reactor, use the specific Flux and DataBuffer aka Flux<DataBuffer>, also consider RxJava3/SmallRye Munity

Describe alternative solutions or features you’ve considered

• The existing ByteStream to implements Publisher
• Kotlin Couroutines, support Flow as data type.

Acknowledge

• I may be able to implement this feature request

AWS Kotlin SDK version used

0.15.2-beta

Platform (JVM/JS/Native)

JVM

Operating System and version

Windows 10

[ianbotsf]

It should be possible to adapt a ByteStream into a Flow. For example:

const val bufferSize = 4096

suspend fun main() {
    val s3 = S3Client.fromEnvironment { }

    val req = GetObjectRequest {
        bucket = "some-bucket"
        key = "some-key"
    }

    s3.getObject(req) { resp ->
        val stream = resp.body as ByteStream.OneShotStream
        val flow = produceFlow(stream.readFrom())
        consumeFlow(flow)
    }

    println("Complete!")
}

fun produceFlow(reader: SdkByteReadChannel): Flow<ByteArray> {
    val buffer = ByteArray(bufferSize)
    return flow {
        var bytes = reader.readAvailable(buffer)
        while (bytes != -1) {
            if (bytes > 0) {
                emit(buffer.copyOfRange(0, bytes))
            }
            bytes = reader.readAvailable(buffer)
        }
    }
}

suspend fun consumeFlow(chunks: Flow<ByteArray>) {
    var total = 0
    RandomAccessFile("/tmp/chunk-download.zip", "rw").use { file ->
        chunks.collect { chunk ->
            println("Received a ${chunk.size} byte chunk, writing to file (written $total bytes so far)")
            file.write(chunk)
            total += chunk.size
        }
        println("Finished writing file, wrote $total bytes")
    }
}

This code sample emits individually-allocated buffer chunks into a Flow in produceFlow. It then writes those chunks to a RandomAccessFile in consumeFlow. Note that the consuming code needs to execute in the context of the getObject response lambda (otherwise, the stream will be freed and the bytes will no longer be available).

[hantsy]

Aws SDK for Java includes some async APIs(based on Java 8 CompletableFuture), I hope this Aws SDK for Kotlin will include built-in Kotlin Coroutines/Flow(or ReactiveStreams API) for the async support.