Intermittent truncation of s3 objects

Describe the bug

We have a 30kb JSON file stored on s3. We are using the kotlin AWS SDK to read this object from s3 and write it to a file; we do .getObject, and then .body.writeToFile. A small percentage of the time (~ 1 out of 1000 runs), when we read the file contents back, the data is truncated and does not contain the full content from the object.

Expected behavior

Calling s3Client.getObject(...).writeToFile(myFile) followed by myFile.bufferedReader().readText() should always result in a String containing the full contents of the s3 object.

Current behavior

About 1 out of 1000 runs, the readText() call returns a String that contains a truncated version of the full content of the s3 object. When we try to deserialize we see exceptions like the one below. I might be inclined to chalk this up to user error (and it still may be), but the Fatal error from the crt seemed concerning:

Caused by: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 29887: Expected quotation mark '"', but had '"' instead
JSON input: .....   "nameServers": [
          "ns-2046.aws
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:524)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail$kotlinx_serialization_json(AbstractJsonLexer.kt:221)
	at kotlinx.serialization.json.internal.StringJsonLexer.consumeKeyString(StringJsonLexer.kt:88)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:310)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeString(StreamingJsonDecoder.kt:256)
	at kotlinx.serialization.internal.StringSerializer.deserialize(Primitives.kt:142)
	at kotlinx.serialization.internal.StringSerializer.deserialize(Primitives.kt:138)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:36)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:535)
	at kotlinx.serialization.internal.ListLikeSerializer.readElement(CollectionSerializers.kt:80)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:36)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at momento.accounts.DnsConfig$$serializer.deserialize(AccountRegistryTypes.kt:47)
	at momento.accounts.DnsConfig$$serializer.deserialize(AccountRegistryTypes.kt:47)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:36)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeNullableSerializableElement(AbstractDecoder.kt:79)
	at momento.accounts.MomentoSsoGlobalRegistryAccount$$serializer.deserialize(AccountRegistryTypes.kt:72)
	at momento.accounts.MomentoSsoGlobalRegistryAccount$$serializer.deserialize(AccountRegistryTypes.kt:72)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:36)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at momento.accounts.MomentoSsoOrg$$serializer.deserialize(AccountRegistryTypes.kt:148)
	at momento.accounts.MomentoSsoOrg$$serializer.deserialize(AccountRegistryTypes.kt:148)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:36)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at momento.accounts.AllMomentoOrgs$$serializer.deserialize(AccountRegistryTypes.kt:175)
	at momento.accounts.AllMomentoOrgs$$serializer.deserialize(AccountRegistryTypes.kt:175)
	at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:59)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:36)
	at kotlinx.serialization.json.Json.decodeFromString(Json.kt:100)
	at momento.accounts.AccountRegistry.getAllOrgs(AccountRegistry.kt:22)
	... 11 more
Fatal error condition occurred in /Users/Shared/Jenkins/Home/workspace/aws-crt-java-osx-x64/aws-crt-java/src/native/crt.c:207: result == JNI_OK
Exiting Application
################################################################################
Stack trace:
################################################################################
1   AWSCRT_1651014013292137092625430118 0x00000001086d5ac6 aws_fatal_assert + 70
2   AWSCRT_1651014013292137092625430118 0x000000010864e14b aws_jni_get_thread_env + 171
3   AWSCRT_1651014013292137092625430118 0x0000000108651036 s_on_http_conn_manager_shutdown_complete_callback + 22
4   AWSCRT_1651014013292137092625430118 0x000000010865cfa4 s_aws_http_connection_manager_finish_destroy + 324
5   AWSCRT_1651014013292137092625430118 0x000000010865ce28 s_final_destruction_task + 56
6   AWSCRT_1651014013292137092625430118 0x00000001086e454c s_run_all + 460
7   AWSCRT_1651014013292137092625430118 0x00000001086b6f83 s_event_thread_main + 1715
8   AWSCRT_1651014013292137092625430118 0x00000001086e27cd thread_fn + 253
9   libsystem_pthread.dylib             0x00007ff815337514 _pthread_start + 125
10  libsystem_pthread.dylib             0x00007ff81533302f thread_start + 15

Steps to Reproduce

Do this in a loop. The s3 object in our case is 30kb of JSON.

            val tempFile = kotlin.io.path.createTempFile()
            s3Client.getObject(
                GetObjectRequest {
                    this.bucket = myBucket
                    this.key = myKey
                }
            ) {
                it.body?.writeToFile(tempFile) ?: throw IllegalStateException("Missing body on s3 getObject response")
            }

            Json.decodeFromString(tempFile.bufferedReader().readText())

Possible Solution

No response

Context

No response

AWS Kotlin SDK version used

v0.10.1-beta

Platform (JVM/JS/Native)

JVM

Operating System and version

OSX, AWS Lambda via AL2 docker image

[cprice404]

I haven’t observed this yet with 0.14.4-beta. I also simplified my logic and am doing nothing but decodeToString in the closure now, so it’s not entirely apples-to-apples, but we can close this ticket and I can re-open, or open a new one, if we observe it again.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.