Amazon.Lambda.KafkaEvents.KafkaEvent cannot be deserialized

Description

The latest version of Amazon.Lambda.KafkaEvents.KafkaEvent cannot deserialize events when using SelfManagedKafka as source events.

Reproduction Steps

The hander function signature looks like

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
...
...
    public async Task FunctionHandler(KafkaEvent kafkaEvent, ILambdaContext context){...}

Logs

Error converting the Lambda event JSON payload to type Amazon.Lambda.KafkaEvents.KafkaEvent: The JSON value could not be converted to System.String. Path: $.records.TOPIC[0].partition | LineNumber: 0 | BytePositionInLine: 377.: JsonSerializerException at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.Deserialize[T](Stream requestStream) at lambda_method(Closure , Stream , Stream , LambdaContextInternal )

at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& readStack, Utf8JsonReader& reader, Exception ex) at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack) at System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader) at System.Text.Json.JsonSerializer.ParseCore(ReadOnlySpan1 utf8Json, Type returnType, JsonSerializerOptions options) at System.Text.Json.JsonSerializer.Deserialize[TValue](ReadOnlySpan1 utf8Json, JsonSerializerOptions options) at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.Deserialize[T](Stream requestStream) Cannot get the value of a token type ‘Number’ as a string.: InvalidOperationException at System.Text.Json.Utf8JsonReader.GetString() at System.Text.Json.Serialization.Converters.JsonConverterString.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) at System.Text.Json.JsonPropertyInfoNotNullable`4.OnRead(ReadStack& state, Utf8JsonReader& reader) at System.Text.Json.JsonPropertyInfo.Read(JsonTokenType tokenType, ReadStack& state, Utf8JsonReader& reader) at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)

Environment

• Build Version: https://www.nuget.org/packages/Amazon.Lambda.KafkaEvents/1.0.0
• OS Info: MacOs BigSur 11.6.1
• Build Environment: cli/terminal
• Targeted .NET Platform: .NET Core 3.1 (C#/PowerShell)

Resolution

I noticed that there has been a change to the type Partition property - used to be int now is string. Any reason for that? https://github.com/aws/aws-lambda-dotnet/blame/b49dfae373538aad2811340cd2a03b478cb59318/Libraries/src/Amazon.Lambda.KafkaEvents/KafkaEvent.cs#L46

---

[ashishdhingra]

Got confirmation from service team that partition should be modeled as int. They will work to update the documentation (cannot share the ETA):

• Using Lambda with self-managed Apache Kafka
• Using Lambda with Amazon MSK

We would work in fixing this in Amazon.Lambda.KafkaEvents package.

[ganeshnj]

@gabriel-myob It seems there is inconsistency between the payload you are receiving (partition is int) and docs (partition is string).

I quickly checked Java code, and they appear to model partition as int as well.

This should be an easy fix but we should make sure it works with a real Kafka event.