Sharding with >= 10 partitions lead to non correct offset tracking (JDBC only)

Lagom Version (1.2.x / 1.3.x / etc)

1.4.x

API (Scala / Java / Neither / Both)

Scala (not sure if reproduce in java)

Library Dependencies

val mysqlDriver = "mysql" % "mysql-connector-java" % "8.0.13"

reproduces also with postgres and h2

Expected Behavior

1. create an entity with an id that gets mapped to a shard >9
2. publish event to topic
3. read_side_offsets should track offset of the published message with correct tag:

topicProducer-greetings	org.example.hello.impl.HelloEvent18	1

Actual Behavior

1. http POST 127.0.0.1:9000/api/hello/test message=hello
2. event is published twice to topic
3. state of read_side_offsets as following:

topicProducer-greetings	org.example.hello.impl.HelloEvent1	1
topicProducer-greetings	org.example.hello.impl.HelloEvent18	1

Reproducible Test Case

project at: https://github.com/mikearnaldi/lagom-issue

Instructions

1. cd hello-impl; docker-compose up -d; cd ..
2. sbt runAll
3. http POST 127.0.0.1:9000/api/hello/test message=hello

the event gets a tag of org.example.hello.impl.HelloEvent18 but gets forwarded to read side twice with org.example.hello.impl.HelloEvent1 and org.example.hello.impl.HelloEvent18

Note

1. I am using httpie as a client to issue http requests
2. I was not able to reproduce with number of shards < 10

[octonato]

We think we can build a workaround for this by filtering out the undesired events in memory.

If you query for FooEvent1 and fetches FooEvent10 because of this bug, we can filter out all FooEvent10 and only pass FooEvent1 to read-side processors or topic producers.

It has the drawback that we will be fetching more events than needed, but at least we won’t impact the offset table.

[octonato]

There is a workaround in akka-persistence-jdbc v3.5.3. Next Lagom version will contain it (see #2693)