External call inside topology

Description

If you have already built a Kafka Streams application and you had to call an external service or database in the middle of one of your topologies, you’ve probably faced some issues if that call takes too long which can drive your stream processing application in serious troubles and most likely your overall system in some kind of snowball effect. If not (or not yet 😉), your application is prone such issues and you should be aware about that.

It’s true with JAVA implementation but also in another way with librdkafka based clients.

max.poll.interval.ms = 300000

The aims of this purpose is to allow execute external async call inside your Streamiz topology.

[LGouellec]

@RogerSep,

Because after some reflexion internally, I think it makes more sense to have these ones compared with just an ExternalCallAsync().

As well, developers could map/flatMap key/value or just value with an asynchronous processing (calling a database, HTTP Api, or so on …).

It would be great ! For documentation, I had thought two things :

• Update the stateless page for adding MapAsync(), FlatMapAsync(), MapValuesAsync(), FlatMapValuesAsync() and ForeachAsync(). A brief describe for each new DSL method like the other examples.
• Add a dedicated webpage in two parts : How Streamiz do to make asynchronous processing ? (which pattern use, …) and in second part more a tutoriel with a schema to illustrate the use case, a docker-compose stack, a topology implementation and some screenshots on “How make an asynchronous processing with Streamiz” and I had thought about a use case like : Stream topic A - MapValuesAsync(…) with a mongodb collection - Join an another stream B - Publish in topic C.

And later I will add the page regarding the threading model.

[LGouellec]

Example use case :

            StreamBuilder builder = new StreamBuilder();

            var client = new MongoClient(
                "mongodb://admin:admin@localhost:27017"
            );
            var database = client.GetDatabase("streamiz");

            builder
                .Stream<string, string>("person")
                .MapValuesAsync(async (record, _) => {
                    var persons = await database
                        .GetCollection<Person>("adress")
                        .FindAsync((p) => p.name.Equals(record.Key))
                        .Result.ToListAsync();
                    return persons.FirstOrDefault()?.address.city;
                },
                    RetryPolicy
                        .NewBuilder()
                        .NumberOfRetry(10)
                        .RetryBackOffMs(100)
                        .RetriableException<Exception>()
                        .RetryBehavior(EndRetryBehavior.BUFFERED)
                        .Build())
                .To("person-city");