Performance issues with the SDK

Hello folks!

I’ve noticed a weird performance behaviour using the .Net SDK.

The test code:

[Fact]
        public async Task MultiTest()
        {
            var streamId = "Test6k";
            var toAppend = new List<Event>();
            for (int i = 0; i < 6000; i++)
            {
                toAppend.Add(new Event { StreamId = streamId, Type = "TestEvent", Payload = i.ToString() });
            }

            var sw = Stopwatch.StartNew();
            await this._storage.AppendEvents(toAppend.ToArray());
            sw.Stop();
            var appendTime = sw.ElapsedMilliseconds;
            Console.WriteLine($"Write: {appendTime}ms");

            for (int i = 0; i < 10; i++)
            {
                sw.Restart();
                var read = await this._storage.ReadStream(streamId);
                sw.Stop();
                var readTime = sw.ElapsedMilliseconds;
                Console.WriteLine($"Read: {readTime}ms");
                await Task.Delay(1000);
            }
        }

The output:

Write: 2005ms
Read: 444ms
Read: 77ms
Read: 66ms
Read: 228ms
Read: 211ms
Read: 173ms
Read: 67ms
Read: 63ms
Read: 61ms
Read: 59ms

Ignoring the Write, I’m trying to make sure the performance is consistent across Reads but from this simple test, you can see that the first Read takes 3+ times longer than the second and then in the middle it jump in again.

The SQL query being issued agains the database is:

SELECT e.id, e.SId, e.E, e.P, e.T, e.G FROM e WHERE e.id <> '_S' AND e.id <> '_H' ORDER BY e.id ASC

The partition key is being passed to the C# SDK and all the docs are from the same partition key so no cross-partition/scan is involved.

Can someone explain to me why it is so unstable the performance over this simple loop?

We are using netcoreapp3.1 with the SDK on 3.14.0.

The implementation for the ReadStream(string streamId) method on that test that encapsulate the SDK and actually make the query is this one:

var events = new List<Event>();
                var pk = new PartitionKey(streamId);

                var iterator = this._container.GetItemQueryStreamIterator(
                    GET_STREAM_QUERY,
                    requestOptions: new QueryRequestOptions { PartitionKey = pk, MaxItemCount = -1 }
                );

                do
                {
                    using ResponseMessage response = await iterator.ReadNextAsync();

                    using var result = await JsonDocument.ParseAsync(response.Content);
                    var records = result.RootElement.GetProperty(DOCUMENTS_PROP).EnumerateArray();
                    foreach (var r in records)
                    {
                        var id = r.GetProperty(EventFields.ID_FIELD).GetString();

                        var @event = new Event
                        {
                            StreamId = streamId,
                            Sequence = long.Parse(id),
                            Type = r.GetProperty(EventFields.EVENT_TYPE_FIELD).GetString(),
                            Timestamp = r.GetProperty(EventFields.TIMESTAMP_FIELD).GetDateTimeOffset(),
                        };

                        if (r.TryGetProperty(EventFields.EVENT_GROUP_FIELD, out JsonElement eg))
                        {
                            @event.Group = eg.GetString();
                        }

                        if (r.TryGetProperty(EventFields.PAYLOAD_FIELD, out JsonElement p))
                        {
                            @event.Payload = p.GetString();
                        }

                        events.Add(@event);
                    }
                } while (iterator.HasMoreResults);

Any clarifications would be appreciated.

Thank you!

[ealsur]

@galvesribeiro if you see the variation on Direct mode, could you please attach the diagnostics?

For Gateway mode, there is no latency SLA, so latencies can vary. Having said that, the client-side latency can be an effect of something in the wire too.

I would advise to measure and capture performance on the production environment, on Direct mode, and see what are the numbers there. If you are experiencing high latency on Direct mode on the production machines, the diagnostics will help troubleshoot.

Unless the production machines will not be in Azure and they are the ones using the fiber connection?

[msftbot[bot]]

Closing due to in-activity, pease feel free to re-open.