question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CosmosDB bulk upsert extremly slow

See original GitHub issue

Below is how I am using the bulk upsert feature in ComosDB. This code is extremely slow even for 15 documents.

var container = _database.GetContainer(collectionName);
            var cts = new List<Task>();
            foreach (var d in documents)
            {
                var t = container.UpsertItemAsync(d,
                    new PartitionKey(d.GetPartitionKey())
                );

                cts.Add(t);
            }
            await Task.WhenAll(cts);

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
bartelinkcommented, May 5, 2021

Thanks for being patient, will do (and apologies for not reading the answers you provided in the thread distinguishing between and referencing docs for Batch vs Bulk).

0reactions
j82wcommented, May 6, 2021

This is a good article explaining Cosmos DB partitioning: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview

If it’s only 100 documents then doing individual read items is the recommend way. The SDK is thread safe so you can send multiple requests in parallel. Once the SDK connects to a single physical machine it will reuse the connection for future requests. Using the Benchmark project on Linux VM with 4 cores we are able to get over 10,000 request per a second.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot slow requests in Azure Cosmos DB .NET SDK
In Azure Cosmos DB, you might notice slow requests. ... Bulk mode is a throughput optimized mode meant for high data volume operations, ......
Read more >
How to troubleshoot slow inserts to Azure Cosmos from ...
1 Answer 1 · Thanks. Turning off bulk mode was the solution. With AllowBulkExecution = false Insert times are now a factor of...
Read more >
Maximizing The Throughput of Azure Cosmos DB Bulk Import ...
The order of the data is very important considering how the Bulk Executor Library works. It pulls batches of data from the ingestion...
Read more >
Need help to insert huge data into cosmos db from
I am trying to insert 6GB of data into cosmos db using OLTP Connector. Container RU's:40000. Cluster Config: image.png cfg = { "spark.cosmos....
Read more >
Improve performance & scalability of Azure Cosmos DB ...
In this blog, we will focus on Azure Cosmos DB SQL API that stores data in JSON documents and queries the same through...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found