CosmosDB bulk upsert extremly slow
See original GitHub issueBelow is how I am using the bulk upsert feature in ComosDB. This code is extremely slow even for 15 documents.
var container = _database.GetContainer(collectionName);
var cts = new List<Task>();
foreach (var d in documents)
{
var t = container.UpsertItemAsync(d,
new PartitionKey(d.GetPartitionKey())
);
cts.Add(t);
}
await Task.WhenAll(cts);
Issue Analytics
- State:
- Created 2 years ago
- Comments:10 (7 by maintainers)
Top Results From Across the Web
Troubleshoot slow requests in Azure Cosmos DB .NET SDK
In Azure Cosmos DB, you might notice slow requests. ... Bulk mode is a throughput optimized mode meant for high data volume operations, ......
Read more >How to troubleshoot slow inserts to Azure Cosmos from ...
1 Answer 1 · Thanks. Turning off bulk mode was the solution. With AllowBulkExecution = false Insert times are now a factor of...
Read more >Maximizing The Throughput of Azure Cosmos DB Bulk Import ...
The order of the data is very important considering how the Bulk Executor Library works. It pulls batches of data from the ingestion...
Read more >Need help to insert huge data into cosmos db from
I am trying to insert 6GB of data into cosmos db using OLTP Connector. Container RU's:40000. Cluster Config: image.png cfg = { "spark.cosmos....
Read more >Improve performance & scalability of Azure Cosmos DB ...
In this blog, we will focus on Azure Cosmos DB SQL API that stores data in JSON documents and queries the same through...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Thanks for being patient, will do (and apologies for not reading the answers you provided in the thread distinguishing between and referencing docs for Batch vs Bulk).
This is a good article explaining Cosmos DB partitioning: https://docs.microsoft.com/en-us/azure/cosmos-db/partitioning-overview
If it’s only 100 documents then doing individual read items is the recommend way. The SDK is thread safe so you can send multiple requests in parallel. Once the SDK connects to a single physical machine it will reuse the connection for future requests. Using the Benchmark project on Linux VM with 4 cores we are able to get over 10,000 request per a second.