question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[cosmos] Using the bulk API

See original GitHub issue
  • cosmos:
  • 3.9.0:
  • macosx:
  • nodejs
    • 12.16.3:
  • browser
    • name/version:
  • typescript
    • version:
  • Is the bug related to documentation in

Describe the bug I am updating some code to make use of the bulk insert API made available in the 3.9.0 update.

I am trying to update the following code:

for (i = 0; i < rows.length; i++) {
    container.items.create(rows[i]);
}

So far I have tried:

container.items.create(rows);

this returned a 400 BadRequest One of the specified inputs is invalid I then had a look into the source and have been trying this (also trying to specify partitionKey).

operations = [];
for (i = 0; i < rows.length; i++) {
    operations.push({
            operationType: "Create",
            resourceBody: rows[i]
        });
}
container.items.bulk(operations);

This doesn’t give me an error but also nothing appears in the cosmos container.

An example of rows[i] is a flat json object with string values:

{
  "Town": "blah",
  "City": "blah",
  ...
}

To Reproduce Steps to reproduce the behavior:

  1. A working cosmos database with a container
const cosmosClient = new CosmosClient({ endpoint:COSMOS_ENDPOINT, key:COSMOS_KEY });
const database = cosmosClient.database(COSMOS_DB_NAME);
const { container } = await database.containers.createIfNotExists({ id:CONTAINER_NAME });
  1. Verify the for loop container.items.create(rows[i]); (from above) works
  2. Replace with either of the two mentioned attempts.

Expected behavior I expect the bulk API to allow me to bulk create records to avoid having to iterate over a collection, creating one at a time. Additionally I expect the second of my attempts to give me some error as it is not working.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context I originally asked the question in this thread https://github.com/Azure/azure-sdk-for-js/issues/7479 and @zfoster recommended a dedicated bug report. Thanks!

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:20 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
NateShaqcommented, Aug 21, 2020

This bulk insert is blazing fast.

`

  const { database }  = await client.databases.createIfNotExists({ id: "PRIMARY" });
  const { container } = await database.containers.createIfNotExists({ id: tableConfig.containerName }); 
  var i = item.length;
  let operations = [];
  while( i ){
        
        var pos = item.length-i;
        //Ensure the Item ID is set
        var superItem = item[pos];
        superItem.id  = item[pos].id;
        
        //Save Some Timestamps
        var now = new Date().toISOString();   
        superItem.createdAt = now;
        superItem.updatedAt = now;
        superItem.createdMs = new Date(now).getTime();
        
        operations.push({
            operationType: "Create",
            resourceBody: superItem
        });
           
        //Send Operations Every 100 or when Count is 1 
        if( i % 100 == 0 || i == 1){
           try { var response = await container.items.bulk(operations);
                console.log("Zresponse:" + JSON.stringify(response));
           } catch (error) {
                console.log("Error Occured in Bulk Operation" + error);
           }
           //Add some code to handle RU Max Reached. 
           operations = []; //Important Clear Array
        }
        
        i--;
  }

`

1reaction
NateShaqcommented, Jun 19, 2022

@tbshrr Thank you! I had that line commented out from previous attempts. Thanks for the reminder. const { BulkOperationType } = require('@azure/cosmos') operations.push({ operationType: BulkOperationType.Upsert, id: superItem.id, resourceBody: superItem });

Code is working as expected now.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Azure Cosmos DB bulk executor library overview
The bulk executor library allows you to perform bulk operations in Azure Cosmos DB through bulk import and bulk update APIs.
Read more >
Move multiple documents in bulk with the Azure Cosmos DB ...
The easiest way to learn how to perform a bulk operation is to attempt to push many documents to an Azure Cosmos DB...
Read more >
When not to use Azure Cosmos DB Bulk Executor?
Bulk operations are optimized for large number of documents. For such a low volume, you can simply call them concurrently.
Read more >
Using Bulk Operations of Azure Cosmos DB by using SDK
Azure Cosmos DB SQL API provides a way to retrieve documents from the database by using a T-SQL like language. This language is...
Read more >
Microsoft Azure Cosmos DB SQL API mappings configured to ...
Identify the non-partitioned collections in the mapping that uses the bulk API. · Create a new passthrough mapping and create a pipeline for...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found