Batch operations only create first entity in batch
See original GitHub issueWhich service(blob, file, queue, table) does this issue concern?
Tables
Which version of the Azurite was used?
3.16.0
Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)
npm
What’s the Node.js version?
v16.13.1.
What problem was encountered?
When creating a batch operation from the Go SDK, only the first entity in the batch is created.
Steps to reproduce the issue?
If possible, please provide the debug log using the -d parameter, replacing <pathtodebuglog> with an appropriate path for your OS, or review the instructions for docker containers:
2022-03-08T22:44:06.714Z info: Azurite Blob service is starting on 127.0.0.1:10000
2022-03-08T22:44:06.716Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2022-03-08T22:44:06.716Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2022-03-08T22:44:06.769Z info: BlobGCManager:start() Starting BlobGCManager. Set status to Initializing.
2022-03-08T22:44:06.770Z info: BlobGCManager:start() Trigger mark and sweep loop. Set status to Running.
2022-03-08T22:44:06.770Z info: BlobGCManager:markSweepLoop() Start next mark and sweep.
2022-03-08T22:44:06.771Z info: BlobGCManager:markSweep() Get all extents.
2022-03-08T22:44:06.773Z info: BlobGCManager:start() BlobGCManager successfully started.
2022-03-08T22:44:06.794Z info: BlobGCManager:markSweep() Got 0 extents.
2022-03-08T22:44:06.794Z info: BlobGCManager:markSweep() Get referred extents.
2022-03-08T22:44:06.795Z info: BlobGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2022-03-08T22:44:06.796Z info: BlobGCManager:markSweepLoop() Mark and sweep finished, taken 26ms.
2022-03-08T22:44:06.796Z info: BlobGCManager:markSweepLoop() Sleep for 600000ms.
2022-03-08T22:44:06.799Z info: Azurite Blob service successfully listens on http://127.0.0.1:10000
2022-03-08T22:44:06.800Z info: Azurite Queue service is starting on 127.0.0.1:10001
2022-03-08T22:44:06.801Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2022-03-08T22:44:06.801Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2022-03-08T22:44:06.827Z info: QueueGCManager:start() Starting QueueGCManager, set status to Initializing
2022-03-08T22:44:06.828Z info: QueueGCManager:start() Trigger mark and sweep loop, set status to Running.
2022-03-08T22:44:06.828Z info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2022-03-08T22:44:06.828Z info: QueueGCManger:markSweep() Get all extents.
2022-03-08T22:44:06.829Z info: QueueGCManager:start() QueueGCManager successfully started.
2022-03-08T22:44:06.829Z info: QueueGCManager:marksweep() Get 0 extents.
2022-03-08T22:44:06.830Z info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2022-03-08T22:44:06.830Z info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2022-03-08T22:44:06.830Z info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 2ms.
2022-03-08T22:44:06.830Z info: QueueGCManager:markSweepLoop() Sleep for 60000
2022-03-08T22:44:06.832Z info: Azurite Queue service successfully listens on http://127.0.0.1:10001
2022-03-08T22:44:06.832Z info: Azurite Table service is starting on 127.0.0.1:10002
2022-03-08T22:44:06.832Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2022-03-08T22:44:06.833Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2022-03-08T22:44:06.852Z info: Azurite Table service successfully listens on http://127.0.0.1:10002
2022-03-08T22:44:11.804Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: TableStorageContextMiddleware: RequestMethod=POST RequestURL=http://127.0.0.1/devstoreaccount1/Tables RequestHeaders:{"host":"127.0.0.1:10002","user-agent":"azsdk-go-internal/v0.6.0 azsdk-go-azcore/v0.21.0 (go1.17; Windows_NT)","content-length":"25","accept":"application/json;odata=minimalmetadata","authorization":"SharedKeyLite devstoreaccount1:gBWqSvjN3/S5xmxI/1ePjjBM/BkWaHp8zfPgzZHpxAI=","content-type":"application/json","dataserviceversion":"3.0","x-ms-date":"Tue, 08 Mar 2022 22:44:11 GMT","x-ms-version":"2019-02-02","accept-encoding":"gzip"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2022-03-08T22:44:11.805Z d7ee8af0-8035-4121-b770-652fc0d07c66 debug: tableStorageContextMiddleware: Dispatch pattern string: /Tables
2022-03-08T22:44:11.805Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: tableStorageContextMiddleware: Account=devstoreaccount1 tableName=undefined
2022-03-08T22:44:11.805Z d7ee8af0-8035-4121-b770-652fc0d07c66 verbose: DispatchMiddleware: Dispatching request...
2022-03-08T22:44:11.808Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: DispatchMiddleware: Operation=Table_Create
2022-03-08T22:44:11.808Z d7ee8af0-8035-4121-b770-652fc0d07c66 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2022-03-08T22:44:11.808Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: TableSharedKeyLiteAuthenticator:validate() Start validation against account shared key authentication.
2022-03-08T22:44:11.809Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: TableSharedKeyLiteAuthenticator:validate() [STRING TO SIGN]:"Tue, 08 Mar 2022 22:44:11 GMT\n/devstoreaccount1/devstoreaccount1/Tables"
2022-03-08T22:44:11.810Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: TableSharedKeyLiteAuthenticator:validate() Calculated authentication header based on key1: SharedKeyLite devstoreaccount1:gBWqSvjN3/S5xmxI/1ePjjBM/BkWaHp8zfPgzZHpxAI=
2022-03-08T22:44:11.810Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: TableSharedKeyLiteAuthenticator:validate() Signature 1 matched.
2022-03-08T22:44:11.811Z d7ee8af0-8035-4121-b770-652fc0d07c66 verbose: DeserializerMiddleware: Start deserializing...
2022-03-08T22:44:11.816Z d7ee8af0-8035-4121-b770-652fc0d07c66 debug: deserialize(): Raw request body string is (removed all empty characters) {"TableName":"testTable"}
2022-03-08T22:44:11.818Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: HandlerMiddleware: DeserializedParameters={"options":{"queryOptions":{},"dataServiceVersion":"3.0"},"version":"2019-02-02","tableProperties":{"tableName":"testTable"},"body":"ReadableStream"}
2022-03-08T22:44:11.821Z d7ee8af0-8035-4121-b770-652fc0d07c66 verbose: SerializerMiddleware: Start serializing...
2022-03-08T22:44:11.825Z d7ee8af0-8035-4121-b770-652fc0d07c66 debug: Serializer: Raw response body string is {"odata.metadata":"http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element","TableName":"testTable"}
2022-03-08T22:44:11.825Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: Serializer: Start returning stream body.
2022-03-08T22:44:11.825Z d7ee8af0-8035-4121-b770-652fc0d07c66 info: EndMiddleware: End response. TotalTimeInMS=21 StatusCode=201 StatusMessage=Created Headers={"server":"Azurite-Table/3.16.0","content-type":"application/json;odata=minimalmetadata","x-ms-request-id":"d7ee8af0-8035-4121-b770-652fc0d07c66","x-ms-version":"2021-04-10","date":"Tue, 08 Mar 2022 22:44:11 GMT","preference-applied":"return-content"}
2022-03-08T22:44:11.830Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: TableStorageContextMiddleware: RequestMethod=POST RequestURL=http://127.0.0.1/devstoreaccount1/$batch RequestHeaders:{"host":"127.0.0.1:10002","user-agent":"azsdk-go-internal/v0.6.0 azsdk-go-azcore/v0.21.0 (go1.17; Windows_NT)","content-length":"1335","accept":"application/json;odata=minimalmetadata","authorization":"SharedKeyLite devstoreaccount1:BZDL6F8lMigFYOVJiK9Iu3iMvatet+yw/brzrc8tIa4=","content-type":"multipart/mixed; boundary=batch_1900ea73-92bf-401d-451c-a17207ccecd2","dataserviceversion":"3.0","x-ms-date":"Tue, 08 Mar 2022 22:44:11 GMT","x-ms-version":"2019-02-02","accept-encoding":"gzip"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2022-03-08T22:44:11.831Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc debug: tableStorageContextMiddleware: Dispatch pattern string: /$batch
2022-03-08T22:44:11.831Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: tableStorageContextMiddleware: Account=devstoreaccount1 tableName=$batch
2022-03-08T22:44:11.831Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc verbose: DispatchMiddleware: Dispatching request...
2022-03-08T22:44:11.831Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: DispatchMiddleware: Operation=Table_Batch
2022-03-08T22:44:11.832Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2022-03-08T22:44:11.832Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: TableSharedKeyLiteAuthenticator:validate() Start validation against account shared key authentication.
2022-03-08T22:44:11.832Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: TableSharedKeyLiteAuthenticator:validate() [STRING TO SIGN]:"Tue, 08 Mar 2022 22:44:11 GMT\n/devstoreaccount1/devstoreaccount1/$batch"
2022-03-08T22:44:11.832Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: TableSharedKeyLiteAuthenticator:validate() Calculated authentication header based on key1: SharedKeyLite devstoreaccount1:BZDL6F8lMigFYOVJiK9Iu3iMvatet+yw/brzrc8tIa4=
2022-03-08T22:44:11.833Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: TableSharedKeyLiteAuthenticator:validate() Signature 1 matched.
2022-03-08T22:44:11.834Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc verbose: DeserializerMiddleware: Start deserializing...
2022-03-08T22:44:11.834Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: HandlerMiddleware: DeserializedParameters={"version":"2019-02-02","options":{"dataServiceVersion":"3.0"},"multipartContentType":"multipart/mixed; boundary=batch_1900ea73-92bf-401d-451c-a17207ccecd2","contentLength":1335,"body":"ReadableStream"}
2022-03-08T22:44:11.836Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc debug: TableHandler:batch() Raw request string is "--batch_1900ea73-92bf-401d-451c-a17207ccecd2\r\nContent-Type: multipart/mixed; boundary=changeset_16a8eca4-6929-4696-788e-f99e877606c0\r\n\r\n--changeset_16a8eca4-6929-4696-788e-f99e877606c0\r\nContent-Id: 0\r\nContent-Transfer-Encoding: binary\r\nContent-Type: application/http\r\n\r\nPOST http://127.0.0.1:10002/devstoreaccount1/testTable HTTP/1.1\r\nAccept: application/json;odata=minimalmetadata\r\nContent-Length: 71\r\nContent-Type: application/json;odata=nometadata\r\nDataserviceversion: 3.0\r\nDate: Tue, 08 Mar 2022 22:44:11 GMT\r\nPrefer: return-no-content\r\nX-Ms-Date: Tue, 08 Mar 2022 22:44:11 GMT\r\nX-Ms-Version: 2019-02-02\r\n\r\n{\"PartitionKey\":\"uuid\",\"RowKey\":\"rkey1\",\"price\":5,\"product\":\"product1\"}\r\n--changeset_16a8eca4-6929-4696-788e-f99e877606c0\r\nContent-Id: 1\r\nContent-Transfer-Encoding: binary\r\nContent-Type: application/http\r\n\r\nPOST http://127.0.0.1:10002/devstoreaccount1/testTable HTTP/1.1\r\nAccept: application/json;odata=minimalmetadata\r\nContent-Length: 72\r\nContent-Type: application/json;odata=nometadata\r\nDataserviceversion: 3.0\r\nDate: Tue, 08 Mar 2022 22:44:11 GMT\r\nPrefer: return-no-content\r\nX-Ms-Date: Tue, 08 Mar 2022 22:44:11 GMT\r\nX-Ms-Version: 2019-02-02\r\n\r\n{\"PartitionKey\":\"uuid\",\"RowKey\":\"rkey2\",\"price\":10,\"product\":\"product2\"}\r\n--changeset_16a8eca4-6929-4696-788e-f99e877606c0--\r\n\r\n--batch_1900ea73-92bf-401d-451c-a17207ccecd2--\r\n"
2022-03-08T22:44:11.848Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc debug: TableHandler:batch() Raw response string is "--batchresponse_1900ea73-92bf-401d-451c-a17207ccecd2\r\nContent-Type: multipart/mixed; boundary=changesetresponse_16a8eca4-6929-4696-788e-f99e877606c0\r\n\r\n--changesetresponse_16a8eca4-6929-4696-788e-f99e877606c0\r\nContent-Type: application/http\r\nContent-Transfer-Encoding: binary\r\n\r\nHTTP/1.1 204 No Content\r\nX-Content-Type-Options: nosniff\r\nCache-Control: no-cache\r\nDataServiceVersion: 3.0;\r\nLocation: http://127.0.0.1:10002/devstoreaccount1/testTable(PartitionKey='uuid',RowKey='rkey1')\r\nDataServiceId: http://127.0.0.1:10002/devstoreaccount1/testTable(PartitionKey='uuid',RowKey='rkey1')\r\nETag: W/\"datetime'2022-03-08T22%3A44%3A11.8300000Z'\"\r\n\r\n--changesetresponse_16a8eca4-6929-4696-788e-f99e877606c0--\r\n--batchresponse_1900ea73-92bf-401d-451c-a17207ccecd2--\r\n"
2022-03-08T22:44:11.849Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc verbose: SerializerMiddleware: Start serializing...
2022-03-08T22:44:11.849Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: Serializer: Start returning stream body.
2022-03-08T22:44:11.854Z ebd9cbb4-2dd7-40d9-998e-b857254ea3bc info: EndMiddleware: End response. TotalTimeInMS=24 StatusCode=202 StatusMessage=Accepted Headers={"server":"Azurite-Table/3.16.0","content-type":"multipart/mixed; boundary=batchresponse_1900ea73-92bf-401d-451c-a17207ccecd2","x-ms-request-id":"ebd9cbb4-2dd7-40d9-998e-b857254ea3bc","x-ms-version":"2021-04-10","date":"Tue, 08 Mar 2022 22:44:11 GMT"}
2022-03-08T22:44:11.856Z 9386f21d-19c3-452c-9689-53368b46d261 info: TableStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/Tables('testTable') RequestHeaders:{"host":"127.0.0.1:10002","user-agent":"azsdk-go-internal/v0.6.0 azsdk-go-azcore/v0.21.0 (go1.17; Windows_NT)","accept":"application/json","authorization":"SharedKeyLite devstoreaccount1:dm9zyaWLEq3TNb9gKAIhJfqqwAbdysWb+uyRB14cK64=","x-ms-date":"Tue, 08 Mar 2022 22:44:11 GMT","x-ms-version":"2019-02-02","accept-encoding":"gzip"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2022-03-08T22:44:11.856Z 9386f21d-19c3-452c-9689-53368b46d261 debug: tableStorageContextMiddleware: Dispatch pattern string: /Tables('testTable')
2022-03-08T22:44:11.856Z 9386f21d-19c3-452c-9689-53368b46d261 info: tableStorageContextMiddleware: Account=devstoreaccount1 tableName=testTable
2022-03-08T22:44:11.856Z 9386f21d-19c3-452c-9689-53368b46d261 verbose: DispatchMiddleware: Dispatching request...
2022-03-08T22:44:11.857Z 9386f21d-19c3-452c-9689-53368b46d261 info: DispatchMiddleware: Operation=Table_Delete
2022-03-08T22:44:11.857Z 9386f21d-19c3-452c-9689-53368b46d261 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2022-03-08T22:44:11.857Z 9386f21d-19c3-452c-9689-53368b46d261 info: TableSharedKeyLiteAuthenticator:validate() Start validation against account shared key authentication.
2022-03-08T22:44:11.857Z 9386f21d-19c3-452c-9689-53368b46d261 info: TableSharedKeyLiteAuthenticator:validate() [STRING TO SIGN]:"Tue, 08 Mar 2022 22:44:11 GMT\n/devstoreaccount1/devstoreaccount1/Tables('testTable')"
2022-03-08T22:44:11.857Z 9386f21d-19c3-452c-9689-53368b46d261 info: TableSharedKeyLiteAuthenticator:validate() Calculated authentication header based on key1: SharedKeyLite devstoreaccount1:dm9zyaWLEq3TNb9gKAIhJfqqwAbdysWb+uyRB14cK64=
2022-03-08T22:44:11.858Z 9386f21d-19c3-452c-9689-53368b46d261 info: TableSharedKeyLiteAuthenticator:validate() Signature 1 matched.
2022-03-08T22:44:11.858Z 9386f21d-19c3-452c-9689-53368b46d261 verbose: DeserializerMiddleware: Start deserializing...
2022-03-08T22:44:11.858Z 9386f21d-19c3-452c-9689-53368b46d261 info: HandlerMiddleware: DeserializedParameters={"version":"2019-02-02","options":{}}
2022-03-08T22:44:11.862Z 9386f21d-19c3-452c-9689-53368b46d261 verbose: SerializerMiddleware: Start serializing...
2022-03-08T22:44:11.863Z 9386f21d-19c3-452c-9689-53368b46d261 info: EndMiddleware: End response. TotalTimeInMS=7 StatusCode=204 StatusMessage=undefined Headers={"server":"Azurite-Table/3.16.0","content-type":"application/json;odata=minimalmetadata","x-ms-request-id":"9386f21d-19c3-452c-9689-53368b46d261","x-ms-version":"2021-04-10","date":"Tue, 08 Mar 2022 22:44:11 GMT"}
2022-03-08T22:45:06.728Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2022-03-08T22:45:06.729Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2022-03-08T22:45:06.815Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2022-03-08T22:45:06.815Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2022-03-08T22:45:06.831Z info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2022-03-08T22:45:06.831Z info: QueueGCManger:markSweep() Get all extents.
2022-03-08T22:45:06.831Z info: QueueGCManager:marksweep() Get 0 extents.
2022-03-08T22:45:06.831Z info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2022-03-08T22:45:06.831Z info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2022-03-08T22:45:06.831Z info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 0ms.
2022-03-08T22:45:06.832Z info: QueueGCManager:markSweepLoop() Sleep for 60000
2022-03-08T22:45:06.850Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2022-03-08T22:45:06.850Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
Have you found a mitigation/solution?
Issue Analytics
- State:
- Created 2 years ago
- Comments:8 (5 by maintainers)
Top Results From Across the Web
Creating an S3 Batch Operations job - AWS Documentation
Create a job to perform large-scale Batch Operations on Amazon S3 objects ... Operations job, you can request a completion report for all...
Read more >Getting Started with Batch Operations in EF Extensions (EFE)
Batch Operations method allows to perform UPDATE or DELETE operations directly in the database using a LINQ Query without loading entities in the...
Read more >'CREATE FUNCTION' must be the first statement in a query ...
You can avoid the. should be the first statement in a batch file. error without adding GO statements by putting the sql inside...
Read more >How to trigger consecutive batch operations using the output ...
The only issue is how to pass the output of the first one to the second one. Another thing: in the first_data_batch function...
Read more >Batch Insert/Update with Hibernate/JPA - Baeldung
Let's first look at how we can use batch inserts when we're dealing with only one entity type. We'll use the previous code...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Based on the response tracing from Azurite’s batch handling, all the elements are being returned, and appear correctly serialized so I’m looking into some traces via fiddler to see where there might be a difference.
This could be an issue with something in middleware or in the http handling of the response.
The URL encoding is OK 😄 , and I’ve validated in an additional unit test for this.
The ordering of elements is also OK, and led me to simplify the deserialization logic (also a good thing).
Once I have compared the service trace against Azurite’s trace, I’m hoping to have a better idea of the root cause, if it is visible in the communication between client and server.
The Go SDK is URL encoding some of the characters inside the batch request.
This causes a problem during deserialization. It also uses a different ordering of some elements as opposed to all other batch clients, I shall address this as well. I shall submit a PR to correct this.