Frequent 500 responses from Azurite

Which service(blob, file, queue, table) does this issue concern?

Queue

Which version of the Azurite was used?

Docker image (e6649aa0ef87), 2019.11 Version 3.3.0-preview

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub

What’s the Node.js version?

N/A…running as docker image

What problem was encountered?

During local executions from async Python server (running Quart/Flask), Queue service calls are stalling with messages similar to the following:

2019-11-26T20:31:04.552Z 1486809f-4d17-4a86-8250-c9eb0206539d info: QueueStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/jobs/messages RequestHeaders:{"host":"127.0.0.1:10001","accept":"application/xml","x-ms-version":"2018-03-28","x-ms-date":"Tue, 26 Nov 2019 20:30:21 GMT","x-ms-client-request-id":"91ac80f0-108b-11ea-940e-9cb6d0225524","user-agent":"azsdk-python-storage-queue/12.0.0 Python/3.6.8 (Linux-4.4.0-18362-Microsoft-x86_64-with-Ubuntu-18.04-bionic)","authorization":"SharedKey devstoreaccount1:a0MzuFLOAJdijnntUljPOWd1gx85GbVLtU+mV+Gc1L0=","accept-encoding":"gzip, deflate"} ClientIP=172.19.0.1 Protocol=http HTTPVersion=1.1
2019-11-26T20:31:04.552Z 1486809f-4d17-4a86-8250-c9eb0206539d info: QueueStorageContextMiddleware: Account=devstoreaccount1 Queue=jobs Message=messages MessageId=undefined
2019-11-26T20:31:04.553Z 1486809f-4d17-4a86-8250-c9eb0206539d verbose: DispatchMiddleware: Dispatching request...
2019-11-26T20:31:04.553Z 1486809f-4d17-4a86-8250-c9eb0206539d info: DispatchMiddleware: Operation=Messages_Dequeue
2019-11-26T20:31:04.553Z 1486809f-4d17-4a86-8250-c9eb0206539d verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2019-11-26T20:31:04.553Z 1486809f-4d17-4a86-8250-c9eb0206539d info: QueueSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2019-11-26T20:31:04.554Z 1486809f-4d17-4a86-8250-c9eb0206539d info: QueueSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:91ac80f0-108b-11ea-940e-9cb6d0225524\nx-ms-date:Tue, 26 Nov 2019 20:30:21 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1/jobs/messages"
2019-11-26T20:31:04.554Z 1486809f-4d17-4a86-8250-c9eb0206539d info: QueueSharedKeyAuthenticator:validate() Calculated authentication header based on key1: devstoreaccount1:a0MzuFLOAJdijnntUljPOWd1gx85GbVLtU+mV+Gc1L0=
2019-11-26T20:31:04.554Z 1486809f-4d17-4a86-8250-c9eb0206539d info: QueueSharedKeyAuthenticator:validate() Signature 1 matched.
2019-11-26T20:31:04.554Z 1486809f-4d17-4a86-8250-c9eb0206539d verbose: DeserializerMiddleware: Start deserializing...
2019-11-26T20:31:04.554Z 1486809f-4d17-4a86-8250-c9eb0206539d info: HandlerMiddleware: DeserializedParameters={"options":{"requestId":"91ac80f0-108b-11ea-940e-9cb6d0225524"},"version":"2018-03-28"}
2019-11-26T20:31:04.555Z 1486809f-4d17-4a86-8250-c9eb0206539d debug: OperationQueue.operate() Schedule incoming job f8e38d62-2dfa-4e1e-9f09-3cc1d5e3b3a1
2019-11-26T20:31:04.555Z 1486809f-4d17-4a86-8250-c9eb0206539d debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:1
2019-11-26T20:31:04.555Z 1486809f-4d17-4a86-8250-c9eb0206539d verbose: FSExtentStore:readExtent() Creating read stream. LocationId:Default extentId:a62641c4-6f1f-4911-aae4-ddd549769613 path:/opt/azurite/__queuestorage__/a62641c4-6f1f-4911-aae4-ddd549769613 offset:165 count:55 end:219
2019-11-26T20:31:04.556Z 1486809f-4d17-4a86-8250-c9eb0206539d debug: OperationQueue.operate() Job f8e38d62-2dfa-4e1e-9f09-3cc1d5e3b3a1 completes callback, resolve.
2019-11-26T20:31:04.556Z 1486809f-4d17-4a86-8250-c9eb0206539d debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:0
2019-11-26T20:31:04.556Z 1486809f-4d17-4a86-8250-c9eb0206539d debug: OperationQueue:execute() return. Operation.length === 0
2019-11-26T20:31:04.556Z 1486809f-4d17-4a86-8250-c9eb0206539d error: ErrorMiddleware: Received an error, fill error information to HTTP response
2019-11-26T20:31:04.556Z 1486809f-4d17-4a86-8250-c9eb0206539d error: ErrorMiddleware: ErrorName=Error ErrorMessage=ENOENT: no such file or directory, open '/opt/azurite/__queuestorage__/a62641c4-6f1f-4911-aae4-ddd549769613' ErrorStack="Error: ENOENT: no such file or directory, open '/opt/azurite/__queuestorage__/a62641c4-6f1f-4911-aae4-ddd549769613'"
2019-11-26T20:31:04.557Z 1486809f-4d17-4a86-8250-c9eb0206539d error: ErrorMiddleware: Set HTTP code: 500
2019-11-26T20:31:04.557Z 1486809f-4d17-4a86-8250-c9eb0206539d info: EndMiddleware: End response. TotalTimeInMS=5 StatusCode=500 StatusMessage=undefined Headers={"server":"Azurite-Queue/3.3.0-preview"}

This causes the Python code (running Azure storage SDK) to enter a retry loop, and ultimately stall.

Steps to reproduce the issue?

Unknown…seems to be random.

Have you found a mitigation/solution?

Shutting down the image for 5-10 mins, then restarting seems to help, but problem will surface again.

[mjyeaney]

I’m not sharing any folders with my host machine…i.e., nothing is showing on my host c:\ drive (windows 10 as well) or on my WSL host (Ubuntu 18.04). I specifically have drive sharing turned off in Docker - no mounts in play here.

[XiaoningLiu]

Thanks @dafuchs I believe your investigation is pretty close to the truth. Will have a fix!