question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unhandled StorageException: (400) Bad Request thrown from SDK logging code

See original GitHub issue

Occasionally, while running multiple instances in Azure App Services (not in development), our continuous WebJob terminates irregularly. The following is from Application Logging:

Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass42.<PutBlobImpl>b__41(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End()
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromStream(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromByteArray(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<TryExecuteAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Timers.RecurrentTaskSeriesCommand.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

The above is with 1.1.2. It also occurs with 2.0.0-beta2. The call stack looks slightly different:

Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass42.<PutBlobImpl>b__41(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End()
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromStream(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromByteArray(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<TryExecuteAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Timers.RecurrentTaskSeriesCommand.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.<>c__DisplayClass7.<OnUnhandledExceptionAsync>b__6()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

The worst part, is that it seems that our WebJob process sometimes doesn’t (most the time, it does) get restarted by the Azure platform after it terminates irregularly. We’ve had times where all our instances eventually get into this state, and processing comes to a stand-still. We have to Stop and re-Start the WebJob in the Azure portal for processing to resume.

Issue Analytics

  • State:open
  • Created 7 years ago
  • Comments:17 (6 by maintainers)

github_iconTop GitHub Comments

3reactions
awahntehcommented, Nov 17, 2017

@paulbatum: how does one force the version of WIndowsAzure Storage SDK in Azure functions? I have tried adding a dependentAssembly binding redirect in the web.config file for my AzureFunctions blade to no avail. Still get the same error: <bindingRedirect oldVersion="0.0.0.0-8.1.4.0" newVersion="7.2.1.0" />

Did no one test this before releaseing v8.x into the wild?

0reactions
Piekopcommented, Feb 4, 2020

Hi all, I know this has been a while, but we’re experiencing the same issues with our webjobs. Host dll is version 2.2.0.0. The servicebustriggers are stalling while others keep un running correctly. We don’t have any storage operations in our own code, but the SDK is logging operations to the blobstorage. More specific we’re experiencing a 400 (bad request MD5mistmatch) Any help is much appreciated.

Microsoft.WindowsAzure.Storage.StorageException : The remote server returned an error: (400) Bad Request. —> System.Net.WebException : The remote server returned an error: (400) Bad Request., " at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode,HttpStatusCode actualStatusCode,T retVal,StorageCommandBase1 cmd,Exception ex) at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Shared\Protocol\HttpResponseParsers.Common.cs : 54", " at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass57.<PutBlobImpl>b__56(RESTCommand1 cmd,HttpWebResponse resp,Exception ex,OperationContext ctx) at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs : 3131", at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs : 299 , End of inner exception, at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs : 77, at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromStream(IAsyncResult asyncResult) at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs : 844, at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromByteArray(IAsyncResult asyncResult) at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs : 1323, at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult) at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs : 1482, at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass7.<CreateCallbackVoid>b__5(IAsyncResult ar) at c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs : 120, " at async Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.UpdateOutputBlob(CancellationToken cancellationToken,Boolean flushAndClose)“, " at async Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.SaveAndCloseAsync(FunctionInstanceLogEntry item,CancellationToken cancellationToken)”, at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??), " at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken)", Additional Details:, UserSpecifiedMd5: 1B2M2Y8AsgTpgAmY7PhCfg==, ServerCalculatedMd5: uaYUEDgc/d/MnqA6p6KV0A==, ,

Read more comments on GitHub >

github_iconTop Results From Across the Web

Spring REST using Jackson - 400 bad request logging
In my case I was seeing 400 responses in my client code but no logging on the server side (even with a custom...
Read more >
the remote server returned an error .(400 bad request)
The 400 (Bad Request) status code indicates that the server cannot or will not process the request because the received syntax is invalid, ......
Read more >
Response status code does not indicate success: 400 (Bad ...
The remote server returned an error: (400) Bad Request. Description: An unhandled exception occurred during the execution of the current web ...
Read more >
Troubleshooting | Cloud Storage
Logging raw requests; Adding custom headers; Error codes ... 400: Bad Request ... Solution: The value you used in your Content-Range header is...
Read more >
Handling Exceptions Returned from the Web API
A 400 error, BadRequest, is used when you have validation errors from data posted back by the user. You pass a ModelStateDictionary object ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found