Dashboard indexer fails when message contains line breaks

If the output from a function invocation contains ‘\r\n’ line breaks (the ones I’ve seen have all been in the message), the indexer fails to index that invocation, and retries repeatedly.

Overwriting the original output file with a duplicate with those line breaks removed resolves the issue.

Example input here: http://254shades.blob.core.windows.net/public/example.txt

Stack trace in the failed index logs:

Microsoft.WindowsAzure.Storage.StorageException: Specified value has invalid CRLF characters.
Parameter name: value ---> System.ArgumentException: Specified value has invalid CRLF characters.
Parameter name: value
   at System.Net.WebHeaderCollection.CheckBadChars(String name, Boolean isHeaderValue)
   at System.Net.WebHeaderCollection.Add(String name, String value)
   at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpWebRequestFactory.AddMetadata(HttpWebRequest request, IDictionary`2 metadata)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<PutBlobImpl>b__37(HttpWebRequest r, OperationContext ctx)
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ProcessStartOfRequest[T](ExecutionState`1 executionState, String startLogMessage)
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromStreamHelper(Stream source, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromByteArray(Byte[] buffer, Int32 index, Int32 count, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadText(String content, Encoding encoding, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Dashboard.Data.BlobConcurrentTextStore.CreateOrUpdate(String id, IDictionary`2 metadata, String text) in c:\BuildAgent\work\1bab5238e8bc2cc\public\src\Dashboard\Data\BlobConcurrentTextStore.cs:line 134
   at Dashboard.Data.RecentInvocationIndexWriter.CreateOrUpdate(FunctionInstanceSnapshot snapshot, DateTimeOffset timestamp) in c:\BuildAgent\work\1bab5238e8bc2cc\public\src\Dashboard\Data\RecentInvocationIndexWriter.cs:line 33
   at Dashboard.Indexers.FunctionIndexer.CreateOrUpdateIndexEntries(FunctionInstanceSnapshot snapshot, DateTimeOffset timestamp, WebJobRunIdentifier webJobRunId) in c:\BuildAgent\work\1bab5238e8bc2cc\public\src\Dashboard\Indexers\FunctionIndexer.cs:line 103
   at Dashboard.Indexers.FunctionIndexer.ProcessFunctionCompleted(FunctionCompletedMessage message) in c:\BuildAgent\work\1bab5238e8bc2cc\public\src\Dashboard\Indexers\FunctionIndexer.cs:line 163
   at Dashboard.Indexers.Indexer.UpdateCore() in c:\BuildAgent\work\1bab5238e8bc2cc\public\src\Dashboard\Indexers\Indexer.cs:line 67
   at Dashboard.Indexers.Indexer.Update() in c:\BuildAgent\work\1bab5238e8bc2cc\public\src\Dashboard\Indexers\Indexer.cs:line 35
Request Information
RequestID:
RequestDate:
StatusMessage:

[gplwhite]

FYI - have been running into a similar issue for the last twelve months or more and have finally tracked the root cause down. Similar to above I noticed that the webjob output blobs found in the azure-jobs-host-output that were failing to index, contained in my case \n linefeed characters only - not CR/LF pairs.

It’s taken me a while to work out how they’re getting in there, but I now realise the messages that contain these characters are messages that had previously failed and been put in the poisoned message queue. When these occur I manually create a new message by copying and pasting the contents from the poisoned message queue into a new queue message using Microsoft Azure Storage Explorer. It seems that when viewing the poisoned message in storage explorer, the storage explorer UI is formatting the JSON message contents for readability. The formatting consists of \n line feeds. When manually copying the message contents over to a new message the \n are copied too.

Unfortunately the fix in https://github.com/Azure/azure-webjobs-sdk/commit/7e11100d1117ea7663a06d727457fbac003ba374 above only handles \r\n pairs and so doesn’t solve my issue. It would be nicer if it stripped \r and \n characters individually.

Is the Webjobs Dashboard still being maintained? Will submitting a pull request make it into production?

[mathewc]

Thanks!