question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

QueueTrigger stops triggering - how to troubleshoot?

See original GitHub issue

Here is the scenario.

We have one WebJob with 2 functions using QueueTrigger

  • first function is responsible for generating mail content (polling mail-generator queue)
  • second function is sending the generated email (polling mail queue)

The first function outputs a message to the mail queue using as shown below.

 public async Task ProcessMailGeneratorQueue([QueueTrigger("%AzureSettings:MailGeneratorQueueName%")] string message, ILogger logger, ExecutionContext context, [Queue("%AzureSettings:MailQueueName%")] CloudQueue outputQueue)
        {
            MailNotificationModel model = JsonConvert.DeserializeObject<MailNotificationModel>(message, new MailNotificationModelConverter());
            logger.LogInformation($"Generating {model.ReferenceKey}");

            var email = await _mailGenerator.PrepareMailMessage(model);

            CloudQueueMessage queueMessage = new CloudQueueMessage(JsonConvert.SerializeObject(email));
            await outputQueue.AddMessageAsync(queueMessage);
        }

This has been working well for several months. However in the past ~5 weeks ago, we have had several occurrences when the 2nd function is not triggering anymore while the queue contains many messages.

If we restart the web job from the portal (i.e. Stop, Wait for it to stop and Start), the function works properly again.

At this stage, we are not sure how we can further troubleshoot the problem as we have no error traces (we are using AppInsights). I am therefore looking for some advice as how we can further troubleshoot this issue. Is there something we should look after? or something we should add to our logging?

Any help or idea is welcome as this is really annoying and makes the whole solution useless if we cannot rely on it.

Related information

Our volume for this WebJob remain quite “small”. We are not talking about thousands of messages, nor even hundreds. At peak time, we could have couple of dozens of messages per minute.

We are using the following package versions

<PackageReference Include="Microsoft.Azure.WebJobs" Version="3.0.3" /> <PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="3.0.1" /> <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.2" /> <PackageReference Include="Microsoft.Azure.WebJobs.Logging.ApplicationInsights" Version="3.0.3" /> <PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="2.2.0" /> <PackageReference Include="Microsoft.Extensions.Configuration" Version="2.2.0" /> <PackageReference Include="Microsoft.Extensions.Configuration.CommandLine" Version="2.2.0" /> <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="2.2.0" /> <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.2.0" /> <PackageReference Include="WindowsAzure.Storage" Version="9.3.3" />

Work around

At this point, only workaround is to stop/start the WebJob in order for it to dequeue again and we have created a function that retrieves the number of messages in queue and alert us when the number of messages reaches a certain threshold (25 in the last 5 minutes)

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
brettsamcommented, Jul 25, 2019

Note that we’ve also just added more logs to queue triggers (#2277) – when this is released to nuget (which should be fairly soon), please give it a try to see if the logs it generates are helpful. You’ll need to enable Debug-level logs to get them to show up, but the logs will write every time we poll the queue and every time we delay a poll.

As for what storage logs are useful – you can enable diagnostic logs to be written to the $logs blob container on the same storage account. This will tell us everytime we attempted to read/write/delete a queue message. You can find those settings here (note the “Logging” section):

image

1reaction
brettsamcommented, Jul 25, 2019

I’m going to close this as I believe #2277 will help diagnose this problem. If not, or if you get more details, please feel free to re-open and we can take another look.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Queue Trigger Azure Function stoped working
One of my azure function is stopped the queue trigger process for some reason and stops the process and request is moving to...
Read more >
Azure Queue Trigger Function not firing
Now my problem is this is working fine when I run locally . But after deploying the function app and then if I...
Read more >
Queue Trigger within Orchestrator to stop at a specific time
Trigger based in time to consume Queue Item run ones and finish when no have more items to proceed (as Queue Triggered does)...
Read more >
Azure Function and .NET 5: How to retry with Queue Trigger
Create Queue Trigger function ... 1. Create new folder, and new function in it. ... 2. Select QueueTrigger. ... 3. Open with Visual...
Read more >
Azure Queue Trigger Function not firing - appsloveworld.com
Make sure the configuration entity is set up in the portal. I was using a custom config key for the connection name, but...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found