question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cluster workers not being able to pick up Jobs / double pick up

See original GitHub issue

The following program will create two cluster workers, then one job task. The job task will be marked as failed immediately. Every second the job is marked back as inactive which should be picked up again by a single worker. However occasionally more than one worker is picking up the job cause it to be executed twice.

Is there something wrong with my logic or is this an issue with the cluster implementation in kue?

var kue = require('kue');
var jobs = kue.createQueue();
var cluster = require('cluster');

if(cluster.isMaster){
    cluster.fork();
    cluster.fork();
    jobs.create('test').save();

    setInterval(function () {
        kue.Job.rangeByState( 'failed', 0, -1, 'asc', function( err, jobs ) {
            if(err){
                console.log(err);
            }
            else{
                jobs.forEach( function ( job ) {
                    job.state('inactive').save();
                });
            }
        });
    }, 1000);
}

if(cluster.isWorker){
    jobs.process('test', function ( job, done ) {
        console.log('worker' , cluster.worker.id + ' has control of: ' + job.id + ' | ' +  new Date());
        done('failed');
    });
}
else{
    jobs.on('job failed', function ( errorMessage ) {
        console.log('Job failed');
    })
}

When running the following program the jobs are eventually are not picked up at all.

Note: The done call is never called in this example which causes the state of the jobs to be changed from active to inactive. Instead of failed to inactive like the previous example.

var kue = require('kue');
var jobs = kue.createQueue();
var cluster = require('cluster');

if(cluster.isMaster){
    cluster.fork();
    cluster.fork();
    jobs.create('test').save();

    setInterval(function () {
        kue.Job.rangeByState( 'active', 0, -1, 'asc', function( err, jobs ) {
            if(err){
                console.log(err);
            }
            else{
                jobs.forEach( function ( job ) {
                    job.state('inactive').save();
                });
            }
        });
    }, 1000);
}

if(cluster.isWorker){
    jobs.process('test', function ( job, done ) {
        console.log('worker' , cluster.worker.id + ' has control of: ' + job.id + ' | ' +  new Date());
    });
}
else{
    jobs.on('job failed', function ( errorMessage ) {
        console.log('Job failed');
    })
}

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Reactions:1
  • Comments:10

github_iconTop GitHub Comments

1reaction
JperFcommented, Nov 1, 2018

Hey @victornikitin,

Unfortunately I have not looked into this for more than 2 years. So I am not sure where Kue’s code stands nor if this is still an issue.

1reaction
behradcommented, Jul 14, 2016
queue = createQueue()
queue.rangeByState...
Read more comments on GitHub >

github_iconTop Results From Across the Web

Cluster with 2 or 3 workers, jobs are not getting "distributed ...
However I always see the job being picked up by the first worker, the other one(s) aren't picking up the job. What am...
Read more >
NodeJS Agenda scheduler: cluster with 2 or 3 workers, jobs ...
Now suppose I have 2 workers (processes) and I call "agenda.now()" from worker 1, then it can be picked up (processed) by any...
Read more >
Best practices: Cluster configuration | Databricks on AWS
Learn best practices when creating and configuring Databricks clusters.
Read more >
Best practices: Cluster configuration - Azure Databricks
Learn best practices when creating and configuring Azure Databricks clusters.
Read more >
Fine Parallel Processing Using a Work Queue - Kubernetes
In this example, as each pod is created, it picks up one unit of ... to run this tutorial on a cluster with...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found