Preventing concurrent runs/builds?

Hello. I am working on a project where we used to have a bunch of cron jobs running in Jenkins, but business needs have dictated we need to migrate away from this resource and into an already-existing k8s infrastructure, whose devops pipeline unfortunately does not allow for the management of cronjobs. As a result, next best possible thing is to stand up an application that manages the cronjobs as an abstraction, which is what we’re doing with this library.

My question is, we have a certain job that runs every half hour. However, because of the data that is being handled, it may take longer than 30minutes to execute this job. In the past, in Jenkins, we were able to simply include options { disableConcurrentBuilds() } in our Jenkinsfile, and the next job would only run after the previous execution terminated. However, i have done some digging through the docs and i haven’t found anything that would allow for this with cron. Perhaps I am missing something? would love some additional insight on this, as well as any workarounds if this functionality does not exist out-of-the-box.

Thanks!

[claytongulick]

I just realized that my last comment was a bit sparse on details, so here’s how I’m doing it.

In a file lock.js, I have a simple utility that looks like:

const lockfile = require('proper-lockfile');

module.exports = async function(name, fn) {
    let release;
    let execution_error;
    try {
        release = await lockfile.lock(name);
    }
    catch(err) {
        console.error(`Failed to aquire lock: ${name}, error: ${err}`);
        return;
    }

    try {
        await fn();
    }
    catch(err) {
        console.error(`Failed to execute lock function: ${err}`);
        execution_error = err;
    }

    try {
        await release();
    }
    catch(err) {
        console.err(`Failed to release lock: ${name}, error: ${err}`);
        throw err;
    }

    if(execution_error)
        throw execution_error;
}

This will give you a wrapper function that’ll let you do an exclusive lock on any function, and propagate errors back up to the calling job.

The way I structure my node-cron jobs is a single job per file, in a jobs folder (with an index.js). When I import index.js, all the jobs run. I also, however, want to be able to run jobs on-demand, from the cli. So my job files look like this:

let CronJob = require('cron').CronJob;
let path = require('path');
let Sync = require('../etl/legacy/sync');
let winston = require('winston');

let job_name = path.basename(__filename);

async function job_function() {
    winston.info(`Starting job: ${job_name}`)
    try {
        await lock(__filename, async () => {
            await Sync.run({families: true, new_only: true});
        });
    }
    catch(err) {
        winston.error(err);
    }
}


if(require.main == module) {
    job_function();
}
else {
    let job = new CronJob({
        cronTime: '0 * * * * *', //every minute
        onTick: job_function,
    });

    job.start();
    module.exports = job;
}

Where in this case Sync is an external helper library that handles some data synchronization, it can be any function though.

Also, just a FYI, in my index.js, I configure the winston transports and logging levels so that I get alerted in multiple ways as soon as a job fails for any reason.

[claytongulick]

Just want to jump in here with my $0.02.

First, @ncb000gt I moved from agenda to this library. I had a bunch of problems with agenda, especially with multiple worker processes. Agenda-dash (the UI) was written as an experimental first project by the author, and they did a great job for a first project, but it wasn’t at a quality level that I could use in production. YMMV.

Second, to help @ethanfoxIBM and others that are struggling with this, the scope of the issue for overlapping job runs is a long-standing problem (40 years) that has several strategies, but the most common, mature and solid solution I’ve seen is the classic lockfile (this is how pretty much everything in unix works).

Node has a robust and simple lockfile library that should make solving this problem a breeze on any platform and file system (even network file systems) https://github.com/moxystudio/node-proper-lockfile

HTH!