Would like to handle jobs in batch

Hi !

Something i was wondering about, could we add batch support for worker ? Let say i have 500 jobs which each of them fetch data to an external API or database for example. Batching them in a single request would help reduce the load on these external resources and also reduce latency in general. Example :

const zbBatchWorker = zbc.createBatchWorker(
    'test-worker',
    'demo-service',
    {
        batch: 50, // # of jobs per batch
        timeout: 1000, // or 1 sec
    },
    async (payloads, complete) => {
        const ids = jobs.map((j) => j.id);
        const users = await fetch(`someUrl/user?ids=${ids.join(',')}`);

        for(const [key] of Object.keys(ids)) {
            const index = Number(key);
            const user = users[index];

            try {
                // ...Do some work... 

                if(...) {
                    complete.success(index, ...)
                } else {
                    complete.failure(index, ...)
                }
            } catch(error) {
                complete.error(index, ...)
            }
        }

        /*
            When the batch is done,
            this commit each job and
            call the right procedure (grpc)
            base on 'complete'
        */
        await complete.done();
    }
);

I can pass the number of jobs per batch or a timeout after which the batch is handle anyway. That said, it could be link to the maxActiveJobs propertie.

[jbeaudoin11]

I think it’s exactly what i want in term of functionality. Thanks !

[jwulf]

Will be out this week in the 0.23.0-alpha.1 release.

Here are the docs from the README:

The ZBBatchWorker Job Worker

The ZBBatchWorker Job Worker batches jobs before calling the job handler. Its fundamental differences from the ZBWorker are:

• Its job handler receives an array of one or more jobs.
• The jobs have success, failure, error, and forwarded methods attached to them.
• The handler is not invoked immediately, but rather when enough jobs are batched, or a job in the batch is at risk of being timed out by the Zeebe broker.

You can use the batch worker if you have tasks that benefit from processing together, but are not related in the BPMN model.

An example would be a high volume of jobs that require calls to an external system, where you have to pay per call to that system. In that case, you may want to batch up jobs, make one call to the external system, then update all the jobs and send them on their way.

The batch worker works on a first-of batch size or batch timeout basis.

You must configure both jobBatchMinSize and jobBatchMaxTime. Whichever condition is met first will trigger the processing of the jobs:

• Enough jobs are available to the worker to satisfy the minimum job batch size;
• The batch has been building for the maximum amount of time - “we’re doing this now, before the earliest jobs in the batch time out on the broker”.

You should be sure to specify a timeout for your worker that is jobBatchMaxTime plus the expected latency of the external call plus your processing time and network latency, to avoid the broker timing your batch worker’s lock and making the jobs available to another worker. That would defeat the whole purpose.

Here is an example of using the ZBBatchWorker:

import { API } from './lib/my-awesome-external-api'
import { ZBClient, BatchedJob } from 'zeebe-node'

const zbc = new ZBClient()

// Helper function to find a job by its key
const findJobByKey = jobs => key => jobs.filter(job => job.jobKey === id)?.[0] ?? {}

const handler = async (jobs: BatchedJob[], worker: ZBBatchWorker) => {
    worker.log("Let's do this!")
    const {jobKey, variables} = job
    // Construct some hypothetical payload with correlation ids and requests
    const req = jobs.map(job => ({id: jobKey, data: variables.request}))
    // An uncaught exception will not be managed by the library
    try {
        // Our API wrapper turns that into a request, and returns
        // an array of results with ids
        const outcomes = await API.post(req)
        // Construct a find function for these jobs
        const getJob = findJobByKey(jobs)
        // Iterate over the results and call the succeed method on the corresponding job,
        // passing in the correlated outcome of the API call
        outcomes.forEach(res => getJob(res.id)?.success(res.data))
    } catch (e) {
        jobs.forEach(job => job.failure(e.message))
    }
}

const batchWorker = zbc.createBatchWorker({
    taskType: 'get-data-from-external-api',
    taskHandler: handler,
    jobBatchMinSize: 10, // at least 10 at a time
    jobBatchMaxTime: 60, // or every 60 seconds, whichever comes first
    timeout: 80 // 80 second timeout means we have 20 seconds to process at least
})