Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Bulk Helpers : Add a max_wait_time field to trigger bulk after a period

See original GitHub issue

Hi, Is it possible to add a max_wait_time to the streaming_bulk and parrallel_bulk helpers to trigger a bulk after a period event if less than chunk_size or max_chunk_bytes documents are generated ? My action method is a kafka consumer, and when a reach the end of the topics i consume, I have to wait chunk_size messages before next bulk.

Issue Analytics

State:
Created 7 years ago
Reactions:2
Comments:5 (2 by maintainers)

Top GitHub Comments

2reactions

rohitgcscommented, Mar 5, 2018

@seuf if you were able to implement this I would love to know how you did it because I don’t see how to put a timeout on a kafka consumer (?)

@HonzaKral @fxdgear is there any particular reason we’re not implementing a timer of some sort to the bulk api? Because a kafka consumer is something that stays on a while True loop forever - otherwise there would be a delay in the overall process to get it up and running again.

Something like a timeout=True, max_wait_time =60 or timeout=False Just trying to understand the issue since it will definitely be a popular use case for streamers using Kafka, Airflow, etc.

0reactions

seufcommented, Mar 5, 2018

@rohitgcs Sorry I don’t remember I had sucess to implement this. the projects have moved direction since last year ans this lib is not more used.

Top Results From Across the Web

Bulk Apex Triggers Unit - Trailhead - Salesforce

The following sections demonstrate the main ways of bulkifying your Apex code in triggers: operating on all records in the trigger, and performing...

I want to write trigger to automatically upload files in bulk on ...

You will want to create a CDL record that uses the AccountId as the LinkedEntityId, and then pull the ContentDocumentId from whereever it's ......

Writing Bulk Triggers for Salesforce.com - Jeff Douglas

According to the Apex documentation, a trigger is an Apex script that executes before or after insert, update, or delete events occur, such...

2 Initial Bulk Match and Load Tool - Oracle Help Center

The processing properties described in the following table configure how the IBML Tool processes data. In these properties, you define a name for ......

Documentation - Apache Kafka

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you ... Update server.properties on all brokers and add the...