How to subscribe/assign a consumer based on a timestamp instead of offset?

Hiya!

I’m looking into building in timestamp based subscription support to KafkaSSE and Wikimedia EventStreams. Using the latest node-rdkafka and a Kafka 1.0 cluster, I can get the message timestamps from the consumer no problem. But, if I’d like to use one of these to assign the consumer to a particular position in the partition, how do I do so? Ideally, I’d be able to just provide the timestamp in the assignment I give to consumer assign(), e.g.

kafkaConsumer.assign([{
    topic: 'my-topic',
    partition: 0,
    timestamp: 1515618893265,
}]

I understand that librdkafka doesn’t work this way (right?). It seems with librdkafka, you must use first make a request for the offset associated with a timestamp, and then use that in your consumer assignment. I could use this if it was supported in node-rdkafka.

Is there a way to do this now that I am missing?

[ottomata]

@webmakersteve thanks again so much for this. It enabled me to implement and deploy: https://wikitech.wikimedia.org/wiki/EventStreams#Timestamp_Historical_Consumption See also https://stream.wikimedia.org/?doc#!/Streams/get_v2_stream_streams

Now folks can publicly consume Wikimedia events (like realtime article edit metadata) and provide a historical timestamp from which they want to start consuming. Very cool.

[webmakersteve]

Will add one to the README