Transactional producer

The transactional producer allows an application to send messages to multiple partitions (and topics) atomically. Each transactional producer has a unique transactionalId, which is used to enable transaction recovery across multiple sessions.

There can be only one open transaction per instance of the producer.

Proposed API:

const producer = kafka.producer({ transactional: true })

const run = async () => {
  await producer.connect()
  const tx = await producer.tx()

  try {
    for (let i = 0; i < 100; i++) {
      await tx.add({
        topic,
        messages: [{ key: `key-${i}`, value: `value-${i}` }]
      })
    }
 
    await tx.commit()
  } catch (e) {
    // Investigate class of errors where the producer can't recover, i.e: ProducerFencedException
    await tx.abort()
  }

  await producer.disconnect()
}

Workflow:

await producer.connect():

• Should call initProducerId to initialize the producer, since it can only happen once per instance it could be combined with connect
• Add a new log line to indicate the use of a transactional producer

const tx = await producer.tx():

I still have mixed feeling about this design, but the idea is to create a transaction object where the user will be able to add records, commit, and abort.

• It should check for other producers with the same ID
• In memory flag to indicate that a transaction was created. There can be only one open transaction per instance of the producer.

await tx.add({ ... }):

Adds the records to some record accumulator

await tx.commit():

• Calls AddPartitionsToTxn
• Calls EndTxnRequest with COMMIT

await tx.abort():

• Clear the in-memory accumulator
• Calls EndTxnRequest with ABORT

We have to investigate the errors that cause the consumer to disconnect instead of abort. From the Java client, we have: ProducerFencedException, OutOfOrderSequenceException, and AuthorizationException where the authorization error is specific to the current transactionalId

await producer.disconnect():

Should close the connection as usual

Open questions: Do we need guidelines for the transactionalId?

Should we auto-generate the transactionId if it’s not defined?

NOTE:

The API design is still a draft

Depends on #165 #166 #167 #170

@Nevon @ianwsperber ^

[ianwsperber]

@tulios Created #200 so we can separately track the idempotent producer work

[tulios]

I’ve checked other libraries, and everybody is following this pattern. Let’s follow suit.