Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error while attempting to create/find topic(s) 'quickstart-avro-offsets'

See original GitHub issue

Hi, Trying to run Kafka Connect using this tutorial. Kafka Connect Docker container stops straight away after running.

[2017-10-05 04:18:52,360] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder) org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'quickstart-avro-offsets' at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:245) at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99) at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126) at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109) at org.apache.kafka.connect.runtime.Worker.start(Worker.java:146) at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:99) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:194) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 3 larger than available brokers: 1 at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213) at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:226) ... 11 more Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 3 larger than available brokers: 1 [2017-10-05 04:18:52,366] DEBUG stopping org.eclipse.jetty.server.Server@7a4446e2 (org.eclipse.jetty.util.component.AbstractLifeCycle) [2017-10-05 04:18:52,368] DEBUG Graceful shutdown org.eclipse.jetty.server.Server@7a4446e2 by (org.eclipse.jetty.server.Server) [2017-10-05 04:18:52,369] DEBUG stopping ServerConnector@68309532{HTTP/1.1}{0.0.0.0:28083} (org.eclipse.jetty.util.component.AbstractLifeCycle) [2017-10-05 04:18:52,369] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect)

I do have the Kafka topic though. Any ideas?

Issue Analytics

State:
Created 6 years ago
Comments:11 (1 by maintainers)

Top GitHub Comments

9reactions

filipambcommented, Dec 19, 2017

Problem was caused by wrong properties included into tutorial. Proper command (which worked for me) to launch kafka connect docker is:

docker run -d
–name=kafka-connect-avro
–net=host
-e CONNECT_BOOTSTRAP_SERVERS=localhost:29092
-e CONNECT_REST_PORT=28083
-e CONNECT_GROUP_ID=“quickstart-avro”
-e CONNECT_CONFIG_STORAGE_TOPIC=“quickstart-avro-config”
-e CONNECT_OFFSET_STORAGE_TOPIC=“quickstart-avro-offsets”
-e CONNECT_STATUS_STORAGE_TOPIC=“quickstart-avro-status”
-e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
-e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
-e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
-e CONNECT_KEY_CONVERTER=“io.confluent.connect.avro.AvroConverter”
-e CONNECT_VALUE_CONVERTER=“io.confluent.connect.avro.AvroConverter”
-e CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=“http://localhost:8081”
-e CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=“http://localhost:8081”
-e CONNECT_INTERNAL_KEY_CONVERTER=“org.apache.kafka.connect.json.JsonConverter”
-e CONNECT_INTERNAL_VALUE_CONVERTER=“org.apache.kafka.connect.json.JsonConverter”
-e CONNECT_REST_ADVERTISED_HOST_NAME=“localhost”
-e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG
-v /tmp/quickstart/file:/tmp/quickstart
-v /tmp/quickstart/jars:/etc/kafka-connect/jars
confluentinc/cp-kafka-connect:4.0.0

I changed or added bolded parts of this command

0reactions

junaidatiquecommented, Jun 10, 2022

For future user who are using debezium, I was using “topic.creation.default.replication.factor” = 3, where as I had only 1 broker. So check your replication factor.

Top Results From Across the Web

Not able to run Kafka Connect in distributed mode - Error ...

When starting up Kafka Connect in distributed mode, it will try to create or expect several topics to be present in Kafka.

Error while attempting to create/find topic(s) 'connect-offsets'

Using datamountaineer/kafka-connect-cassandra:1.1.0 docker image. I have the following configs set in connector properties: ...

Kafka Connect should check for existence of internal topics ...

ConnectException : Error while attempting to create/find topic(s) 'connect-offsets' connect | at org.apache.kafka.connect.util.TopicAdmin.

Migrating Data from Legacy Systems with Apache Kafka— part 1

The next step is to create the topics that Kafka Connect Container will use for it base functionality, these are: Offsets, Config and...

debezium/user - Gitter

DistributedHerder] org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'prod-wms2-oms-debezium-connect-offsets' ...