Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Named Kafka cluster configurations for topic blueprint

See original GitHub issue

Kafka connection details can either be provided during installation (as simply, bootstrap.servers) or explicitly within a blueprint using advanced topic configuration. In addition, we should let users define a reusable named Kafka connection configuration for as many clusters as they like. The blueprint can then reference these configurations by a simple name.

A Kafka connection configuration will include the following information:

  • name
  • bootstrap.servers connection string
  • producer client config
  • consumer client config

During an application deployment the cloudflow operator will first lookup any referenced Kafka connection configurations and propagate them along to streamlets as they’re created.

If a blueprint doesn’t reference any Kafka cluster configurations then the current behaviour will be preserved. The order of precedence to load a configuration in this case will be:

  • use the default Kafka cluster configuration, if it exists
  • use the advanced connection info provided in the blueprint, if it exists
  • use the bootstrap.servers configuration passed to the operator during install

Before deployment the CLI will verify that secrets containing any referenced Kafka cluster configurations exist.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:10 (10 by maintainers)

github_iconTop GitHub Comments

seglocommented, Sep 10, 2020

My mistake. Helm can create secrets, but I was getting tripped over the fact that it expects the data field to be base64. Using the stringData field lets you provide non-encoded data.

RayRoestenburgcommented, Sep 4, 2020

Looks like Robert did already write some things down in #669 maybe combine this @seglo.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Documentation - Apache Kafka
A topic is a category or feed name to which messages are published. For each topic, the Kafka cluster maintains a partitioned log...
Read more >
Kafka Broker Configurations for Confluent Platform
This topic provides configuration parameters available for Confluent Platform. The Apache Kafka® broker configuration parameters are organized by order of ...
Read more >
Composing applications using blueprints
... Named Kafka Clusters; Advanced Topic Configuration; Blueprint Verification. In Cloudflow, a streaming application is composed of connected Streamlet s.
Read more >
kafka-configs | CDP Public Cloud - Cloudera Documentation
In an environment managed by Cloudera Manager the kafka-configs tool can be used to set, describe, or delete topic properties. ... Cloudera does...
Read more >
Kafka Architecture - Cloudurable
Zookeeper sends changes of the topology to Kafka, so each node in the cluster knows when a new broker joined, a Broker died,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found