Wildcard topic

For now we provide topic names in a properties file as following:

topics=kcbq-quickstart,app.click

If there are a few topics to deal with, It’s ok to list all the topics in a comma-seperated way. But when you have to deal with hundreds or thousands of topics, it doesn’t help. In those situations, we would like to use a wildcard expression to specify topics.

How about implementing a method to retrive all the topic names matching a given wildcard expression in SchemaRegistrySchemaRetriever class?

[kmillanr]

Hello, any update on the solution?

[kmillanr]

Ah, wasn’t aware of those additional limitations. Thanks for the heads up. Regarding the script idea, if ever get to implement it please let me know, would be great to get to know your approach. I did mine with a timer, every n seconds check the length of the topic list. But that solution is to clunky, I am looking for ways for the script to “passively listen/observe” the list length… if you got any hints that would be awosome. All the best

On Tue, Oct 22, 2019 at 4:09 PM salihoto [e-mail] wrote:

I was think on creating some python script mechanism that would reload the connector whenever the length of the topic list changes within the kafka cluster (create/delete tables go through to the kafka cluster but not through the sink connector). But haven’t gotten around the mechanism yet.

Actually I was thinking about the same solution. By the way, you mentioned altering operations work well, but dropping column or renaming column is not working. I am not sure but this might be about Big query limitations. Another thing, In avro mod,e deleting opeartion doesn’t reflect to kafka but in json mode it’s available as the after part is null.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wepay/kafka-connect-bigquery/issues/92?email_source=notifications&email_token=AHYXGMH756EKVIHQP7XZON3QP4CRZA5CNFSM4DP6R4PKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB54C6A#issuecomment-544981368, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHYXGMDFOY7MMT5STHJZ6N3QP4CRZANCNFSM4DP6R4PA .