Consumer Groups in RBAC mode

I deployed the following topology.yaml using the latest docker image:

docker images | grep -i topology
purbon/kafka-topology-builder latest e477795a8c07  10 days ago  468MB

---
context: "test-topo-builder"
source: "test"
projects:
  - name: "project"
    consumers:
      - principal: "User:barnie"
        group: agroup
    producers:
      - principal: "User:barnie"
    topics:
      - name: "topic"
        config:
          replication.factor: "1"
          num.partitions: "1"

Afterwards I try to produce to the topic using first User:barnie and afterwards a second user, that is not authorized:

kafka-console-producer --bootstrap-server localhost:9094 --producer.config barnie.properties --topic
test-topo-builder.test.project.topic
> 1 2 3 4

Works like a charme 👍

kafka-console-producer --bootstrap-server localhost:9094 --producer.config eva.properties --topic
test-topo-builder.test.project.topic
> 1 2 3 4
[2020-10-05 16:11:26,695] WARN [Producer clientId=console-producer]
Error while fetching metadata with correlation id 3 :
{test-topo-builder.test.project.topic=TOPIC_AUTHORIZATION_FAILED}
(org.apache.kafka.clients.NetworkClient)
[2020-10-05 16:11:26,697] ERROR [Producer clientId=console-producer]
Topic authorization failed for topics
[test-topo-builder.test.project.topic] (org.apache.kafka.clients.Metadata)
[2020-10-05 16:11:26,698] ERROR Error when sending message to topic
test-topo-builder.test.project.topic with key: null, value: 7 bytes with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not
authorized to access topics: [test-topo-builder.test.project.topic]

Does not work as intended 👍

I am trying now to consume the topic:

kafka-console-consumer --bootstrap-server localhost:9094 --consumer.config barnie.properties --topic
test-topo-builder.test.project.topic --consumer-property group.id=agroup

[2020-10-05 16:15:27,520] ERROR Error processing message, terminating
consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.GroupAuthorizationException: Not
authorized to access group: agroup
Processed a total of 0 messages

Does not work, because kafka-topology-builder is NOT creating a Resource-Owner Role that maps the User to the consumer-group.

After creating such a mapping the consuming works:

kafka-console-consumer --bootstrap-server localhost:9094 --consumer.config barnie.properties --topic
test-topo-builder.test.project.topic --consumer-property group.id=agroup
--from-beginning

1
2
3
4

If I use a different consumer-group “bgroup” to consume, I get the same error as above:

kafka-console-consumer --bootstrap-server localhost:9094 --consumer.config barnie.properties --topic
test-topo-builder.test.project.topic --consumer-property group.id=bgroup

[2020-10-05 16:15:27,520] ERROR Error processing message, terminating
consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.GroupAuthorizationException: Not
authorized to access group: agroup
Processed a total of 0 messages

I again gave User:barnie the role Resource Owner for the consumer group bgroup. However now consuming works! - I think it shouldn’t?

I am also a bit suspicious, because I get no ACLs at all displayed: (alice is a super-user)

kafka-acls --bootstrap-server localhost:9094 --command-config alice.properties --list --topic '*'

[2020-10-05 16:32:20,397] WARN The configuration
'sasl.login.callback.handler.class' was supplied but isn't a known
config. (org.apache.kafka.clients.admin.AdminClientConfig)
[2020-10-05 16:32:20,397] WARN The configuration 'sasl.jaas.config' was
supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig)

So I have in summary 2 questions:

• How do I create a Role assignment “ResourceOwner of a Group” for a user using topology builder. Because imho without that I can not consume - right?

• Why can bgroup read my topic. In my understanding there should be an ACL preventing my group “bgroup” to read the topic, but there seems to be none. Or am I understanding all of this wrong? 🤔 😄

[purbon]

true, thanks for finding it.

fix is on its way.

Missatge de Fobhep [e-mail] del dia dt., 6 d’oct. 2020 a les 14:19:

I tested again with your newest version and I think there might still either a bug or a mistake on my side 😉

Checking the Rolebindings

confluent iam rolebinding list --kafka-cluster-id EFsiypqXS1SJt-0Tk7KdBg --principal User:barnie Principal | Role | ResourceType | Name | PatternType ±------------±---------------±-------------±-------------------------------------±------------+

Deploying the descriptor:

---

context: “test-topo-builder” source: “test” projects:

• name: “project” consumers:
  • principal: “User:barnie” group: agroup producers:
  • principal: “User:barnie” topics:
  • name: “topic” config: replication.factor: “1” num.partitions: “1”

After running the descriptor:

confluent iam rolebinding list --kafka-cluster-id EFsiypqXS1SJt-0Tk7KdBg --principal User:barnie Principal | Role | ResourceType | Name | PatternType ±------------±---------------±-------------±-------------------------------------±------------+ User:barnie | DeveloperRead | Topic | test-topo-builder.test.project.topic | LITERAL User:barnie | DeveloperWrite | Topic | test-topo-builder.test.project.topic | LITERAL User:barnie | ResourceOwner | Topic | agroup | LITERAL

Shouldn’t the most bottom ResourceType be “Group” ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/purbon/kafka-topology-builder/issues/95#issuecomment-704229962, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQXPDU7BFANFVDSCCXX6DSJMDMHANCNFSM4SE5OWVQ .

– Pere Urbon-Bayes Software Architect https://twitter.com/purbon https://www.linkedin.com/in/purbon/

[purbon]

release artifact including a solution for this and others has been released https://github.com/purbon/kafka-topology-builder/releases/tag/1.0.0-rc.2

as well in docker hub with the tag 1.0.0-rc.2