Julie doesn't like Placement contraints

Describe the bug I am trying to use Julie together with replica placement contraints and that seems to raise some errors.

To Reproduce

My brokers have the following config:

confluent.log.placement.constraints={"version": 1,"replicas":[{"count": 1,"constraints":{"rack": "rack-1"}},{"count": 1,"constraints":{"rack": "rack-2"}},{"count": 1,"constraints":{"rack": "rack-3"}}]}

So if I create a topic test1 manually - the constraints will be respected:

$ kafka-topics --bootstrap-server broker1:9092 --create --topic testcontstraints
Created topic testcontstraints.

$ kafka-topics --bootstrap-server broker1:9092 --describe --topic testcontstraints
Topic: testcontstraints	PartitionCount: 1	ReplicationFactor: 3	Configs: compression.type=snappy,min.insync.replicas=2,segment.bytes=1073741824,confluent.placement.constraints={"version":1,"replicas":[{"count":1,"constraints":{"rack":"rack-1"}},{"count":1,"constraints":{"rack":"rack-2"}},{"count":1,"constraints":{"rack":"rack-3"}}],"observers":[]}
	Topic: testcontstraints	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Offline: 

However if I create now a topic via julie using the following descriptor:

descriptor.yaml

---
context: "julie"
source: "test"
projects:
  - name: "schulung"
    topics:
      - name: "constraints"

Sadly the constraints seem not to be respected:

$ kafka-topics --bootstrap-server broker1:9092 --describe --topic julie.test.schulung.constraints

Topic: julie.test.schulung.constraints	PartitionCount: 3	ReplicationFactor: 2	Configs: compression.type=snappy,min.insync.replicas=2,segment.bytes=1073741824
	Topic: julie.test.schulung.constraints	Partition: 0	Leader: 2	Replicas: 2,3	Isr: 2,3	Offline: 
	Topic: julie.test.schulung.constraints	Partition: 1	Leader: 3	Replicas: 3,1	Isr: 3,1	Offline: 
	Topic: julie.test.schulung.constraints	Partition: 2	Leader: 1	Replicas: 1,2	Isr: 1,2	Offline: 

If I go further and try to force a constraint on the topic descriptor like this:

---
context: "julie"
source: "test"
projects:
  - name: "schulung"
    topics:
      - name: "constraints2"
        config:
          confluent.placement.constraints: '{"version": 1,"replicas":[{"count": 1,"constraints":{"rack": "rack-1"}},{"count": 1,"constraints":{"rack": "rack-2"}},{"count": 1,"constraints":{"rack": "rack-3"}}]}'

I get this error

julie-ops --brokers broker1:9092 --clientConfig prod.properties --topology descriptor4.yaml
[INFO ] 2021-03-29 23:31:56.867 [main] TopologyBuilderAdminClientBuilder - Connecting AdminClient to broker:9092
[INFO ] 2021-03-29 23:31:56.867 [main] TopologyBuilderAdminClientBuilder - Connecting AdminClient to broker1:9092
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.admin.AdminClientConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[ERROR] 2021-03-29 23:31:59.632 [main] TopologyBuilderAdminClient - java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
[ERROR] 2021-03-29 23:31:59.632 [main] TopologyBuilderAdminClient - java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
[ERROR] 2021-03-29 23:31:59.634 [main] ExecutionPlan - Something happen running action {
  "Action" : "create",
  "Operation" : "com.purbon.kafka.topology.actions.topics.SyncTopicAction",
  "Topic" : "julie.test.schulung.constraints2"
}
java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:196) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.actions.topics.SyncTopicAction.syncTopic(SyncTopicAction.java:60) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.actions.topics.SyncTopicAction.run(SyncTopicAction.java:46) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.ExecutionPlan.execute(ExecutionPlan.java:88) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.ExecutionPlan.run(ExecutionPlan.java:69) [julie-ops.jar:?]
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:189) [julie-ops.jar:?]
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:201) [julie-ops.jar:?]
	at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:196) [julie-ops.jar:?]
	at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:144) [julie-ops.jar:?]
	at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:134) [julie-ops.jar:?]
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ~[julie-ops.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ~[julie-ops.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) ~[julie-ops.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createAllTopics(TopologyBuilderAdminClient.java:202) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:191) ~[julie-ops.jar:?]
	... 9 more
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
[ERROR] 2021-03-29 23:31:59.634 [main] ExecutionPlan - Something happen running action {
  "Action" : "create",
  "Operation" : "com.purbon.kafka.topology.actions.topics.SyncTopicAction",
  "Topic" : "julie.test.schulung.constraints2"
}
java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:196) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.actions.topics.SyncTopicAction.syncTopic(SyncTopicAction.java:60) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.actions.topics.SyncTopicAction.run(SyncTopicAction.java:46) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.ExecutionPlan.execute(ExecutionPlan.java:88) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.ExecutionPlan.run(ExecutionPlan.java:69) [julie-ops.jar:?]
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:189) [julie-ops.jar:?]
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:201) [julie-ops.jar:?]
	at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:196) [julie-ops.jar:?]
	at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:144) [julie-ops.jar:?]
	at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:134) [julie-ops.jar:?]
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ~[julie-ops.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ~[julie-ops.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) ~[julie-ops.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createAllTopics(TopologyBuilderAdminClient.java:202) ~[julie-ops.jar:?]
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:191) ~[julie-ops.jar:?]
	... 9 more
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
Exception in thread "main" java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:196)
	at com.purbon.kafka.topology.actions.topics.SyncTopicAction.syncTopic(SyncTopicAction.java:60)
	at com.purbon.kafka.topology.actions.topics.SyncTopicAction.run(SyncTopicAction.java:46)
	at com.purbon.kafka.topology.ExecutionPlan.execute(ExecutionPlan.java:88)
	at com.purbon.kafka.topology.ExecutionPlan.run(ExecutionPlan.java:69)
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:189)
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:201)
	at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:196)
	at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:144)
	at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:134)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createAllTopics(TopologyBuilderAdminClient.java:202)
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:191)
	... 9 more
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.

[purbon]

I am actually happy to change it if useful for others! 😃

Missatge de Fobhep @.***> del dia dt., 30 de març 2021 a les 11:17:

ok so - I would then open another issue to discuss the defaults 😅 and we can leave this one open for the constraints

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kafka-ops/julie/issues/241#issuecomment-810059859, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQXPE4VCJNJMPFULVOJH3TGGJKFANCNFSM42AOGSXQ .

– Pere Urbon-Bayes Software Architect https://twitter.com/purbon https://www.linkedin.com/in/purbon/

[purbon]

I did a test with the latest version of Julie, using

---
context: "o"
projects:
  - name: "f"
    consumers:
      - principal: "User:NewApp2"
    topics:
      - name: "t"
        config:
          confluent.placement.constraints:  "{\"version\":1,\"replicas\":[{\"count\":1,\"constraints\":{\"rack\":\"rack-1\"}},{\"count\":1,\"constraints\":{\"rack\":\"rack-2\"}}],\"observers\":[]}"

and you can see the placement constraints properly set.

$ docker exec kafka kafka-topics --bootstrap-server kafka:29092 \                                                        2.7.0
                  --describe --topic o.f.t
Topic: o.f.t	TopicId: dJImanTbSd2sbUjLVDMoVA	PartitionCount: 1	ReplicationFactor: 2	Configs: confluent.placement.constraints={"version":1,"replicas":[{"count":1,"constraints":{"rack":"rack-1"}},{"count":1,"constraints":{"rack":"rack-2"}}],"observers":[]}
	Topic: o.f.t	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2	Offline:

I do have the feeling this was caused when Julie use to send a specific replication factor, but now when not configured, as seen in the example, all set as expected.

As you can see when using this topology:

---
context: "o"
projects:
  - name: "f"
    consumers:
      - principal: "User:NewApp2"
    topics:
      - name: "b"
        config:
          replication.factor: "1"
          confluent.placement.constraints:  "{\"version\":1,\"replicas\":[{\"count\":1,\"constraints\":{\"rack\":\"rack-1\"}},{\"count\":1,\"constraints\":{\"rack\":\"rack-2\"}}],\"observers\":[]}"

that causes this error:

re set. Both cannot be used at the same time.
java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:200)
	at com.purbon.kafka.topology.actions.topics.CreateTopicAction.createTopic(CreateTopicAction.java:41)
	at com.purbon.kafka.topology.actions.topics.CreateTopicAction.run(CreateTopicAction.java:36)
	at com.purbon.kafka.topology.ExecutionPlan.execute(ExecutionPlan.java:126)
	at com.purbon.kafka.topology.ExecutionPlan.run(ExecutionPlan.java:101)
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:212)
	at com.purbon.kafka.topology.JulieOps.run(JulieOps.java:227)
	at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:212)
	at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:161)
	at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:147)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createAllTopics(TopologyBuilderAdminClient.java:211)
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClient.createTopic(TopologyBuilderAdminClient.java:193)
	... 9 more
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Both replicationFactor and confluent.placement.constraints are set. Both cannot be used at the same time.

as seen in this issue.

closing this issue for now, feel free to reopen if necessary.