question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Multiple SortedSet in Redis sink not working

See original GitHub issue

Issue Guidelines

Please review these questions before submitting any issue?

What version of the Stream Rector are you reporting this issue for?

1.0.0

Are you running the correct version of Kafka/Confluent for the Stream reactor release?

Yes, kafka 2.12-1.0.0

Have you consulted our FAQs page first?

Yes, I have.

Do you have a supported version of the data source/sink .i.e Cassandra 3.0.9?

I hope so but the exception occurs when connect’s configuration is parsed before trying to connect to redis.

Have you read the docs?

Yes, I have.

What is the expected behaviour?

Connect to redis instance.

What was observed?

An exception occurs when a kcql from manual (SELECT temperature, humidity FROM sensorsTopic PK sensorID STOREAS SortedSet(score=timestamp)) is used:

[2018-05-08 15:05:56,586] ERROR WorkerSinkTask{id=redis-signals-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
        at com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkTask$$anonfun$4.apply(RedisSinkTask.scala:66)
        at com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkTask$$anonfun$4.apply(RedisSinkTask.scala:66)
        at scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
        at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
        at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
        at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
        at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
        at com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkTask.start(RedisSinkTask.scala:66)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:267)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:163)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2018-05-08 15:05:56,589] ERROR WorkerSinkTask{id=redis-sink-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)

I can run sink in cache and sorted set mode but not in multiple sorted set mode. When trying to set a prefix: INSERT INTO SENSOR- SELECT temperature, humidity FROM sensorsTopic PK sensorID STOREAS SortedSet(score=timestamp) there is a different exception:

java.lang.AssertionError: assertion failed: They keyword PK (Primary Key) is not supported in Redis INSERT_SS mode. Please review the KCQL syntax of connector
        at scala.Predef$.assert(Predef.scala:170)
        at com.datamountaineer.streamreactor.connect.redis.sink.writer.RedisInsertSortedSet$$anonfun$3.apply(RedisInsertSortedSet.scala:50)
        at com.datamountaineer.streamreactor.connect.redis.sink.writer.RedisInsertSortedSet$$anonfun$3.apply(RedisInsertSortedSet.scala:45)
        at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
        at com.datamountaineer.streamreactor.connect.redis.sink.writer.RedisInsertSortedSet.<init>(RedisInsertSortedSet.scala:45)
        at com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkTask$$anonfun$start$2.apply(RedisSinkTask.scala:77)
        at com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkTask$$anonfun$start$2.apply(RedisSinkTask.scala:75)
        at scala.Option.map(Option.scala:146)
        at com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkTask.start(RedisSinkTask.scala:75)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:267)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:163)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

What is your Connect cluster configuration (connect-avro-distributed.properties)?

I’ve tried it in a standalone mode.

What is your connector properties configuration (my-connector.properties)?

name=redis-sink connector.class=com.datamountaineer.streamreactor.connect.redis.sink.RedisSinkConnector tasks.max=1 topics=sensorsTopic connect.redis.host=127.0.0.1 connect.redis.port=6379 connect.redis.kcql=SELECT temperature, humidity FROM sensorsTopic PK sensorID STOREAS SortedSet(score=timestamp)

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
rdebokxcommented, May 29, 2018

Alternatively you can use the following dependencies in kafka-connect-redis/build.gradle and rebuild the shadowJar.

dependencies {
        compile "org.apache.calcite:calcite-linq4j:$link4jVersion"
        compile "redis.clients:jedis:$jedisVersion"
        testCompile "com.google.code.gson:gson:$gsonVersion"
        testCompile "com.orange.redis-embedded:embedded-redis:$embeddedRedisVersion"
    }

Where I used link4jVersion = "1.8.0"

I’m not sure if this is the desired solution, however, so I have not submitted this as a PR.

0reactions
stheppicommented, May 29, 2018

I would add this dependency in the main project gradle such that all sinks/sources get this

Read more comments on GitHub >

github_iconTop Results From Across the Web

Multiple SortedSet in Redis sink not working #437 - GitHub
I can run sink in cache and sorted set mode but not in multiple sorted set mode. When trying to set a prefix:...
Read more >
Redis: Group & Sum multiple ZSET Sorted Sets into one ...
One way I tried is to fetch this data into Ruby objects and do the grouping logic. I am looking for ways to...
Read more >
Redis Sink - Lenses Documentation
This Redis sink connector allows you to write messages from Kafka to Redis. ... Field selection; Target sortedSet, cache or multiple sorted set...
Read more >
How To Manage Sorted Sets in Redis - DigitalOcean
To try out this command, run the following zadd command to create a sorted set where each member has the same score:
Read more >
Redis sorted sets
A Redis sorted set is a collection of unique strings (members) ordered by an associated score. When more than one string has the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found