Provide a java-friendly API

• work on JavaDStream
• not (?) rely on implicits

[BenFradet]

Have you tried having AbstractFunction1 extend Serializable:

Function1<String, ProducerRecord<String, String>> f = new AbstractFunction1<String, ProducerRecord<String, String>>() extends Serializable {
        @Override
        public ProducerRecord<String, String> apply(final String s) {
            return new ProducerRecord<>("my-topic", s);
        }
    };

?

[rfmvlc]

Hello Ben,

It doesn’t work on Java RDDs? -> @BenFradet work on JavaDStream…

@sunone5 @huylv

Do you have any working sample on Java please?

I’ve got a mismatch error in the function:

 required: scala.collection.immutable.Map<String,Object>,Function1<String,ProducerRecord<K,V>>,Option<Callback>
  found: scala.collection.mutable.Map<String,Object>,<anonymous SerializableFunc<String,ProducerRecord<String,String>>>,Option<Object>
  reason: no instance(s) of type variable(s) A,B exist so that scala.collection.mutable.Map<A,B> conforms to scala.collection.immutable.Map<String,Object>
  where K,V,T,A,B are type-variables:
    K extends Object declared in method <K,V>writeToKafka(scala.collection.immutable.Map<String,Object>,Function1<T,ProducerRecord<K,V>>,Option<Callback>)
    V extends Object declared in method <K,V>writeToKafka(scala.collection.immutable.Map<String,Object>,Function1<T,ProducerRecord<K,V>>,Option<Callback>)
    T extends Object declared in class KafkaWriter
    A extends Object declared in method <A,B>mapAsScalaMap(java.util.Map<A,B>)
    B extends Object declared in method <A,B>mapAsScalaMap(java.util.Map<A,B>)

with this sample code

imports//

import com.github.benfradet.spark.kafka.writer.DStreamKafkaWriter;
import com.github.benfradet.spark.kafka.writer.KafkaWriter;
import com.github.benfradet.spark.kafka.writer.RDDKafkaWriter;
import org.apache.commons.lang.math.RandomUtils;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SparkSession;
import scala.Function1;
import scala.Option;
import scala.collection.JavaConversions;

import java.util.Calendar;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;


 KafkaWriter<String> kafkaWriter = new RDDKafkaWriter<>(lines.rdd(), scala.reflect.ClassTag$.MODULE$.apply(String.class));

        kafkaWriter.writeToKafka(JavaConversions.mapAsScalaMap(producerConfig),
                new SerializableFunc<String, ProducerRecord<String,String>>() {
                    @Override
                    public ProducerRecord<String, String> apply(final String s) {
                        return new ProducerRecord<>("sometopic", s);
                    }
                }
                ,
                Option.empty()
        );

import scala.Serializable;
import scala.runtime.AbstractFunction1;


abstract class SerializableFunc<T, R> extends AbstractFunction1<T, R> implements Serializable {}

Cheers!