Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Flink pulsar sink NotSerializableException

See original GitHub issue

<dependency> <groupId>org.apache.pulsar</groupId> <artifactId>pulsar-flink</artifactId> <version>2.4.0</version> </dependency> Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: The implementation of the RichSinkFunction is not serializable. The object probably contains or references non serializable fields. at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:99) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1559) at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:185) at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1225) at wordcount.WordCountProcess.main(WordCountProcess.java:63) Caused by: java.io.NotSerializableException: org.apache.pulsar.client.impl.DefaultBatcherBuilder at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:534) at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:81) `

    stream.addSink(new FlinkPulsarProducer<>(
            SERVICE_URL,
            OUT_TOPIC,
            new AuthenticationDisabled(),
            wordWithCount -> wordWithCount.toString().getBytes(),
            wordWithCount -> wordWithCount.getWord()
    ));
    env.execute("flink window count!");`

Issue Analytics

State:
Created 4 years ago
Reactions:2
Comments:12 (5 by maintainers)

Top GitHub Comments

2reactions

ChangWindecommented, Sep 19, 2019

@vruc @idantony thanks for your help

0reactions

sijiecommented, Jan 14, 2020

This has been fixed by #5068

Top Results From Across the Web

[GitHub] [pulsar] cnicy opened a new issue #4721: Flink pulsar sink ...

cnicy opened a new issue #4721: Flink pulsar sink NotSerializableException URL: https://github.com/apache/pulsar/issues/4721 Exception in thread "main" ...

PulsarSerializationSchema (Flink : 1.15-SNAPSHOT API)

sinkContext - Runtime information i.e. partitions, subtaskId. sinkConfiguration - All the configure options for the Pulsar sink. You can add custom options.

Custom Predicate Evaluation in Flink gives java.io ...

apache.flink.api.common.InvalidProgramException: java.util.function.Predicate$$Lambda$167/20853837@390a07a0 is not serializable ...

Pulsar Flink Connector (Deprecated) - StreamNative Hub

Pulsar Flink Connector allows Flink to read data from Pulsar and write data ... to Pulsar and provides exactly-once source semantics and at-least-once...

Writing Custom Sink Connectors for Pulsar I/O - YouTube

You currently have massive amounts of data sitting on Apache Pulsar and likely — even more data coming in every second.