Function localrun quits when using partitioned topic.
See original GitHub issueFirst i write a demo package by gradle
plugins {
id 'java'
id "com.github.johnrengelman.shadow" version "2.0.4"
}
package com.codelipenghui.pulsar.function;
import org.apache.pulsar.functions.api.Context;
import org.apache.pulsar.functions.api.Function;
public class WordCountFunction implements Function<String, Void> {
@Override
public Void process(String input, Context context) throws Exception {
context.incrCounter("test-case-total", 1);
return null;
}
}
I got info :
[root@qa-5-171 apache-pulsar-2.2.0-incubating-SNAPSHOT]# bin/pulsar-admin functions localrun --jar target/pulsar-functions-0.1.0-SNAPSHOT-jar-with-dependencies.jar --className pulsarfunctions.starter.sdk.WordCountFunction --tenant public --namespace default --name word-count-test --inputs persistent://public/default/plat.correctness.verification --output persistent://public/default/word-count-test-output
19:58:34.946 [main] INFO org.apache.pulsar.functions.runtime.ProcessRuntimeFactory - Java instance jar location is not defined, using the location defined in system environment : /opt/apache-pulsar-2.2.0-incubating-SNAPSHOT/instances/java-instance.jar
19:58:34.954 [main] INFO org.apache.pulsar.functions.runtime.ProcessRuntimeFactory - Python instance file location is not defined using the location defined in system environment : /opt/apache-pulsar-2.2.0-incubating-SNAPSHOT/instances/python-instance/python_instance_main.py
19:58:34.974 [main] INFO org.apache.pulsar.functions.runtime.RuntimeSpawner - RuntimeSpawner starting function word-count-test - 0
19:58:34.987 [main] INFO org.apache.pulsar.functions.runtime.ProcessRuntime - ProcessBuilder starting the process with args java -cp /opt/apache-pulsar-2.2.0-incubating-SNAPSHOT/instances/java-instance.jar -Dpulsar.functions.java.instance.jar=/opt/apache-pulsar-2.2.0-incubating-SNAPSHOT/instances/java-instance.jar -Dlog4j.configurationFile=java_instance_log4j2.yml -Dpulsar.log.dir=/opt/apache-pulsar-2.2.0-incubating-SNAPSHOT/logs/functions -Dpulsar.log.file=word-count-test org.apache.pulsar.functions.runtime.JavaInstanceMain --jar target/pulsar-functions-0.1.0-SNAPSHOT-jar-with-dependencies.jar --instance_id 0 --function_id ac8f5862-34cd-4f0e-869e-39f5717a99a2 --function_version 1498f7f5-5483-42c2-b2ac-c07c5c1626ce --tenant public --namespace default --name word-count-test --function_classname pulsarfunctions.starter.sdk.WordCountFunction --auto_ack true --processing_guarantees ATLEAST_ONCE --pulsar_serviceurl http://localhost:8080/ --use_tls false --tls_allow_insecure false --hostname_verification_enabled false --max_buffered_tuples 1024 --port 16821 --source_type_classname "java.lang.String" --source_subscription_type SHARED --source_topics_serde_classname {"persistent://public/default/plat.correctness.verification":""} --sink_type_classname "java.lang.Void" --sink_topic persistent://public/default/word-count-test-output
19:58:34.995 [main] INFO org.apache.pulsar.functions.runtime.ProcessRuntime - Started process successfully
19:58:41.297 [main] INFO org.apache.pulsar.admin.cli.CmdFunctions - RuntimeSpawner quit because of
19:58:41.299 [Thread-3] INFO org.apache.pulsar.admin.cli.CmdFunctions - Shutting down the localrun runtimeSpawner ...
And then i use https://github.com/streamlio/pulsar-functions-java-starter.git
I got the same.
Issue Analytics
- State:
- Created 5 years ago
- Reactions:1
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Pulsar admin CLI
The pulsar-admin tool enables you to manage Pulsar installations, including clusters, brokers, namespaces, tenants, and more.
Read more >Function Calls
Function Calls. There are several cases when the optimizer cannot perform pruning. One common reasons is when an operator is used on top...
Read more >Chapter 6, Restrictions and Limitations on Partitioning
Previously, the HANDLER statement was not supported with partitioned tables. ... As discussed in Server SQL Modes, the results of many MySQL functions...
Read more >How do I manage Partition Functions and Partition Schemes ...
I have figured out how to resolve the issue if the partition is one ... I run into problems with managing partition functions...
Read more >Dynamic Partitioning in Kinesis Data Firehose
Creating partitioning keys with an AWS Lambda function. For compressed or encrypted data records, or data that is in any file format other...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Topic of plat.correctness.verification is a partitioned topic with six partitions. When i set input to plat.correctness.verification local run will quit. When i set input to plat.correctness.verification-partition-0 local run can process success.
Log for quit
I think the problem is state storage is not enabled in cluster mode by default. I am updating the documentation in #2335 . I will verify local run and update here.