wordseg_best throw an error com.johnsnowlabs.nlp.annotators.pos.perceptron.AveragedPerceptron; local class incompatible: stream classdesc serialVersionUID

Description

I’m trying with wordseg_best, follow the example given. But it doesn’t work as expected (the error message in below)

I do try with other languages (zh, ja, ko), all working fine.

Expected Behavior

Should be shown as output provided in example

+-----------------------------------+---------------------------------------------------------+
|text                               |result                                                   |
+-----------------------------------+---------------------------------------------------------+
|จวนจะถึงร้านที่คุณจองโต๊ะไว้แล้วจ้ะ|[จวน, จะ, ถึง, ร้าน, ที่, คุณ, จอง, โต๊ะ, ไว้, แล้ว, จ้ะ]|
+-----------------------------------+---------------------------------------------------------+

Current Behavior

Error happened

An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 7.0 failed 4 times, most recent failure: Lost task 6.3 in stage 7.0 (TID 3395) (ip-10-0-4-134.ap-southeast-1.compute.internal executor 2): java.io.InvalidClassException: com.johnsnowlabs.nlp.annotators.pos.perceptron.AveragedPerceptron; local class incompatible: stream classdesc serialVersionUID = -7114715142956979922, local class serialVersionUID = 6642857758815297725
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850)

Possible Solution

Steps to Reproduce

I write a unit test for it

from pyspark.sql import SparkSession
from sparknlp.annotator import *
from sparknlp.base import DocumentAssembler, Pipeline

class TestThaiNlp(unittest.TestCase):

    def setUp(self):
        self.spark = SparkSession.builder \
            .master('local') \
            .appName('vision') \
            .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:3.2.2") \
            .getOrCreate()
        self.df = self.spark.createDataFrame([['จวนจะถึงร้านที่คุณจองโต๊ะไว้แล้วจ้ะ']], ["text"])

    def test_sparknlp(self):
        document_assembler = DocumentAssembler() \
            .setInputCol('text') \
            .setOutputCol('document')
        word_seg = WordSegmenterModel.pretrained('wordseg_best', 'th') \
            .setInputCols('document') \
            .setOutputCol('token')
        pipeline = Pipeline(stages=[document_assembler, word_seg])
        result = pipeline.fit(self.df).transform(self.df)
        result.show(2, False)

    def tearDown(self):
        self.spark.stop()

Context

I’m actually trying to do a benchmark to compare with pythainlp. However it doesn’t.

Your Environment

• EMR: 6.3.0
• spark version: 3.1.1
• sparknlp version: 3.2.2

[hadoop@ip-10-0-4-81 ~]$ java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment Corretto-8.282.08.1 (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM Corretto-8.282.08.1 (build 25.282-b08, mixed mode)

• Spark NLP version 3.2.2:
• Apache NLP version 3.1.1:
• Java version openjdk version "1.8.0_282":
• Setup and installation (Pypi, Conda, Maven, etc.):

apt-get update -y && apt-get install -y default-jre python3-dev
pip install pyspark PyICU pythainlp spark-nlp==3.2.2

• Operating System and version: Amazon Linux release 2 (Karoo)

[jslim89]

Thanks @maziyarpanahi @dcecchini .

it works now

[jslim89]

Hi @maziyarpanahi thanks for the quick update. Then I will wait for the new version