TypeError: persist() missing 1 required positional argument: 'model_dir'

Rasa NLU version: 0.14.4

Operating system: OS X

Content of model configuration file:

language: "zh"

pipeline:
    - name: "nlp_mitie"
      model: "data/total_word_feature_extractor_zh.dat"
    - name: "tokenizer_jieba_search.JiebaTokenizer"
      dictionary_path: "data/jieba_dict.txt"
    - name: "ner_mitie"
    - name: "ner_synonyms"
    - name: "intent_entity_featurizer_regex"
    - name: "intent_featurizer_mitie"
    - name: "intent_classifier_sklearn"

Issue: I substituted tokenizer_jieba with the customized one which is I copied from here (https://raw.githubusercontent.com/RasaHQ/rasa_nlu/fff482663cd33ec16b16a39847e58884f4b4c9a2/rasa_nlu/tokenizers/jieba_tokenizer.py) without chaning anything, and renamed it to tokenizer_jieba_search.py.

It gives error log below:

Traceback (most recent call last):
  File "/Users/kai/.pyenv/versions/3.6.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/kai/.pyenv/versions/3.6.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/kai/.virtualenvs/rasa-play-re5z2bZX/lib/python3.6/site-packages/rasa_nlu/train.py", line 184, in <module>
    num_threads=cmdline_args.num_threads)
  File "/Users/kai/.virtualenvs/rasa-play-re5z2bZX/lib/python3.6/site-packages/rasa_nlu/train.py", line 160, in do_train
    fixed_model_name)
  File "/Users/kai/.virtualenvs/rasa-play-re5z2bZX/lib/python3.6/site-packages/rasa_nlu/model.py", line 233, in persist
    update = component.persist(dir_name)
TypeError: persist() missing 1 required positional argument: 'model_dir'

If I use tokenizer_jieba, it would works fine.

Since I didn’t change the tokenizer code (except its file name), I thought it will works. Was I missed something?

[akelad]

@erohmensing can you take a look instead?

[erohmensing]

@kaiwk I’m glad it works! If you create a PR, we’ll check it out. What sorts of different tokenizer modes does it support?