TypeError: object of type 'generator' has no len()

$ python sentiment_evaluate.py
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\utils\deprecation.py:
143: FutureWarning: The sklearn.svm.classes module is  deprecated in version 0.2
2 and will be removed in version 0.24. The corresponding classes / functions sho
uld instead be imported from sklearn.svm. Anything that cannot be imported from
sklearn.svm is now part of the private API.
  warnings.warn(message, FutureWarning)
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator LinearSVC from version 0.20.2 when using versi
on 0.23.2. This might lead to breaking code or invalid results. Use at your own
risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\utils\deprecation.py:
143: FutureWarning: The sklearn.preprocessing.label module is  deprecated in ver
sion 0.22 and will be removed in version 0.24. The corresponding classes / funct
ions should instead be imported from sklearn.preprocessing. Anything that cannot
 be imported from sklearn.preprocessing is now part of the private API.
  warnings.warn(message, FutureWarning)
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator LabelEncoder from version 0.20.2 when using ve
rsion 0.23.2. This might lead to breaking code or invalid results. Use at your o
wn risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator _SigmoidCalibration from version 0.20.2 when u
sing version 0.23.2. This might lead to breaking code or invalid results. Use at
 your own risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator CalibratedClassifierCV from version 0.20.2 whe
n using version 0.23.2. This might lead to breaking code or invalid results. Use
 at your own risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator TfidfTransformer from version 0.20.2 when usin
g version 0.23.2. This might lead to breaking code or invalid results. Use at yo
ur own risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator TfidfVectorizer from version 0.20.2 when using
 version 0.23.2. This might lead to breaking code or invalid results. Use at you
r own risk.
  warnings.warn(
2020-08-27 14:38:47 [INFO] Sentiment: Started.
Traceback (most recent call last):
  File "sentiment_evaluate.py", line 40, in <module>
    Sentiment().evaluate()
  File "sentiment_evaluate.py", line 36, in evaluate
    print(self._asari_sa(example))
  File "sentiment_evaluate.py", line 31, in _asari_sa
    res = self.asari.ping(text=sent)['top_class']
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\asari\api.py", line 2
7, in ping
    vector = self.preprocessor.transform([text])
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 1880, in transform
    X = super().transform(raw_documents)
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 1250, in transform
    _, X = self._count_vocab(raw_documents, fixed_vocab=True)
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 1110, in _count_vocab
    for feature in analyze(doc):
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 111, in _analyze
    doc = ngrams(doc)
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 242, in _word_ngrams
    n_original_tokens = len(original_tokens)
TypeError: object of type 'generator' has no len()

[hnishi]

こちら、私の方でも調べてみました。

とりあえず、Janome==0.3.7 をインストールすれば、エラーなく動きました。

@Hironsan さんの preprocessor では Janome v0.3.7 を使用しており、Janome v0.4.0 以上を asari v0.0.4 と組み合わせて使うと、上記エラーが発生するようです。 おそらく preprocessor に、 Janome v0.4.0 で廃止された tokenize() の stream オプションを使用しているためだと考えられます。

v0.4.0 から，ストリーミングモードのみサポートしており， stream オプションは廃止されました。

tokenize() メソッドに stream = True オプションを与えると，ストリーミングモードで動作します。ストリーミングモードでは，部分的な解析が完了する都度，解析結果を返します。戻り値はリストではなく generator になります。

https://mocobeta.github.io/janome/#v0-3-1-v0-3-10

[hourglasshoro]

とりあえず、https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py の242行目を

n_original_tokens = len(list(original_tokens))

のように書き換えると動きます。

preprocess.joblibで読み込んでいるTfidfVectorizerがうまく動作していないように思います。