NLU model does not recognize anything after upgrade to 1.3.2 from 1.2.5 for intent with underscore in name

Rasa version: 1.3.2

Python version: 3.6

Operating system (windows, osx, …): Ubuntu

Issue: After updating to Rasa 1.3.2 and retraining using rasa train, the NLU model does not recognize anything. Intents all have the same confidence of 0.024. Nothing has changed in the training data, domain, or config. I also tried to train only the NLU model and it worked fine.

During training, it founds 1647 intent examples (which is correct) and finishes with an accuracy of 0.994. Entities are correctly extracted and the core model seems to work just fine. I have added the training logs below.

Any idea what could happen and how we could properly train our NLU model?

Command or request that led to error: Typing who are you? in the rasa shell nlu gives:

{
  "intent": {
    "name": "ask_pictures",
    "confidence": 0.024387534707784653
  },
  "entities": [],
  "intent_ranking": [
    {
      "name": "ask_pictures",
      "confidence": 0.024387534707784653
    },
    {
      "name": "ask_talk_to",
      "confidence": 0.024387534707784653
    },
    ...
  ],
  "text": "who are you?"
}

Training output

Training NLU model...
2019-09-11 11:50:18 INFO     rasa.nlu.utils.spacy_utils  - Trying to load spacy model with name 'en'
2019-09-11 11:50:54 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en'.
2019-09-11 11:50:55 INFO     rasa.nlu.training_data.training_data  - Training data stats:
        - intent examples: 1647 (52 distinct intents)
        - Found intents: 'ask_identity', 'affirm', 'dont_know', [...]
        - Number of response examples: 0 (0 distinct response)
        - entity examples: 435 (16 distinct entities)
        - found entities: 'location', ...

2019-09-11 11:50:55 INFO     rasa.nlu.model  - Starting to train component SpacyNLP
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Starting to train component SpacyTokenizer
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Starting to train component SpacyEntityExtractor
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:51:01 INFO     rasa.nlu.model  - Starting to train component SpacyFeaturizer
2019-09-11 11:51:02 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:51:02 INFO     rasa.nlu.model  - Starting to train component CRFEntityExtractor
2019-09-11 11:51:10 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:51:10 INFO     rasa.nlu.model  - Starting to train component EntitySynonymMapper
2019-09-11 11:51:10 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:51:10 INFO     rasa.nlu.model  - Starting to train component EmbeddingIntentClassifier
Epochs: 100%|██████████████████████████████████████████████████| 300/300 [02:11<00:00,  2.28it/s, loss=0.543, acc=0.994]
2019-09-11 11:53:25 INFO     rasa.utils.train_utils  - Finished training embedding policy, train loss=0.543, train accuracy=0.994
2019-09-11 11:53:25 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:53:25 INFO     rasa.nlu.model  - Starting to train component DucklingHTTPExtractor
2019-09-11 11:53:25 INFO     rasa.nlu.model  - Finished training component.
2019-09-11 11:53:26 INFO     rasa.nlu.model  - Successfully saved model into '/tmp/tmpdivjkb3c/nlu'
NLU model training completed.

Content of configuration file (config.yml) (if relevant):

language: "en"
pipeline:
  - name: "SpacyNLP"
  - name: "SpacyTokenizer"
  - name: "SpacyEntityExtractor"
    dimensions: ["PERSON", ...]
  - name: "RegexFeaturizer"
  - name: "SpacyFeaturizer"
  - name: "CRFEntityExtractor"
    features: [
               ...
              ]
  - name: "EntitySynonymMapper"
  - name: "EmbeddingIntentClassifier"
  - name: "DucklingHTTPExtractor"
    url: "http://duckling.alpaca.casa"
    dimensions: ["time", ...]
    timezone: "America/New_York"

[Ghostvv]

it is the problem with featurization of intents that we introduce in 1.3, we’re working on fixing it

[nbeuchat]

Yes, works fine now in 1.3.3. Thanks for the quick fix!