Rasa end-to-end test does not evaluate the NLU model on the provided end-to-end stories

Rasa version: 1.2.5

Python version: 3.6

Operating system (windows, osx, …): Ubuntu (through WSL)

Issue: When testing an end-to-end story with rasa test and the --e2e flag, the NLU evaluation is done on the normal training data instead of on the data in my end-to-end story files. The Core evaluation works as expected on my end-to-end stories.

The documentation reads:

Rasa lets you evaluate dialogues end-to-end, running through test conversations and making sure that both NLU and Core make correct predictions.

To do this, you need some stories in the end-to-end format, which includes both the NLU output and the original text.

Therefore, my expectation was to have:

• The Core evaluation creating the failed_stories.md and story_confmat.pdf for the stories in data_test
• The NLU evaluation creating the confmat.png, errors.json, and hist.png only for the sentences that are in the data_test/happy.md file. Right now, it evaluates the NLU model on the data in the data/nlu.md file.

FYI, our folder structure is:

data
|_ stories.md
|_ nlu.md
data_test
|_ happy.md
|_ etc.

Please note that I do have the test data in end-to-end format (which we got using the /conversations/{{sender_id}}/story endpoint). Here is a small sample of such an conversation:

## 2373134909417533
* greeting: Hello
    - utter_greet

...

* form: affirm: Yes it does
    - form: form_apartment_interest
    - slot{"apt_interested": true}
    - slot{"requested_slot": "user_email"}
* explain: why do you need this?
    - action_explain
    - action_listen
    - form_apartment_interest
    - slot{"requested_slot": "user_email"}

Did I misunderstand how the end-to-end evaluation works?

Command or request that led to error:

rasa test --stories data_test --e2e --out results

I also tried the following:

rasa test --stories data_test --nlu data_test --e2e --out results

In this case, nothing is returned for the NLU evalutation.

[sara-tagger]

Thanks for raising this issue, @JustinaPetr will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[stale[bot]]

This issue has been automatically closed due to inactivity. Please create a new issue if you need more help.