EmbeddingIntentClassifier behaviour not reproducable with DIETClassifier

Problem

The migration docs imply that the following configuration in Rasa 2.x:

- name: "DIETClassifier"
  epochs: 300 
  hidden_layers_sizes: 
    text: [256, 128]
  number_of_transformer_layers: 0
  weight_sparsity: 0
  intent_classification: True
  entity_recognition: False
  use_masked_language_model: False
  BILOU_flag: False
  random_seed: 1

should give you the same results as the following configurations in Rasa 1.x:

- name: "EmbeddingIntentClassifier"
  epochs: 300
  random_seed: 1

However, we have created a minimum example with moodbot that gives different results with the two configurations.

This is a high priority issue that impacts a customer who has experienced a performance drop after migrating from rasa version 1.5.1 to 2.5.0 using the above settings

Definition of Done

• Confirm whether or not the results should be identical: they shouldn’t if the Rasa version is different. See this comment.

If not:

• Make this explicit in the migration docs, explaining why.
• Communicate this the customer and create another issue to address the drop in their performance.

If the results should be identical:

• Identify the source of the discrepancy. If it’s a bug, create an issue for the fix.
• Decide whether to create a regression test to ensure identical behaviour. If yes, create an issue for the test.: no regression test needed.

[aeshky]

Next steps

1. Communicate the following to the customer: The two configurations above map to the same underlying model, and should give you the same overall results provided you are using the same rasa version. When comparing the performance of the two configs on two different rasa versions, you may notice a difference in performance. The reason is explained here.

2. Come up with a new suitable configuration for them.

[hsm207]

I’ve communicated to the customer to retrain their 2.5.0 model with the updated config for DIETClassifier