Unexpected retraining of NLU and Core, although only a domain response utter text is changed

This issue relates to #4691 and #4216.

Rasa version: 2.6.2

Python version: Python 3.8.8

Operating system (windows, osx, …): osx

Issue: The cmd rasa train retrains the NLU and/or Core model although it shouldn’t, when only the response text of an existing utter in the domain file is changed. Nothing else of the training data was changed, neither any NLU training examples, nor any intent, actions names etc. nor did the stories / rules data change.

Command or request that led to error:

1. Train the initial model as expected.

rasa train --out models --augmentation 100 --fixed-model-name default --data training -d training/domain.yml -c training/config.yml

1. Change domain.yml content from ‘initial content’ to ‘updated content’
2. Execute training

• expectation: Nothing changed. You can use the old model
• reality: NLU model is trained, Core is not retrained

rasa train --out models --augmentation 100 --fixed-model-name default --data training -d training/domain.yml -c training/config.yml

1. Execute training

• expectation: Nothing changed. You can use the old model
• reality: NLU model is not retrained, but Core is retrained

rasa train --out models --augmentation 100 --fixed-model-name default --data training -d training/domain.yml -c training/config.yml

1. Execute training

• expectation: Nothing changed. You can use the old model
• reality: Nothing changed. You can use the old model

rasa train --out models --augmentation 100 --fixed-model-name default --data training -d training/domain.yml -c training/config.yml

Content of domain file (domain.yml) ‘initial content’:

version: '2.0'
config:
  store_entities_as_slots: true
intents:
  - greet:
      use_entities: false
  - handover:
      use_entities: false
  - thanks:
      use_entities: false
entities: []
slots: {}
responses:
  utter_greet:
    - text: hi
  utter_thanks:
    - text: no prob
session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true
forms: {}

Content of domain file (domain.yml) ‘updated content’:

version: '2.0'
config:
  store_entities_as_slots: true
intents:
  - greet:
      use_entities: false
  - handover:
      use_entities: false
  - thanks:
      use_entities: false
entities: []
slots: {}
responses:
  utter_greet:
    - text: hi 2
  utter_thanks:
    - text: no prob
session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true
forms: {}

Content of nlu training data (nlu.yml):

version: "2.0"
nlu:
- intent: "greet"
  examples: |
    - guten tag
    - hallo
    - moin
    - hi
- intent: "thanks"
  examples: |
    - cool
    - super
    - danke

Content of stories training data (stories.yml):

version: '2.0'
stories:
- story: greet
  steps:
  - intent: greet
  - action: utter_greet
- story: thanks
  steps:
  - intent: thanks
  - action: utter_thanks

Content of rules training data (rules.yml):

version: '2.0'
rules: []

Content of configuration file (config.yml):

language: de

pipeline:
  - name: SpacyNLP
    model: "de_core_news_sm"
  - name: SpacyTokenizer
  - name: RegexFeaturizer
    case_sensitive: False
  - name: LexicalSyntacticFeaturizer
  - name: "CountVectorsFeaturizer"
    lowercase: True
    use_lemma: False
    strip_accents: "unicode"
  - name: "DIETClassifier"
    intent_classification: True
    entity_recognition: True
    ranking_length: 5
    BILOU_flag: False
    random_seed: 1
    epochs: 15
    checkpoint_model: False
    evaluate_on_number_of_examples: 0
    evaluate_every_number_of_epochs: 2
    model_confidence: linear_norm
    constrain_similarities: true
  - name: RegexEntityExtractor
    case_sensitive: False
    use_lookup_tables: True
    use_regexes: True
    use_word_boundaries: True
  - name: EntitySynonymMapper
  - name: FallbackClassifier
    threshold: 0.7
    ambiguity_threshold: 0.15

policies:
- name: MemoizationPolicy
  max_history: 5
- name: TEDPolicy
  max_history: 5
  epochs: 100
  model_confidence: linear_norm
  constrain_similarities: true
- name: RulePolicy
  core_fallback_threshold: 0.2
  core_fallback_action_name: action_default_fallback

[wochinge]

Yep should be fixed in 3.0.

[benos]

Are you sure this is fixed in 3.x?

I’m still having the issue on 3.1

Here I retrain before pulling my latest changes. All gets restored from cache as expected:

Then I pull the changes, which consists of removing one action (in the domain and from one rule):

Which leads to a retrain of the NLU model (obviously followed by core, as expected) :