Mapping Policy Ignores NLU Thresholds--And is there any way to prioritize one policy over another?

Rasa version: 0.14.4

Python version: 3.6

Operating system (windows, osx, …): Redhat

Issue: I have NLU Threshold configured with default 0.3 value. “greet” intent is mapped to “action_greet”. When I get incoming user intent classified as “greet” with 0.24 confidence, I would expect the classification to fail and use the fallback policy. Instead the mapping policy is invoked. So you get the following:

2019-05-21 11:30:25 DEBUG rasa_core.processor - Received user message ‘attending training tomorrow’ with intent ‘{‘name’: ‘greet’, ‘confidence’: 0.24704762503877342}’ and entities ‘[{‘start’: 19, ‘end’: 27, ‘text’: ‘tomorrow’, ‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘confidence’: 1.0, ‘additional_info’: {‘values’: [{‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘grain’: ‘day’, ‘type’: ‘value’}], ‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘grain’: ‘day’, ‘type’: ‘value’}, ‘entity’: ‘time’, ‘extractor’: ‘DucklingHTTPExtractor’}]’

2019-05-21 11:30:25 DEBUG rasa_core.policies.fallback - NLU confidence 0.24704762503877342 is lower than NLU threshold 0.3.

2019-05-21 11:30:25 DEBUG rasa_core.policies.memoization - There is a memorised next action ‘35’ 2019-05-21 11:30:25 DEBUG rasa_core.policies.form_policy - There is an active form ‘student_loan_form’ 2019-05-21 11:30:25 DEBUG rasa_core.policies.ensemble - Predicted next action using policy_4_MappingPolicy

Is there any way for the MappingPolicy to only work when the NLU confidence is greater than threshold?

Content of configuration file (config.yml):

  - name: KerasPolicy
    epochs: 270
    max_history: 3
  - name: FallbackPolicy
    fallback_action_name: 'action_custom_fallback'
  - name: MemoizationPolicy
    max_history: 5
  - name: FormPolicy
  - name: MappingPolicy

Content of domain file (domain.yml) (if used & relevant):

intents:
  # add your intents
  - wait: {triggers: action_waiting}
  - ready: {triggers: action_ready}
  - initialize
  - greet: {triggers: action_greet}
  - affirm
  - inform_none
  - deny
  - repeat: {triggers: action_repeat}
  - more_info: {triggers: action_utter_verify_info}
  - not_sure: {triggers: action_not_sure}
  - inform_loan
  - inform_loan_exceeds
  - inform_loan_under
  - automated_message_hold
  - automated_message_terminate
  - out_of_scope: {triggers: action_utter_back_on_track}
  - ambiguous_affirm_deny: {triggers: action_utter_ambiguous_response}
  - ask_is_machine: {triggers: utter_i_am_machine}
  - want_agent: {triggers: utter_will_get_agent}

[erohmensing]

@lgrinberg, great, let me know how those policy priorities go for you. You’re right that they’re undocumented, this is because we want to encourage the default prioirities (but to do this we have to get them right!). We’re still trying to decide what should get priority in general.

@tahamr83 thank you for catching this bug! Seems to happen here https://github.com/RasaHQ/rasa/blob/master/rasa/core/policies/mapping_policy.py#L97-L106 – the MappingPolicy doesn’t persist and load the priority, a result of the priority and mapping policy PRs being worked on around the same time. Will add to the fix.

[tahamr83]

Ah. I get it. I did go to model directory to find no folder for mapping policy and suspected that could be the bug. Nice