Warning embedding policy exceeds 10% of system memory + low confidence

Rasa Core version: 0.13.2

Issue: I have the following stacktrace when training with my embedding policy :

2019-03-13 11:17:42 DEBUG    rasa_core.training.generator  - Number of augmentation rounds is 0
2019-03-13 11:17:42 DEBUG    rasa_core.training.generator  - Starting data generation round 0 ... (with 1 trackers)
Processed Story Blocks: 100%|██████████| 135/135 [00:00<00:00, 727.23it/s, # trackers=1]
2019-03-13 11:17:42 DEBUG    rasa_core.training.generator  - Finished phase (97 training samples found).
2019-03-13 11:17:42 DEBUG    rasa_core.training.generator  - Data generation rounds finished.
2019-03-13 11:17:42 DEBUG    rasa_core.training.generator  - Found 0 unused checkpoints
2019-03-13 11:17:42 DEBUG    rasa_core.training.generator  - Found 97 training trackers.
2019-03-13 11:17:42 DEBUG    rasa_core.agent  - Agent trainer got kwargs: {}
2019-03-13 11:17:42 DEBUG    rasa_core.policies.embedding_policy  - Started training embedding policy.
2019-03-13 11:17:42 DEBUG    rasa_core.featurizers  - Creating states and action examples from collected trackers (by FullDialogueTrackerFeaturizer(LabelTokenizerSingleStateFeaturizer))...
Processed trackers: 100%|██████████| 97/97 [00:00<00:00, 23073.07it/s]
2019-03-13 11:17:42 DEBUG    rasa_core.featurizers  - The longest dialogue has 16 actions.
2019-03-13 11:17:42 DEBUG    rasa_core.policies.embedding_policy  - Check if num_neg 20 is smaller than number of actions 62, else set num_neg to the number of actions - 1
2019-03-13 11:17:53.145619: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-03-13 11:17:53 INFO     rasa_core.policies.embedding_policy  - Accuracy is updated every 20 epochs
Epochs:   0%|          | 0/2000 [00:00<?, ?it/s]2019-03-13 11:17:59.428659: W tensorflow/core/framework/allocator.cc:122] Allocation of 39259392 exceeds 10% of system memory.
2019-03-13 11:18:00.179547: W tensorflow/core/framework/allocator.cc:122] Allocation of 39259392 exceeds 10% of system memory.
Epochs:   1%|          | 19/2000 [00:33<48:10,  1.46s/it, loss=0.410, acc=0.450]2019-03-13 11:18:28.421940: W tensorflow/core/framework/allocator.cc:122] Allocation of 39259392 exceeds 10% of system memory.
2019-03-13 11:18:28.441308: W tensorflow/core/framework/allocator.cc:122] Allocation of 39259392 exceeds 10% of system memory.
Epochs:   2%|▏         | 39/2000 [01:02<48:21,  1.48s/it, loss=0.284, acc=0.841]2019-03-13 11:18:57.501154: W tensorflow/core/framework/allocator.cc:122] Allocation of 39259392 exceeds 10% of system memory.
Epochs: 100%|██████████| 2000/2000 [34:47<00:00,  1.02s/it, loss=0.099, acc=0.992]
2019-03-13 11:52:40 INFO     rasa_core.policies.embedding_policy  - Finished training embedding policy, loss=0.099, train accuracy=0.992
2019-03-13 11:52:40 DEBUG    rasa_core.featurizers  - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturizer(SingleStateFeaturizer))...
Processed trackers: 100%|██████████| 97/97 [00:00<00:00, 696.27it/s, # actions=378]
2019-03-13 11:52:40 DEBUG    rasa_core.featurizers  - Created 378 action examples.
Processed actions: 378it [00:00, 3328.02it/s, # examples=378]
2019-03-13 11:52:40 DEBUG    rasa_core.policies.memoization  - Memorized 378 unique examples.
2019-03-13 11:52:42 INFO     rasa_core.agent  - Persisted model to '/app/models/chatbot-masante/core'

With this warning : Allocation of 39259392 exceeds 10% of system memory.

My config policy looks like this :

policies:
  - name: EmbeddingPolicy
    epochs: 2000
    attn_shift_range: 3
    featurizer:
      - name: FullDialogueTrackerFeaturizer
        state_featurizer:
          - name: LabelTokenizerSingleStateFeaturizer
  - name: FallbackPolicy
    fallback_action_name: 'action_default_fallback'
    nlu_threshold: 0.7
    core_threshold: 0.3
  - name: MemoizationPolicy
    max_history: 6

And then, when I try to use it, I have really low confidence for the wrong intent :

rasa_core_1  | 2019-03-13 13:03:32 DEBUG    rasa_core.processor  - Predicted next action 'action_reponse_prise_en_charge_specialite' with prob 0.13.

With the keras policy, I have the right intent with prob 1.0.

Could the warning be the problem for the really low confidence ? I tried the evaluation script and the embedding gave me pretty good result, so I don’t understand why it does not work once the core is started.

[huberrom]

I’ll try with 4000 epochs, I’ll come back to you once the training is done

[jyotizorang]

Hi,

I am having the same issue [