Decouple confidence normalization from `ranking_length` parameter

What problem are you trying to solve?

Currently, all ML components (DIETClassifier, ResponseSelector, TEDPolicy) have a parameter called ranking_length which is supposed to be used when the user does not want to see the confidences of all possible candidate labels (intents / actions) and instead want to see a smaller number of labels.

However, what happens by default is that model’s predicted confidences for the labels inside the ranking length are renormalized to sum up to 1. This is wrong because this can artificially boost the confidences of labels and hides the original confidences that were predicted by the model itself. Renormalization step should be decoupled from this parameter.

Removing the renormalization step is also not possible because if a user has a lot of intents, say 200, all the predicted confidences tend to be very low which is problematic for fallback. So, it should be done only when needed, i.e. the user sees a benefit of it. Anecdotal evidence of this happening with customers

What’s your suggested solution?

Have a separate parameter, something like - renormalize_confidences which if set to True applies the renormalization.

Note that the re-normalization step is currently only applied when loss_type is set to cross_entropy and that condition should still stay.

Examples (if relevant)

No response

Is anything blocking this from being implemented? (if relevant)

No response

Definition of Done

• Renormalization logic decoupled from ranking_length parameter.
• New parameter added for renormalization (decide on the name if it the above suggestion doesn’t seem appropriate)
• Changelog and migration guide entry
• Docs
• Tests

[dakshvar22]

@TyDunn Just bringing this issue to your notice. This coupling between ranking_length and the renomalization step has come up quite often as causing problems as to how users (some customers) interpret confidences. I think we should decouple this logic with 3.0 itself because the right thing to do is to not apply this renormalization ideally but for practical reasons if some users find it useful we’ll need to support it. Doing this in a minor will mean that we can’t avoid renormalization by default.

[ka-bu]

No worries, happens to me all the time ^^. Btw, actually there’s no reason to mask the probabilities in TED, is there? The ensemble won’t care about the values that we are setting to 0. So maybe we set the default to 0 instead of LABEL_RANKING_LENGTH ?