Bug with beam-size=1?

In trying to get tests to passing with scoring (#538), I have turned up some weird behavior with scores output by Sockeye. Here are two commands using a transformer model built in the system tests. Notice:

• The invocations differ only in the beam size (1 or 2)
• --skip-topk is not enabled
• With beam size of 1, the scores output should be impossible, since Sockeye outputs negative logprobs.

Any ideas?

CC: @bricksdont

$ python3 -m sockeye.translate -i src --output-type translation_with_score --use-cpu -m model --beam-size 1 2> /dev/null | head
-10.556	7 5 2 7 3 6 5 4 7 7
-10.727	9 2 4 1 6 7 8 6 8
-12.788	8 6 8 7
-10.413	0 5 0 7 5 9 0 6 3 1
-10.731	7 9 2 6 8 5 0 6 5
-12.490	5 6 3 2
-inf	
-11.242	3 9 1 3 8 7
-15.759	2 1
-10.506	8 8 8 2 4 4 5 5 2 5
$ python3 -m sockeye.translate -i src --output-type translation_with_score --use-cpu -m model --beam-size 2 2> /dev/null | head
0.003	7 5 2 7 3 6 5 4 7 7
0.001	9 2 4 1 6 7 8 6 8
0.000	8 6 8 7
0.002	0 5 0 7 5 9 0 6 3 1
0.001	7 9 2 6 8 5 0 6 5
0.001	5 6 3 2
-inf	
0.001	3 9 1 3 8 7
0.001	2 1
0.002	8 8 8 2 4 4 5 5 2 5

[tdomhan]

good point regarding beam size 1. What if we changed sockeye.translate to only enable softmax skipping if the user does not specify an output handler, which uses the score?

[bricksdont]

Yes - skip_softmax is enabled by default for beam size 1. Initially, this was a CLI option, but after some discussion we changed it to default:

PR Discussion: https://github.com/awslabs/sockeye/pull/519 Commit: https://github.com/awslabs/sockeye/pull/519/commits/5c05dc3de1686303787fb7b2d17ad9037ac4564e

I think we simply never considered scoring as a usecase so far. If there is consensus to change it back to optional for inference, I will open a PR.