Transcription being concatenated oddly

I am trying to use the ctc decoding feature with kenlm on the wav2vec2 huggingface’s logits

vocab = ['l', 'z', 'u', 'k', 'f', 'r', 'g', 'i', 'v', 's', 'o', 'b', 'w', 'e', 'd', 'n', 'y', 'c', 'q', 'p', 'h', 't', 'a', 'x', ' ', 'j', 'm', '⁇', '', '⁇', '⁇']
alphabet = Alphabet.build_alphabet(vocab, ctc_token_idx=-3)
# Language Model
lm=LanguageModel(kenlm_model,alpha =0.169,
  beta = 0.055)
# build the decoder and decode the logits
decoder = BeamSearchDecoderCTC(alphabet,lm)

which returns the following output with beam size 64:

yeah jon okay i m calling from the clinic the family doctor clinessegryand this number six four five five one three o five

while when I was previously decoding with https://github.com/ynop/py-ctc-decode with the same lm and parameters getting:

yeah on okay i am calling from the clinic the family dot clinic try and this number six four five five one three o five

I don’t understand why the words are being concatenated together. Do you have any thoughts?

[gkucsko]

see PR #4 for some additional warnings around unigrams as well as improved partial scoring without a trie that should help with word concatenation

[gkucsko]

sounds great will have a look. thanks for the feedback and let me know if other issues come up!