Question marks at the end swallowed

Looks like the example with just question marks is good now:

>>> segmenter.segment("??")
['??']

but the example with double question marks as a token at the end of a sentence still loses the question marks:

>>> segmenter.segment("T stands for the vector transposition. As shown in Fig. ??")
['T stands for the vector transposition.', 'As shown in Fig.']

looks like this is the minimal repro:

>>> segmenter.segment("Fig. ??")
['Fig.']

[nipunsadvilkar]

@danielkingai2 : Fixed above bug & have released char-span functionality today. Didn’t release it yday since I wanted to add tests and update docs.

[nipunsadvilkar]

Yes, I really need to come up with some assertion logic to map respective sentences to the original text. This is the main reason why I’ve been working on https://github.com/nipunsadvilkar/pySBD/tree/sentence-char-span branch because even if pysbd fails to find proper sentence. tok.is_sent_start would remain False and we will get an original text at the end