Hallucinations and VAD [BLANK_AUDIO] Generations

Tested with both small and tiny model sizes.

Using the Streaming example with VAD turned on etc. I’ve tried different settings and tried using a prompt to try and eliminate hallucinations and sound effects but to no avail or getting VAD to properly work I might be missing something because it treats the hallucinations of sounds like words so it struggles to turn on AD. Examples of outputs are below:

When I’m not talking and the background noise is low the following gets transcribed. Ideally, it would run inference in the background and only detect incoming audio from me talking, etc. [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO]

Most of the time with the tiny model, it loves to hallucinate sound effects from no audio or low background noises. (wind blowing), (clicking), (barking)

Are there any settings that I can try that would help eliminate hallucinations from no audio or static or get VAD correctly working?

Great project, excited for any future features or updates.

[Macoron]

Another thing I noticed is that the mic has to be pretty good for VAD to work. When I used my webcam mic the VAD struggled to stop the recording but when I used a jack mic near my face it worked relatively well within ~1 sec. after I was done talking.

The VAD implementation is very basic. Original author of whisper.cpp recommend to use something more robust, like @BaMarcy suggested. But it is other extra dependency, which is out of this project reach.

Still you can try to play with Vad Thd and Vad Freq Thd parameters.

[BaMarcy]

I can recommend Silero VAD model which has ONNX version and that’s the state of the art and open source BTW

https://github.com/snakers4/silero-vad/tree/master/examples/cpp