Detect gibberish

Is your feature request related to a problem? Please describe. Sometimes when chatting with a chatbot application, users end up typing gibberish like: asdasd sadasd or fgfsad dasdsa etc. Is there a way to detect gibberish?

Describe the solution you’d like Detect gibberish input using NLP

Describe alternatives you’ve considered https://github.com/rrenaud/Gibberish-Detector (python based)

[jesus-seijas-sp]

Hello, I updated the algorithm. I was using 4 measures to detect gibberish:

• vowel frequency
• consonant frequency
• unique character frequency
• word over characters frequency I replaced vowel frequency and consonant frequency with vowel over consonant ratio. More changes done:
• y is considered a vowel
• Words with less than 6 characters are not considered gibberish
• The good range for words over characters is now (0.15, 0.3)
• The good range for unique characters is now (0.4, 1)
• The good range for vowel over consonant is (0.5, 1.5)

The unit tests:

    test('Should return false for good sentences', () => {
      expect(isGibberish('This sentence is totally valid.')).toBeFalsy();
      expect(isGibberish('This is not gibberish')).toBeFalsy();
      expect(isGibberish('Esta frase es totalmente correcta')).toBeFalsy();
      expect(isGibberish('goodbye')).toBeFalsy();
      expect(isGibberish('sure')).toBeFalsy();
      expect(isGibberish('very much')).toBeFalsy();
    });
    test('Should return true for gibberish sentences', () => {
      expect(isGibberish('zxcvwerjasc')).toBeTruthy();
      expect(isGibberish('ertrjiloifdfyyoiu')).toBeTruthy();
      expect(isGibberish('ajgñsgj ajdskfig jskf')).toBeTruthy();
      expect(
        isGibberish('euzbfdhuifdgiuhdsiudvbdjibgdfijbfdsiuddsfhjibfsdifdhbfd')
      ).toBeTruthy();
      expect(isGibberish('nmnjcviburili,<>')).toBeTruthy();
      expect(isGibberish('ubkddhepwxfzmpc')).toBeTruthy();
      expect(isGibberish('kwinsghocyevlzep')).toBeTruthy();
      expect(isGibberish('ertrjiloifdfyyoiu')).toBeTruthy();
      expect(isGibberish('asddsa adsdsa asdadsasd')).toBeTruthy();
    });

[jesus-seijas-sp]

Hello, I just published a new version that includes gibberish detection:

const { isGibberish } = require('@nlpjs/utils');

console.log(isGibberish('This sentence is totally valid')); // false
console.log(isGibberish('Esta frase es totalmente correcta')); // false
console.log(isGibberish('ertrjiloifdfyyoiu')); // true
console.log(isGibberish('ajgñsgj ajdskfig jskf')); // true

You have also a function to give you the score, if you prefer the probability:

const { gibberishScore } = require('@nlpjs/utils');

console.log(gibberishScore('This sentence is totally valid')); // 0
console.log(gibberishScore('Esta frase es totalmente correcta')); // 0.12
console.log(gibberishScore('ertrjiloifdfyyoiu')); // 0.57
console.log(gibberishScore('ajgñsgj ajdskfig jskf')); // 0.62