Cyrillic languages support

Hello,

I’ve faced with the following behaviour.

This example works as expected:

const FlexSearch = require('flexsearch');
const index = new FlexSearch();

index.add(1, 'Foobar')
console.log(index.search('Foobar'));
// [ 1 ]

But this one shows no results.

const FlexSearch = require('flexsearch');
const index = new FlexSearch();

index.add(1, 'Фообар')
console.log(index.search('Фообар'));
// []

I’ve tested in node and in browser.

[gmfmi]

Hi @ts-thomas, last year you said:

The next main release will get an improvement of handling all language-specific features.

As FlexSearch is currently in 0.7.0 release candidate, I was wondering if there is some new feature about languages processing.

I recently created a project called SearchinGhost, it is an in-browser search plugins for Ghost CMS powered by FlexSearch. I am really happy with FlexSearch (thank your for the work done!) but I would like to bring multi-lang capabilities. For now, with what I read in the issues, I came up with these language default options:

Latin:

FlexSearch.create({
    encode: "simple",
    tokenize: "forward"
});

Arabic

FlexSearch.create({
    encode: false,
    rtl: true,
    split: /\s+/,
    tokenize: "forward"
});

Cyrilic, indian (any word separated by space language)

FlexSearch.create({
    encode: false,
    split: /\s+/,
    tokenize: "forward"
});

Chinese, Japanese or Korean (with dedicated chars w/o spaces)

FlexSearch.create({
    encode: false,
    tokenize: function(str){
        return str.replace(/[\x00-\x7F]/g, "").split("");
    }
});

Do you think there is any possible improvement/optimisation?

EDIT: I finally found this relevent documentation about the v0.7.0 - https://github.com/nextapps-de/flexsearch/blob/0.7.0/doc/0.7.0.md. Hope this version will be out one day 😃

[ts-thomas]

Also take into account to use the “rtl” option for right-to-left support:

var index = FlexSearch.create({
    encode: false,
    rtl: true,
    split: /\s+/,
    tokenize: "forward"
});