Cyrillic & latin symbols same time search

According to https://github.com/nextapps-de/flexsearch/issues/51 for search by cyrillic symbols we can use bellow options

{
    encode: false,
    split: /\s+/,
    tokenize: "reverse"
}

But this options breaks searching by latin symbols.

For example search by бренд Microsoft (brand Microsoft in english) doesn’t work

How I can fix it?

[gmfmi]

Hi @japanes,

I got the same issue and I finally decided to look into the source code to find a solution that do not require to recompile the code.

Basically, we would like to use the simple encoder (instead of setting it to false) but it removes any characters other than latin letters, numbers and spaces. I twisted a little bit the regexp patterns and now everything works. You can search both in latin and cyrillic!

There is probably a prettier way to do it but this should work for you:

{
    split: /\s+/,
    tokenize: "reverse",
    encode: function(str) {
        var regexp_replacements = {
            "a": /[àáâãäå]/g,
            "e": /[èéêë]/g,
            "i": /[ìíîï]/g,
            "o": /[òóôõöő]/g,
            "u": /[ùúûüű]/g,
            "y": /[ýŷÿ]/g,
            "n": /ñ/g,
            "c": /[ç]/g,
            "s": /ß/g,
            " ": /[-/]/g,
            "": /['!"#$%&\\()\*+,-./:;<=>?@[\]^_`{|}~]/g,
            " ": /\s+/g,
        }
        str = str.toLowerCase();
        for (var key of Object.keys(regexp_replacements)) {
            str = str.replace(regexp_replacements[key], key);
        }
        return str === " " ? "" : str;
    }
}

Hoping that can help you 😉

[angeloashmore]

@mryodo I realize I’m replying almost a year later 😅, but you can pass a FlexSearch instance directly to useFlexSearch. By doing that, you shouldn’t need to have a hard-coded edited version of the hook in your project.

const importedIndex = FlexSearch.create({
  split: /\s+/,
  tokenize: "reverse",
  encode: function (str) {
    var regexp_replacements = {
      a: /[àáâãäå]/g,
      e: /[èéêë]/g,
      i: /[ìíîï]/g,
      o: /[òóôõöő]/g,
      u: /[ùúûüű]/g,
      y: /[ýŷÿ]/g,
      n: /ñ/g,
      c: /[ç]/g,
      s: /ß/g,
      " ": /[-/]/g,
      "": /['!"#$%&\\()\*+,-./:;<=>?@[\]^_`{|}~]/g,
      " ": /\s+/g,
    }
    str = str.toLowerCase()
    for (var key of Object.keys(regexp_replacements)) {
      str = str.replace(regexp_replacements[key], key)
    }
    return str === " " ? "" : str
  },
})

importedIndex.import(yourExistingIndex)

const results = useFlexSearch(query, importedIndex)

Ideally you can instantiate the index outside your React component and only call importedIndex.import once in within your component for better performance.