More flexibility in the processing pipeline

Hi! I like a lot the simplicity of this library. I see that v2 is planned to have a caseSensitive option. In my opinion, the preprocessing of the tokens (and the query) could be more general. For example, lunr.js offers the possibility to define an arbitrary chain of functions for this.

For this library, the following two lines could be replaced by a function passed by the user. https://github.com/fergiemcdowall/search-index/blob/421c8213958ef76bd139f0900c85c88968848be8/src/write.js#L44-L45 The same function should be also applied to the query tokens.

My use-case is the need for phonetic encoders (e.g. https://github.com/nextapps-de/flexsearch#encoders) in languages other than English. For example, a token like Björn could be matched by the user query bjorn (notice that both case-insensitiveness and phonetic normalization are needed).

[eklem]

Create two building blocks that people can choose to use?

• Empty function/step for indexing side
• Empty function/step for query side

And an easy way to communicate and use these two with existing functions / building blocks? Could be external as long as it’s easy to plug in.

[eklem]

So this one should maybe be an example for search-index and for libraries that builds on search-index, it can be actual (empty) functions.