Add ability to change lunr tokenizer

In order to make search more powerful, adding the ability to change the lunr.tokenizer.seperator would be nice.

https://github.com/olivernn/lunr.js/blob/master/lib/tokenizer.js#L69-L76

Out of the box, if you have the string: this.test

The query test will return empty, but this.t will return this.test.

By changing the regex used for tokenization, you could solve problems like this

[saminzadeh]

I am wondering because if we implement this feature for devs then it could be as separator / regex option or as one of list of predefined analyzers (i.e. like in https://www.elastic.co/guide/en/elasticsearch/reference/2.3/analyzer.html).

Hmm, yes that could be nice. But I agree, would probably want to iron that out a bit more before it is introduced into the API interface.

Second seems nice but first (your suggestion) the simplest to start with and very flexible

Yes I think this could be the best starting point. Just having access to the lunr object via itemsjs would be the most flexible for advanced users and will ensure the correct dependency

Something like this:

import lunr from 'itemjs/lunr';

lunr.tokenizer.separator = /[\s\-[\]:]+/g;

or

import ItemsJS from 'itemjs';
const index = ItemsJS(data, config);
index.lunr.tokenizer.separator = /[\s\-[\]:]+/g;

index.search({...})

[cigolpl]

@saminzadeh I’ve introduced simple full text integration with all external search engines in the latest version. You can see here -> https://github.com/itemsapi/itemsjs/blob/master/docs/lunr2-integration.md or in Readme