'json' query does not return expected results from 'application/json'

Hi 👋 I’m a bit stumped by the behaviour I’m seeing in this implementation. For some reason the term json doesn’t return pages with API documentation in it, but application/json does. Is it related to the tokeniser?

I’ve created a jsfiddle here with the same content and the default pipeline functions turned off: https://jsfiddle.net/f3sbrtq2/25/

[hoelzro]

@lewisnyman Hi! Your suspicions about the tokenizer are spot on; it only splits tokens on - characters and whitespace by default. lunr uses full tokens to consult its inverted index - it doesn’t find all documents containing application/json under the set of documents with the standalone json token, hence the “missing” results.

Another thing I noticed about your example; you remove everything from the builder pipeline, but you leave the search pipeline alone - you might want to this.searchPipeline.remove(lunr.stemmer) as well.

Out of curiosity, what are you trying to use lunr to accomplish? By removing all of the processing functions from the pipeline you’re removing a lot of the value of what lunr provides! Or was that just for the purposes of your example?

[lewisnyman]

We’re using middleman-search which helpfully prebuilds the index but is not actively maintained manastech/middleman-search#29