WagtailSearch ElasticSearch backend setting to set results size

Issue Summary

There is no way to specify results count ('size'), returned by elasticsearch2 backend which results into always getting the defaults (10) results.

Steps to Reproduce

1. Set up WagtailSearch with elasticsearch2 backend.
2. Run .search('QUERY').results() always returns 10 results

There is nothing in documentation about specifying the results count. Should be a django setting? Here is what I am doing:

In [7]: query = 'shoes'

In [8]: TenantPage.objects.count()
Out[8]: 88

In [9]: TenantPage.objects.search(query).count()
Out[9]: 80

In [10]: len(TenantPage.objects.search(query).results())
Out[10]: 10

I also tried to add ‘size’ to OPTIONS in Django settings as below:

WAGTAILSEARCH_BACKENDS = {
    'default': {
        'BACKEND': 'wagtail.wagtailsearch.backends.elasticsearch2',
        'INDEX': 'wagtail',
        'INDEX_SETTINGS': {},
        'OPTIONS': {'from': 0, 'size': 100},  # has no effect
        'TIMEOUT': 5,
        'URLS': ['http://localhost:9200']
    }
}

If you trace the Elasticsearch2SearchBackends result class (Elasticsearch2SearchResults) to it’s parent, you end up here:

class ElasticsearchSearchResults(BaseSearchResults):
    fields_param_name = 'fields'
    
    ...

    def _do_search(self):
        # Params for elasticsearch query
        params = dict(
            index=self.backend.get_index_for_model(self.query.queryset.model).name,
            body=self._get_es_body(),
            _source=False,
            from_=self.start,
        )

        params[self.fields_param_name] = 'pk'

        # Add size if set
        if self.stop is not None:
            params['size'] = self.stop - self.start
    ...

If I hardcode params['size']=100 it works. I see some logic to “Add size if set” but nothing about how to actually set it. Is it documented? But I assume it’s being used for pagination and offsetting the results.

Technical details

• Python version: 2.7.3
• Django version: 1.9.6
• Wagtail version: 1.9

[gasman]

Slicing is implemented internally by passing the relevant from / size parameters to Elasticsearch (just as Django querysets implement slicing by adding a LIMIT clause to the SQL), so there shouldn’t be any performance implication there.

I think it would make sense for our ES backend to set a default size of e.g. 100 when no slice has been given - rather than ES’s own default of 10 - so that people are less likely to run into this “taking a slice gives you more results” weirdness. Perhaps we should also have it throw a warning when we run a query with no explicit range specified, similarly to how Django now throws a warning when paginating an unordered queryset?

[nimasmi]

Aside: the [:100] workaround strips the annotation added by .annotate_score(), as does retrieving a result by its index.

>>> results = MyObject.search('foo').annotate_score('_score')
>>> results[0]._score
AttributeError: 'MyObject' object has no attribute '_score'
>>> [x._score for x in results[:100]]
AttributeError: 'MyObject' object has no attribute '_score'
>>> [x._score for x in results]
[0.38437524, 0.24610822, 0.24095446, 0.23447372, 0.22940964, 0.21996033, 0.21083516, 0.1894834, 0.18731217, 0.1755507]