Search: Can't filter QuerySets by FilterFields on RelatedFields

Is your proposal related to a problem?

Using Wagtail Search, it’s not possible to search a QuerySet that’s been filtered by a field on a related object. For example, using the test Book model:

class Book(models.Model, index.Indexed):
    ...
    tags = TaggableManager()

    search_fields = [
        ...
        index.RelatedFields('tags', [
            index.FilterField('slug'),
        ]),
    ]

You can generate a filtered QuerySet like this:

filtered_books = Book.objects.filter(tags__slug='foo')

But then trying to search that QuerySet produces a FilterFieldError:

filtered_books.search('shakespeare')

wagtail.search.backends.base.FilterFieldError: Cannot filter search results with field “slug”. Please add index.FilterField(‘slug’) to Book.search_fields.

I’m filing this as an Enhancement instead of a Bug because it’s already documented in the docs:

Filtering on index.RelatedFields

It’s not possible to filter on any index.FilterFields within index.RelatedFields using the QuerySet API. However, the fields are indexed, so it should be possible to use them by querying Elasticsearch manually.

Filtering on index.RelatedFields with the QuerySet API is planned for a future release of Wagtail.

This comment is several years old, and there doesn’t seem to be an issue for this open yet, so I’m opening this one.

Describe the solution you’d like

Filtering by related fields should work the same as other kinds of filtering.

Describe alternatives you’ve considered

There are some workarounds documented on https://github.com/wagtail/wagtail/issues/5802#issuecomment-580670744: filtering by the ID of the related object, or filtering by a callable and a custom query. Unfortunately this latter approach requires implementing a custom search backend and query compiler.

@kaedroho’s comment at https://github.com/wagtail/wagtail/issues/459#issuecomment-119488251 mentions:

Filtering on related fields while performing a search query looks like it might be tricky due to the way Django represents these in a QuerySet (they’re converted to a Join object https://github.com/django/django/blob/master/django/db/models/sql/datastructures.py#L28-L118).

[rgs258]

@chosak Exactly! That the search backend can’t figure out what to do with the filter on the related field means that, in the case of Postgres search backend, it seems to build a query that fails to properly engage the indexes and so takes considerably longer to run. We worked around this in our specific situation by just filtering the search results after the get_search_backend().search(...) returns.

Our workaround is fine for now, but it is inefficient and won’t scale all that well 😃. To that end, we’re going to also look for ways to have search become aware of related filter fields.

Thank you for sharing you findings, I think they definitely add detail to the problems that need to be addressed.

[rgs258]

My group would find value in this enhancement. @chosak thank you for writing it up!