Search docs: still warns about ES aliasing, but referenced bug appears fixed

The docs for ATOMIC_REBUILD say:

Warning

This option may not work on Elasticsearch version 5.4 and above, due to a bug in the handling of aliases affecting these releases.

But https://github.com/elastic/elasticsearch/pull/25043 appears to fix that issue, and the fix looks to be in >=5.5 - see https://github.com/elastic/elasticsearch/pull/25043#issuecomment-306594733.

So I think that this warning is wrong, and that the issue may only affect version 5.4, and that the warning may be incorrectly guiding people away from using ATOMIC_REBUILD with recent versions of Elasticsearch.

[gasman]

Thanks for the detective work! That sounds like a very plausible explanation of the problems seen on 5.6.3 in #3985 - will go ahead and update the note now.

[plumdog]

@gasman So I can’t make update_index fail.

But.

Comparing the code for the ES search backend between current master and https://github.com/wagtail/wagtail/pull/3637, I spotted a change that I think might explain why:

• time of the PR that avoids 5.4 in tests, https://github.com/wagtail/wagtail/blob/0029291685dbc86eec8fec106d94e0a83569c37d/wagtail/wagtailsearch/backends/elasticsearch.py#L512-L513
• master equivalent code, https://github.com/wagtail/wagtail/blob/master/wagtail/search/backends/elasticsearch2.py#L806-L807

This change was introduced in https://github.com/wagtail/wagtail/pull/4128 which fixed the usage of the ES client (because positional arguments changed order in ES Python client 1.7.0).

---

So I think that anyone using Wagtail without the #4128 fix, which is to say anything not >=2 or >=1.13.2 or >= 1.12.4 and using ES Python client >=1.7.0 would have had broken ATOMIC_REBUILD.

That would explain #3985 (given as “Wagtail Version 1.13”, but opened Oct 31st 2017, and fix didn’t go into 1.13 until Dec 14th 2017, so some unfixed 1.13 version), and “Elasticsearch PY Version: 5.4.0”, so would have had the broken behaviour that #4128 fixed.

Then as for why as of #3637 (June 7th 2017) you saw misbehaving ATOMIC_REBUILD behaviour, I wonder whether it was because this https://github.com/wagtail/wagtail/blob/609d38f17306bbe153184c850f0ecbbaa96f52d7/wagtail/wagtailsearch/tests/test_elasticsearch5_backend.py#L1021-L1023 would have always failed on ES 5.4, because the bug means this always returns True.

Against my test bed, using ES 5.4.2:

>>> from elasticsearch import Elasticsearch
>>> es = Elasticsearch(hosts=['elasticsearch'])
>>> some_index = es.cat.indices(format='json')[0]['index']
>>> print(some_index)
wagtail__wagtaildocs_document_ezdcnhp
>>> es.indices.exists_alias(name='doesnotexist', index=some_index)
True

which is lies, and would break that test. But on ES 5.5.2:

>>> from elasticsearch import Elasticsearch
>>> es = Elasticsearch(hosts=['elasticsearch'])
>>> some_index = es.cat.indices(format='json')[0]['index']
>>> print(some_index)
wagtail__wagtaildocs_document_rbndbkt
>>> es.indices.exists_alias(name='doesnotexist', index=some_index)
False

So I think that the tests failed because the assertions about whether or not aliases existed were getting lies from ES because of the ES 5.4 bug.

---

Which is all an extremely long-winded way of saying, if you are using:

• a Wagtail released after Feb 2018 (>=2, >=1.13.2, >= 1.12.4)
• an Elasticsearch released after July 2017 (!= 5.4.x)

then ATOMIC_REBUILD will work fine. And in fact, I think that a recent Wagtail on ES 5.4 might actually cope OK (at least, I couldn’t break it). But given it definitely does use the “check if alias for index exists” that will be getting lies from ES, so I think it is probably safest to say “ES 5.4 not supported for ATOMIC_REBUILD”.

So I think that means - thanks to #4128 - the issue really is at worst confined to ES 5.4, and the warning should say as much.