Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Not Found error when using `scan`

See original GitHub issue

I’m attempting a large SCAN query using the scan helper function provided by this library. The operations takes about 1-2 hours. After about an hour or so of operation, we get a the following error:

GET /_search/scroll?scroll=5m [status:404 request:0.015s]
Traceback (most recent call last):
  File "/Users/stephenbly/themuse/venv/bin/oz", line 9, in <module>
    load_entry_point('Oz', 'console_scripts', 'oz')()
  File "/Users/stephenbly/themuse/venv/src/oz/oz/cli.py", line 17, in main
    retr = optfn.run(list(oz._actions.values()))
  File "/Users/stephenbly/themuse/venv/lib/python3.5/site-packages/optfn.py", line 128, in run
    return func(*args, **kwargs)
  File "./themuse/common/actions/catalogers/sync_catalogs.py", line 12, in blueshift_sync_catalogs
    jobs.catalog()
  File "./themuse/common/actions/catalogers/jobs.py", line 54, in catalog
    core.catalog(ranking_engine, blueshift_prod_category_uuid, blueshift_sandbox_category_uuid, item_filter, item_map)
  File "./themuse/common/actions/catalogers/core.py", line 34, in catalog
    for result in ranking_engine_cls(db, items_per_page=blueshift_products_per_request).scan_all():
  File "/Users/stephenbly/themuse/musecore/musecore/rankings.py", line 118, in scan_all
    for search_result in search_client.scan(index=self._get_index()):
  File "/Users/stephenbly/themuse/musecore/musecore/search_client.py", line 49, in scan
    for result in results:
  File "/Users/stephenbly/themuse/venv/lib/python3.5/site-packages/elasticsearch/helpers/__init__.py", line 285, in scan
    resp = client.scroll(scroll_id, scroll=scroll)
  File "/Users/stephenbly/themuse/venv/lib/python3.5/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Users/stephenbly/themuse/venv/lib/python3.5/site-packages/elasticsearch/client/__init__.py", line 662, in scroll
    params=params, body=body)
  File "/Users/stephenbly/themuse/venv/lib/python3.5/site-packages/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/Users/stephenbly/themuse/venv/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 106, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/stephenbly/themuse/venv/lib/python3.5/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: TransportError(404, '{"_scroll_id":"c2NhbjswOzE7dG90YWxfaGl0czo3MzEzOTs=","took":1,"timed_out":false,"_shards":{"total":1,"successful":0,"failed":1,"failures":[{"shard":-1,"index":null,"reason":{"type":"search_context_missing_exception","reason":"No search context found for id [1847760]"}}]},"hits":{"total":73139,"max_score":0.0,"hits":[]}}')

It seems like an error with the library if the search context is disappearing (which I believe means an incorrect search ID was used).