Adding Documents Without a Primary Key Silently Fails

Description I’m trying to add a bird taxonomy to Meilisearch. When I add documents without a primary key, they fail to be added with no indication as to why. Per the docs a primary key is required, except in the case where a key can be inferred by having id (case insensitive) in one of the fields. This may point to an upstream bug.

Thanks for this project!

Expected behavior An error indicating that adding documents failed due to a missing primary key, and that no key could be inferred.

Current behavior Documents are not added and no error indicated why they failed.

Screenshots or Logs Logs from the container at debug log level on a failure:

[2021-08-29T03:50:59Z DEBUG meilisearch_http::routes::indexes::documents] called with params: Query(UpdateDocumentsQuery { primary_key: None })
[2021-08-29T03:50:59Z DEBUG meilisearch_http::routes::indexes::documents] returns: Enqueued(Enqueued { update_id: 1, meta: DocumentsAddition { method: ReplaceDocuments, format: Json, primary_key: None }, enqueued_at: 2021-08-29T03:50:59.777395727Z, content: Some(f8f82df4-6dad-408d-87ad-17f183548447) })
[2021-08-29T03:50:59Z DEBUG meilisearch_http::index_controller::index_actor::actor] Processing update 1
[2021-08-29T03:50:59Z INFO  actix_web::middleware::logger] 172.17.0.1:48466 "POST /indexes/birds-fail/documents HTTP/1.1" 202 14 "-" "python-requests/2.26.0" 0.006229

Test Case

import meilisearch
from time import sleep
from copy import deepcopy


if __name__ == "__main__":
    client = meilisearch.Client("http://127.0.0.1:7700", "masterKey")
    fail_index = client.index("birds-fail")
    pass_index = client.index("birds-pass")
    test_index = client.index("birds-test")
    test_documents = [
        {
            "common_name": "Common Ostrich",
            "species_code": "ostric2",
        },
        {
            "common_name": "Somali Ostrich",
            "species_code": "ostric3",
        },
    ]
    test_id_docs = deepcopy(test_documents)
    for x in range(len(test_id_docs)):
        test_id_docs[x]["test_id"] = x
    # Sleep is required, as we need to wait for indexing to complete.
    _ = fail_index.add_documents(test_documents)
    sleep(1)
    print("No primary key: ", fail_index.get_stats())
    _ = pass_index.add_documents(test_documents, primary_key="species_code")
    sleep(1)
    print("With primary key: ", pass_index.get_stats())
    _ = test_index.add_documents(test_id_docs)
    sleep(1)
    print("No primary key, test_id field: ", test_index.get_stats())

Test Output

No primary key:  {'numberOfDocuments': 0, 'isIndexing': False, 'fieldDistribution': {}}
With primary key:  {'numberOfDocuments': 2, 'isIndexing': False, 'fieldDistribution': {'common_name': 2, 'species_code': 2}}
No primary key, test_id field:  {'numberOfDocuments': 2, 'isIndexing': False, 'fieldDistribution': {'common_name': 2, 'species_code': 2, 'test_id': 2}}

Environment (please complete the following information):

• OS: Arch Linux
• MeiliSearch version: 0.21.0, docker: getmeili/meilisearch:latest
• meilisearch-python version: 0.16.0

[curquiza]

Seems to be solve, feel free to re-open if needed 🙂

[curquiza]

Hello @dfloer and @sanders41!

Thanks sanders for helping on this issue 🙂

Indeed, MeiliSearch tasks related to document insertions are asynchronous. More about the asynchronous tasks of MeilISearch: https://docs.meilisearch.com/learn/advanced/asynchronous_updates.html#asynchronous-updates It means you can have information about the error in the update status via these routes: https://docs.meilisearch.com/reference/api/updates.html#get-an-update-status. The message of the error in the update status is currently not really relevant (“missing primary key”), however:

• you can have more information about the error in errorLink
• we are re-doing the error handler of MeilISearch, with better user messages. If you are curious, you can check the specification: https://github.com/meilisearch/specifications/pull/61

So you need to catch the asynchronous errors by checking the status of the updates. The Sanders’ decorator is a really good alternative. Otherwise, you can code your own function!