Do not assume _id is ObjectId

OplogThread assumes that _id in the document is an ObjectId here and here

This also assumes the _ids returned by DocManager.search() and DocManager.get_last_doc() can be coerced to ObjectId.

But mongo does not require _id to be ObjectId.

_id ... may be of any type other than an array

The unit tests don’t even assume it is ObjectId. e.g. the test_solr_doc_manager.p unit tests for search use the string ‘1’ as the _id.

Most of the existing DocManagers coerce the _id to a string and store that, and return that string in search and get_last_doc and OplogThread attempts to coerce that string to ObjectId. So I think fixing this would require DocManagers to return the original _id instance and not a string.

[llvtt]

This is tricky; the _id field is coerced into a string for the purposes of storing it in Elastic/Solr, since ObjectId is a MongoDB-specific type. However, we have to turn those strings back into whatever their original type was, so that we can query MongoDB. This means we need to know the original type of _id, which isn’t encoded anywhere. It could even be different types from document to document (insane, perhaps, but possible).

This is obviously a bug, but I’m not sure how big the impact is. I’m guessing that most are using ObjectId as the type for _id, since that’s the default. Of those people who are using something else, only a fraction might experience a rollback that requires mongo-connector’s intervention (and thus hit this bug). The fix for this will require encoding the original type somehow, and will probably require a breaking change. 😦

[benlavalley]

Looking over the code in oplog_manager.py, I see the proposed changes here to encode the _id as extended JSON were never actually implemented (I’d believed they were based on the reply preceding mine thanking llvtt for resolving the problem).

And thanks for that formatting edit @jaraco, I see you 👍