MongoDB 5.0.6 Compatibility Issues

Hey,

firstup please tell me what additional information is needed to debug this error.

I ran the following command: mongo-connector --unique-key=id -n news-articles.articles -m localhost:27017 -t http://localhost:8983/solr/mongo_solr_collection -d solr_doc_manager

(Yes i am calling the collection with -t because the core does not work (I get an Error 404). )

The connector starts and logs the following:

2022-02-15 18:23:34,584 [ALWAYS] mongo_connector.connector:50 - Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
2022-02-15 18:23:34,586 [ALWAYS] mongo_connector.connector:50 - Platform: Linux-5.10.0-11-amd64-x86_64-with-glibc2.31
2022-02-15 18:23:34,587 [ALWAYS] mongo_connector.connector:50 - pymongo version: 4.0.1
2022-02-15 18:23:34,587 [WARNING] mongo_connector.connector:170 - MongoConnector: Can't find /srv/news_crawler/oplog.timestamp, attempting to create an empty progress log
2022-02-15 18:23:34,597 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 5.0.6
2022-02-15 18:23:34,597 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.solr_doc_manager version: 0.1.0
2022-02-15 18:25:33,715 [ERROR] mongo_connector.util:97 - Call to Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='rs0'), 'news-articles'), 'collection_names') failed too many times in retry_until_ok
Traceback (most recent call last):
  File "/srv/news_crawler/venv/lib/python3.9/site-packages/mongo_connector/util.py", line 79, in retry_until_ok
    return func(*args, **kwargs)
  File "/srv/news_crawler/venv/lib/python3.9/site-packages/pymongo/collection.py", line 2579, in __call__
    raise TypeError("'Collection' object is not callable. If you "
TypeError: 'Collection' object is not callable. If you meant to call the 'collection_names' method on a 'Database' object it is failing because no such method exists.

I don’t know what to do with this. MongoDB is up and running:

Current Mongosh Log ID:	620be3f843ef12a095c537e4
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.1.9
Using MongoDB:		5.0.6
Using Mongosh:		1.1.9
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
   The server generated these startup warnings when booting:
   2022-02-15T17:19:31.349+01:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2022-02-15T17:19:32.383+01:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2022-02-15T17:19:32.384+01:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
Warning: Found ~/.mongorc.js, but not ~/.mongoshrc.js. ~/.mongorc.js will not be loaded.
  You may want to copy or rename ~/.mongorc.js to ~/.mongoshrc.js.
rs0 [direct: primary] test>

and the designated collection has several thousand items in it:

rs0 [direct: primary] news-articles> db.articles.find().count()
147203

Is rs0 setup wrong?

And solr is also up and running:

$ curl http://localhost:8983/solr/admin/cores?action=STATUS
{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "initFailures":{},
  "status":{
    "mongo_solr_collection":{
      "name":"mongo_solr_collection",
      "instanceDir":"/srv/solr-8.11.1/server/solr/mongo_solr_collection",
      "dataDir":"/srv/solr-8.11.1/server/solr/mongo_solr_collection/data/",
      "config":"solrconfig.xml",
      "schema":"managed-schema",
      "startTime":"2022-02-15T15:09:55.153Z",
      "uptime":9084767,
      "index":{
        "numDocs":0,
        "maxDoc":0,
        "deletedDocs":0,
        "indexHeapUsageBytes":0,
        "version":2,
        "segmentCount":0,
        "current":true,
        "hasDeletions":false,
        "directory":"org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/srv/solr-8.11.1/server/solr/mongo_solr_collection/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d83b5e6; maxCacheMB=48.0 maxMergeSizeMB=4.0)",
        "segmentsFile":"segments_1",
        "segmentsFileSizeInBytes":69,
        "userData":{},
        "sizeInBytes":69,
        "size":"69 bytes"}}}}

I really don’t know what I am doing wrong. Would be great if someone could point me in the right direction. 🙏

[ImranJagari]

Hi @bablf I got the same error as you with MongoDB, i have found a workaround if you’re using Solr

Solr Part

---

solr_doc_manager.py just change this at line 292

            batch = list(next(cleaned) for i in range(self.chunk_size))
            while batch:
                self.solr.add(batch, **add_kwargs)
                batch = list(next(cleaned)
                             for i in range(self.chunk_size))

by this

            try:
                batch = list(next(cleaned) for i in range(self.chunk_size))
                while batch:
                    self.solr.add(batch, **add_kwargs)
                    batch = list(next(cleaned)
                                for i in range(self.chunk_size))
            except Exception:
                pass 

downgrade pysolr to 3.8.1

---

Mongo Part

---

downgrade pymongo to 2.9

---

this workaround works for MongoDB 5.0.6 with mongo-connector 3.1.1, pymongo 2.9 and pysolr 3.8.1

Hope it helps you

[gurmitteotia]

Thanks @RicardoM17 ,

I did investigation and found out reason why document update was failing. I have also published a fix.

Primary reason is that “oplog” entry format has been changed in mongodb 5.x for document update. In Mongodb 4.x following oplog entry is generated when fields are changed -

{
	"op" : "u",
	"ns" : "mydb.emps",
	"ui" : UUID("79dc60c3-2fa9-454f-a84e-8b6d694d4ad3"),
	"o" : {
		"$v" : 1,
		"$set" : {
			"company_name" : "new company"
		}
	},
	"o2" : {
		"_id" : ObjectId("6319e61bc84d0826deef55e8")
	},
	"ts" : Timestamp(1662641739, 1),
	"t" : NumberLong(1),
	"wall" : ISODate("2022-09-08T12:55:39.544Z"),
	"v" : NumberLong(2)
}

However in case of mongodb 5 oplog entry has been changed to the following format-

{
	"op" : "u",
	"ns" : "mydb.col",
	"ui" : UUID("beebd3ac-fc25-41c9-98cf-90c87bfe79e0"),
	"o" : {
		"$v" : 2,
		"diff" : {
			"u" : {
				"company_name" : "new name",
				"website_url" : "https://www.newname.com1"
			}
		}
	},
	"o2" : {
		"_id" : ObjectId("5fa1ab5c184246d62af8fc46")
	},
	"ts" : Timestamp(1662644253, 1),
	"t" : NumberLong(4),
	"v" : NumberLong(2),
	"wall" : ISODate("2022-09-08T13:37:33.238Z")
}

And solr-doc-manager plug-in was misinterpreting the “update” document and adding it as new document, which was being rejected by solr. I have put a fix in solr-doc-manager plugin. You can find it in branch - mongo5-fix. I have added unit test to make sure the fix work for both mongodb 4 and mongodb 5. I will try if my fix can be reviewed and merged in the main repository.

Summary The last released versions of mongo-connector (3.1.1) and solr-doc-manager (0.1.0) will not work with latest version of mongodb (5.0.9) and solr (9.0.0). You will encounter two issues-

• Pymongo error as originally reported by @bablf. Fix is to install pymongo version 3.12.3
• Update of document will fail. Solution is described above.