Mongo connector is lag behind mongodb

Hi guys, We are using mongo-connector to feed data from mongodb to elasticsearch. Everything went well until today we have a big insert / update to mongodb and mongo-connector started falling behind and taking hours to catch-up. Here is the mongo-connector config:

"mainAddress": "10.a.b.c:27017",
    "oplogFile": "/var/log/mongo-connector/oplog.timestamp",
    "noDump": false,
    "batchSize": -1,
    "verbosity": 2,
    "continueOnError": true,

 "docManagers": [
        {
            "docManager": "elastic2_doc_manager",
            "targetURL": "10.x.y.z:9200",
            "bulkSize": 1000,
            "__uniqueKey": "_id",
            "args": {
                "clientOptions": {"timeout": 60}
                 }
        }

Elasticsearch is setup as cluster of 3 servers. I’m looking for anyway to make mongo-connector tailing and update data faster.

Thanks everyone.

[sliwinski-milosz]

Good to hear! 😃 Thank you for your patience. Now it is time to prepare some tests and make a pull request. I will do that soon.

[jmmk]

@luisobo @behackett I have been out of town for a few weeks, but I will try and see if I tidy up my solution and make an upstream PR.

@luisobo it’s not just the upserts that can/should be batched, it’s every operation. ElasticSearch allows a “bulk indexing operation” with any number of inserts/updates/deletes that will be run in the order specified (see: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html).

Mongo Connector is just passing each oplog operation to the doc manager to handle it however it sees fit. In order to properly replicate the database state, the operations must be run “one at a time” in the exact order they happened. So if Mongo Connector tried to do any batching logic, it may not be correct for how the downstream data store needs to handle the operations. But since we know we can batch them in a bulk indexing request and that they will be run in order, we can do that for ES to prevent unnecessary network calls.