Users should be able to set the timeout and the chunk_size for elastic search bulk requests.

While dumping huge data to elasticsearch, the mongo-connector can’t work normally because it often crashes due to connection timeout. The default timeout is 10, and can’t be changed. It should be an option of the mongo-connector command so that user can change it when necessary.

2015-10-20 01:48:04,992 [CRITICAL] mongo_connector.oplog_manager:543 - Exception during collection dump
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/mongo_connector/oplog_manager.py", line 495, in do_dump
    upsert_all(dm)
  File "/usr/lib/python2.6/site-packages/mongo_connector/oplog_manager.py", line 479, in upsert_all
    dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)
  File "/usr/lib/python2.6/site-packages/mongo_connector/util.py", line 32, in wrapped
    return f(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/mongo_connector/doc_managers/elastic_doc_manager.py", line 190, in bulk_upsert
    for ok, resp in responses:
  File "/usr/lib/python2.6/site-packages/elasticsearch/helpers/__init__.py", line 138, in streaming_bulk
    raise e
ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host=u'172.31.1.254', port=9200): Read timed out. (read timeout=10))

Thanks,

[Jinshui]

I made the following change (line 4) in elastic_doc_manager to increase the time out to 60, it solves my issue:

1           kw = {}
2           if self.chunk_size > 0:
3               kw['chunk_size'] = self.chunk_size
4+          kw['request_timeout'] = 60
5           responses = streaming_bulk(client=self.elastic,
6                                       actions=docs_to_upsert(),
7                                       **kw)

I hope the request_timeout can be a param of the mongo-connector command, or at least set the default time out to be a larger number, 10 seconds is too short.

[Jinshui]

Please close this issue, both of the timeout and chunk_size can be configured by the configuration file as follows:

{
  "docManagers": [
    {
      "docManager": "elastic_doc_manager",
      "targetURL": "localhost:9200",
      "bulkSize": 200,
      "autoCommitInterval": 0,
      "args": {
        "clientOptions": {"timeout": 60}
      }
    }
  ]
}

Thanks!