mongo-connector oplog error after ReadTimeoutError

Hi guys. I am using mongo-connector to fetch data from the mongodb into elasticsearch.

When my DB gets bigger and i do send requests to mongodb the mongo-connector sometimes raises the following error:

  2016-01-28 12:20:19,436 [WARNING] elasticsearch:82 - PUT http://localhost:9200/mongodb_meta/mongodb_meta/56a9f968a87401d779db2fe7?refresh=false [status:N/A request:10.604s]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 74, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/retry.py", line 222, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
ReadTimeoutError: HTTPConnectionPool(host=u'localhost', port=9200): Read timed out. (read timeout=10)

a view seconds before i was starting an aggregation in the mongodb

[12:20:02.134] - info: starting TenMinutesAggregation Job

The aggregation Job also removes some entries from the mongodb. It seems that ElasticSearch had a small time frame where it was not responsing.

Is it normal that the mongo-connector is crashing in such a case?

And now when i try to restart the mongo-connector it tells me:

2016-01-28 12:20:22,920 [ERROR] mongo_connector.util:87 - Fatal Exception
2016-01-28 12:26:20,504 [ERROR] mongo_connector.oplog_manager:162 - OplogThread: Last entry no longer in oplog cannot recover! Collection(Database(MongoClient(u'localhost', 27017), u'local'), u'oplog.rs')
2016-01-28 12:26:21,385 [ERROR] __main__:302 - MongoConnector: OplogThread <OplogThread(Thread-2, stopped 139793134741248)> unexpectedly stopped! Shutting down

What can i do now? What do i have to do to make this setup stable? Please help me out here. I have installed mongo-connector about 3 month ago via pip.

BR

[aherlihy]

Hi @mikkhait @JuHwon @prigozhinroman this is on the to-do list, there are a few other issues to resolve first but it’s on the way!

[llvtt]

@JuHwon, I just merged a PR that fixes catching some additional exception types. Would you be willing to try against the latest version from the master branch to see if your problem is resolved?