Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Mongodb collection not syncing to elasticsearch

See original GitHub issue

mongo-connector -m remoteip:27017 -t localhost:9200 -d elastic2_doc_manager --namespace-set m_new2.tags -v Im trying to sync mongocollection m_new2.tag from remote server. … but its not syncing. other collections are syncing very well.

but tags collection is not syncing.

Here is mongo-connector.log

2016-10-11 13:47:10,439 [DEBUG] mongo_connector.oplog_manager:257 - OplogThread: Iterating through cursor, document number in this cursor is 640
2016-10-11 13:47:10,439 [DEBUG] mongo_connector.oplog_manager:303 - OplogThread: Skipping oplog entry: 'm_new2.downloads' is not in the namespace set.

Elasticsearch version : 2.4.1

Mongodb version :2.6.12

Mongo-connector latest version :2.5.0

I also posted more information in stackoverflow http://stackoverflow.com/questions/39963348/elasticsearch-not-indexing-some-data-from-mongodb

Issue Analytics

State:
Created 7 years ago
Comments:13 (7 by maintainers)

Top GitHub Comments

1reaction

satsamcommented, Oct 11, 2016

I tired with mongo-connector 2.4 and got different logs mongo-connector -m remoteip:27017 -t localhost:9200 -d elastic2_doc_manager --namespace-set m_new2.tags -v

2016-10-11 21:59:54,575 [INFO] mongo_connector.connector:1062 - Beginning Mongo Connector
2016-10-11 21:59:54,611 [INFO] mongo_connector.oplog_manager:92 - OplogThread: Initializing oplog thread
2016-10-11 21:59:54,638 [INFO] mongo_connector.connector:296 - MongoConnector: Starting connection thread MongoClient(host=['remotehost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset=u'rs0')
2016-10-11 21:59:54,639 [DEBUG] mongo_connector.oplog_manager:196 - OplogThread: Run thread started
2016-10-11 21:59:54,640 [DEBUG] mongo_connector.oplog_manager:198 - OplogThread: Getting cursor
2016-10-11 21:59:54,640 [DEBUG] mongo_connector.oplog_manager:772 - OplogThread: reading last checkpoint as Timestamp(1476217286, 1) 
2016-10-11 21:59:54,641 [DEBUG] mongo_connector.oplog_manager:745 - OplogThread: oplog checkpoint updated to Timestamp(1476217286, 1)
2016-10-11 21:59:54,649 [DEBUG] mongo_connector.oplog_manager:700 - OplogThread: Initiating rollback from get_oplog_cursor
2016-10-11 21:59:54,649 [DEBUG] mongo_connector.oplog_manager:784 - OplogThread: Initiating rollback sequence to bring system into a consistent state.
2016-10-11 21:59:54,650 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:54,651 [INFO] urllib3.connectionpool:203 - Starting new HTTP connection (1): localhost
2016-10-11 21:59:54,691 [DEBUG] urllib3.connectionpool:383 - "POST /_refresh HTTP/1.1" 200 51
2016-10-11 21:59:54,692 [INFO] elasticsearch:66 - POST http://localhost:9200/_refresh [status:200 request:0.042s]
2016-10-11 21:59:54,692 [DEBUG] elasticsearch:68 - > None
2016-10-11 21:59:54,692 [DEBUG] elasticsearch:69 - < {"_shards":{"total":50,"successful":25,"failed":0}}
2016-10-11 21:59:54,694 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:55,138 [DEBUG] urllib3.connectionpool:383 - "GET /mongodb_meta/_search?size=1 HTTP/1.1" 200 327
2016-10-11 21:59:55,139 [INFO] elasticsearch:66 - GET http://localhost:9200/mongodb_meta/_search?size=1 [status:200 request:0.444s]
2016-10-11 21:59:55,139 [DEBUG] elasticsearch:68 - > {"sort": [{"_ts": "desc"}], "query": {"match_all": {}}}
2016-10-11 21:59:55,139 [DEBUG] elasticsearch:69 - < {"took":403,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":211034,"max_score":null,"hits":[{"_index":"mongodb_meta","_type":"mongodb_meta","_id":"57fd49c63f27df4835b236e9","_score":null,"_source":{"ns": "m_new2.downloads", "_ts": 6340304965159878657},"sort":[6340304965159878657]}]}}
2016-10-11 21:59:55,146 [DEBUG] mongo_connector.oplog_manager:809 - OplogThread: last oplog entry is {u'h': -6183552774918902853L, u'ts': Timestamp(1476217286, 1), u'o': {u'date': datetime.datetime(2016, 10, 11, 0, 0), u'song_id': u'57f017ec3f27df7b0d851508', u'_id': ObjectId('57fd49c63f27df4835b236e9'), u'time': 1476217286L}, u'v': 2, u'ns': u'm_new2.downloads', u'op': u'i'}
2016-10-11 21:59:55,147 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:55,336 [DEBUG] urllib3.connectionpool:383 - "GET /mongodb_meta/_search?search_type=scan&scroll=10m&size=1000 HTTP/1.1" 200 339
2016-10-11 21:59:55,337 [INFO] elasticsearch:66 - GET http://localhost:9200/mongodb_meta/_search?search_type=scan&scroll=10m&size=1000 [status:200 request:0.190s]
2016-10-11 21:59:55,337 [DEBUG] elasticsearch:68 - > {"query": {"filtered": {"filter": {"range": {"_ts": {"gte": 6340304965159878657, "lte": 6340304965159878657}}}}}}
2016-10-11 21:59:55,337 [DEBUG] elasticsearch:69 - < {"_scroll_id":"c2Nhbjs1OzY6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTs3OkJpZmFXdWpHUkJ1c1R1c252UTFCR0E7ODpCaWZhV3VqR1JCdXNUdXNudlExQkdBOzk6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTsxMDpCaWZhV3VqR1JCdXNUdXNudlExQkdBOzE7dG90YWxfaGl0czoxOw==","took":165,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[]}}
2016-10-11 21:59:55,339 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:55,441 [DEBUG] urllib3.connectionpool:383 - "GET /_search/scroll?scroll=10m HTTP/1.1" 200 338
2016-10-11 21:59:55,442 [INFO] elasticsearch:66 - GET http://localhost:9200/_search/scroll?scroll=10m [status:200 request:0.103s]
2016-10-11 21:59:55,442 [DEBUG] elasticsearch:68 - > c2Nhbjs1OzY6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTs3OkJpZmFXdWpHUkJ1c1R1c252UTFCR0E7ODpCaWZhV3VqR1JCdXNUdXNudlExQkdBOzk6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTsxMDpCaWZhV3VqR1JCdXNUdXNudlExQkdBOzE7dG90YWxfaGl0czoxOw==
2016-10-11 21:59:55,443 [DEBUG] elasticsearch:69 - < {"_scroll_id":"c2NhbjswOzE7dG90YWxfaGl0czoxOw==","took":91,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[{"_index":"mongodb_meta","_type":"mongodb_meta","_id":"57fd49c63f27df4835b236e9","_score":0.0,"_source":{"ns": "m_new2.downloads", "_ts": 6340304965159878657}}]}}
2016-10-11 21:59:55,444 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:55,448 [DEBUG] urllib3.connectionpool:383 - "GET /_search/scroll?scroll=10m HTTP/1.1" 200 169
2016-10-11 21:59:55,448 [INFO] elasticsearch:66 - GET http://localhost:9200/_search/scroll?scroll=10m [status:200 request:0.005s]
2016-10-11 21:59:55,449 [DEBUG] elasticsearch:68 - > c2NhbjswOzE7dG90YWxfaGl0czoxOw==
2016-10-11 21:59:55,449 [DEBUG] elasticsearch:69 - < {"_scroll_id":"c2NhbjswOzE7dG90YWxfaGl0czoxOw==","took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[]}}
2016-10-11 21:59:55,450 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:55,473 [DEBUG] urllib3.connectionpool:383 - "DELETE /_search/scroll HTTP/1.1" 404 2
2016-10-11 21:59:55,474 [INFO] elasticsearch:66 - DELETE http://localhost:9200/_search/scroll [status:404 request:0.024s]
2016-10-11 21:59:55,475 [DEBUG] elasticsearch:68 - > {"scroll_id": ["c2NhbjswOzE7dG90YWxfaGl0czoxOw=="]}
2016-10-11 21:59:55,475 [DEBUG] elasticsearch:69 - < {}
2016-10-11 21:59:55,488 [DEBUG] mongo_connector.oplog_manager:868 - OplogThread: Rollback, removing inconsistent docs.
2016-10-11 21:59:55,488 [DEBUG] mongo_connector.oplog_manager:887 - OplogThread: Rollback, removed 0 docs.
2016-10-11 21:59:55,488 [DEBUG] mongo_connector.oplog_manager:890 - OplogThread: Rollback, inserting documents from mongo.
2016-10-11 21:59:55,489 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:55,736 [DEBUG] urllib3.connectionpool:383 - "PUT /m_new2/downloads/57fd49c63f27df4835b236e9?refresh=false HTTP/1.1" 200 161
2016-10-11 21:59:55,737 [INFO] elasticsearch:66 - PUT http://localhost:9200/m_new2/downloads/57fd49c63f27df4835b236e9?refresh=false [status:200 request:0.248s]
2016-10-11 21:59:55,737 [DEBUG] elasticsearch:68 - > {"date": "2016-10-11T00:00:00", "song_id": "57f017ec3f27df7b0d851508", "time": 1476217286}
2016-10-11 21:59:55,737 [DEBUG] elasticsearch:69 - < {"_index":"m_new2","_type":"downloads","_id":"57fd49c63f27df4835b236e9","_version":3709,"_shards":{"total":2,"successful":1,"failed":0},"created":false}
2016-10-11 21:59:55,739 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:55,769 [DEBUG] urllib3.connectionpool:383 - "PUT /mongodb_meta/mongodb_meta/57fd49c63f27df4835b236e9?refresh=false HTTP/1.1" 200 161
2016-10-11 21:59:55,770 [INFO] elasticsearch:66 - PUT http://localhost:9200/mongodb_meta/mongodb_meta/57fd49c63f27df4835b236e9?refresh=false [status:200 request:0.031s]
2016-10-11 21:59:55,771 [DEBUG] elasticsearch:68 - > {"ns": "m_new2.downloads", "_ts": 6340304965159878657}
2016-10-11 21:59:55,771 [DEBUG] elasticsearch:69 - < {"_index":"mongodb_meta","_type":"mongodb_meta","_id":"57fd49c63f27df4835b236e9","_version":3710,"_shards":{"total":2,"successful":1,"failed":0},"created":false}
2016-10-11 21:59:55,772 [DEBUG] mongo_connector.oplog_manager:908 - OplogThread: Rollback, Successfully inserted 1  documents and failed to insert 0 documents.  Returning a rollback cutoff time of Timestamp(1476217286, 1) 
2016-10-11 21:59:55,772 [DEBUG] mongo_connector.oplog_manager:745 - OplogThread: oplog checkpoint updated to Timestamp(1476217286, 1)
2016-10-11 21:59:55,773 [DEBUG] mongo_connector.oplog_manager:772 - OplogThread: reading last checkpoint as Timestamp(1476217286, 1) 
2016-10-11 21:59:55,773 [DEBUG] mongo_connector.oplog_manager:745 - OplogThread: oplog checkpoint updated to Timestamp(1476217286, 1)
2016-10-11 21:59:55,781 [DEBUG] mongo_connector.oplog_manager:700 - OplogThread: Initiating rollback from get_oplog_cursor
2016-10-11 21:59:55,781 [DEBUG] mongo_connector.oplog_manager:784 - OplogThread: Initiating rollback sequence to bring system into a consistent state.
2016-10-11 21:59:55,782 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:56,024 [DEBUG] urllib3.connectionpool:383 - "POST /_refresh HTTP/1.1" 200 51
2016-10-11 21:59:56,025 [INFO] elasticsearch:66 - POST http://localhost:9200/_refresh [status:200 request:0.243s]
2016-10-11 21:59:56,025 [DEBUG] elasticsearch:68 - > None
2016-10-11 21:59:56,026 [DEBUG] elasticsearch:69 - < {"_shards":{"total":50,"successful":25,"failed":0}}
2016-10-11 21:59:56,027 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:56,175 [DEBUG] urllib3.connectionpool:383 - "GET /mongodb_meta/_search?size=1 HTTP/1.1" 200 327
2016-10-11 21:59:56,175 [INFO] elasticsearch:66 - GET http://localhost:9200/mongodb_meta/_search?size=1 [status:200 request:0.149s]
2016-10-11 21:59:56,176 [DEBUG] elasticsearch:68 - > {"sort": [{"_ts": "desc"}], "query": {"match_all": {}}}
2016-10-11 21:59:56,176 [DEBUG] elasticsearch:69 - < {"took":143,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":211034,"max_score":null,"hits":[{"_index":"mongodb_meta","_type":"mongodb_meta","_id":"57fd49c63f27df4835b236e9","_score":null,"_source":{"ns": "m_new2.downloads", "_ts": 6340304965159878657},"sort":[6340304965159878657]}]}}
2016-10-11 21:59:56,184 [DEBUG] mongo_connector.oplog_manager:809 - OplogThread: last oplog entry is {u'h': -6183552774918902853L, u'ts': Timestamp(1476217286, 1), u'o': {u'date': datetime.datetime(2016, 10, 11, 0, 0), u'song_id': u'57f017ec3f27df7b0d851508', u'_id': ObjectId('57fd49c63f27df4835b236e9'), u'time': 1476217286L}, u'v': 2, u'ns': u'm_new2.downloads', u'op': u'i'}
2016-10-11 21:59:56,185 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:56,224 [DEBUG] urllib3.connectionpool:383 - "GET /mongodb_meta/_search?search_type=scan&scroll=10m&size=1000 HTTP/1.1" 200 342
2016-10-11 21:59:56,225 [INFO] elasticsearch:66 - GET http://localhost:9200/mongodb_meta/_search?search_type=scan&scroll=10m&size=1000 [status:200 request:0.041s]
2016-10-11 21:59:56,225 [DEBUG] elasticsearch:68 - > {"query": {"filtered": {"filter": {"range": {"_ts": {"gte": 6340304965159878657, "lte": 6340304965159878657}}}}}}
2016-10-11 21:59:56,226 [DEBUG] elasticsearch:69 - < {"_scroll_id":"c2Nhbjs1OzE2OkJpZmFXdWpHUkJ1c1R1c252UTFCR0E7MjA6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTsxNzpCaWZhV3VqR1JCdXNUdXNudlExQkdBOzE4OkJpZmFXdWpHUkJ1c1R1c252UTFCR0E7MTk6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTsxO3RvdGFsX2hpdHM6MTs=","took":36,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[]}}
2016-10-11 21:59:56,227 [DEBUG] urllib3.util.retry:155 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-10-11 21:59:56,266 [DEBUG] urllib3.connectionpool:383 - "GET /_search/scroll?scroll=10m HTTP/1.1" 200 338
2016-10-11 21:59:56,267 [INFO] elasticsearch:66 - GET http://localhost:9200/_search/scroll?scroll=10m [status:200 request:0.040s]
2016-10-11 21:59:56,268 [DEBUG] elasticsearch:68 - > c2Nhbjs1OzE2OkJpZmFXdWpHUkJ1c1R1c252UTFCR0E7MjA6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTsxNzpCaWZhV3VqR1JCdXNUdXNudlExQkdBOzE4OkJpZmFXdWpHUkJ1c1R1c252UTFCR0E7MTk6QmlmYVd1akdSQnVzVHVzbnZRMUJHQTsxO3RvdGFsX2hpdHM6MTs=

0reactions

ShaneHarveycommented, Oct 28, 2016

Thanks, feel free to update this issue if you find out how to reproduce this problem.