"CollectionInvalid: collection <CC> already exists" exception when catching up the op-log

My use case is moving a replica set from A to B.

1. I take an EBS snapshot of A, at time t1.
2. Seed B from that finishing at time t2.
3. Use mongo-connector bring B into sync with A via A’s oplog and keep it in sync. Use noDump to kick that off.

When step 3 starts the oplog is big, and it’s oldest entry is from time t0. It’s slower than one might like if t0 is much earlier than t1, but that’s ok.

But. I’m encountering a problem where mongo connector fails

2017-04-11 20:53:38.814307500  2017-04-11 20:53:38,813 [ERROR] mongo_connector.util:106 - Fatal Exception
2017-04-11 20:53:38.814321500  Traceback (most recent call last):
2017-04-11 20:53:38.814322500    File "/home/bhyde/.pyenv/versions/2.7.11/lib/python2.7/site-packages/mongo_connector/util.py", line 104, in wrapped
2017-04-11 20:53:38.814323500      func(*args, **kwargs)
2017-04-11 20:53:38.814324500    File "/home/bhyde/.pyenv/versions/2.7.11/lib/python2.7/site-packages/mongo_connector/oplog_manager.py", line 283, in run
2017-04-11 20:53:38.814325500      timestamp)
2017-04-11 20:53:38.814325500    File "/home/bhyde/.pyenv/versions/2.7.11/lib/python2.7/site-packages/mongo_connector/util.py", line 35, in wrapped
2017-04-11 20:53:38.814326500      return f(*args, **kwargs)
2017-04-11 20:53:38.814327500    File "/home/bhyde/.pyenv/versions/2.7.11/lib/python2.7/site-packages/mongo_connector/doc_managers/mongo_doc_manager.py", line 150, in handle_command
2017-04-11 20:53:38.814328500      self.mongo[new_db].create_collection(coll)
2017-04-11 20:53:38.814329500    File "/home/bhyde/.pyenv/versions/2.7.11/lib/python2.7/site-packages/pymongo/database.py", line 342, in create_collection
2017-04-11 20:53:38.814329500      raise CollectionInvalid("collection %s already exists" % name)
2017-04-11 20:53:38.814330500  CollectionInvalid: collection <CC> already exists

Which I presume means that <CC> was created during the interval t0…t1.

Mumble … idempotent … mumble.

I guess I’m thinking the work around is to catch the exception, delete the collection, and try again. Or preflight the operation and delete the collection before creating it.

[raffaeleguidi]

Progress on the issue seems to have stalled - but I am still facing it. I would suggest that a --continue-on-error flag would be useful for the oplog as well (–continue-on-oplog-error?) or at least something specific for the collection already exists error

[bhyde]

In my use case, where we don’t delete collections or tinker with their assorted properties; we only add 'em in the most vanilla way. So, I think i’m fine ignoring the error.

In the general case it’s accurate to catch the error, delete the collection, and then recreate the collection. At least I’d think so.

Seeding from the snapshot + noDump is a huge win if the database is huge; for example you don’t need to unwind the compression, serialize, move it over wires, etc. etc.