Add MongoDB storage backend?

I have been using MongoDB lately for logging the results of my computational experiments, and I have found that it is a pretty fun piece of software to use. Since MongoDB is designed to store large amounts of data across a distributed network, It seems like it could be a good match for zarr. I went ahead and made a very simple MutableMapping wrapper to MongoDB: https://gist.github.com/nbren12/9842cf22f173a864a7c8377c01ad06c5, and I am wondering if this is something that might be worth cleaning up and adding to zarr.

It would be interesting to benchmark it’s performance compared to S3 and GCS backends. FWIW, a package which seems to use mongoDB in a similar way to store large binary datasets is https://github.com/manahl/arctic.

[nbren12]

Yes, I agree this should probably be documented as part of Joe’s PR.

On Fri, Dec 28, 2018 at 1:13 AM jakirkham [e-mail] wrote:

TBH just keeping things within the 16MB limit may make sense for some work loads. For instance an uncompressed single precision 2048x2048 array would fit exactly in that limit. A larger array should be possible with compression. Of course this is assuming we want an array of that size in one chunk.

Anyways was mainly curious how this was working out for you in typical use cases. We probably should document this constraint for users less aware of what MongoDB is doing behind the scenes.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zarr-developers/zarr/issues/299#issuecomment-450323393, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUokp7pOKw98uMCzkjNaaqnbG4dxP1Dks5u9eCmgaJpZM4WxJS4 .

[alimanfoo]

@nbren12 re the client object I’d say it’s up to you.