Integrating NDB into gcloud.datastore?

Had a few discussions with @GoogleCloudPlatform/cloud-datastore (particularly @pcostell) so I wanted to summarize the things we covered. This issue is to clarify the goals for Datastore support in gcloud-python, and discuss and decide on a course of action that everyone likes for how to reconcile what we have today with the other libraries out there.

---

Current state of the world

Currently, I see two styles that people want to use when interacting with Datastore:

1. a “simple” more key-value-based style (similar to Amazon’s Simple DB) where you’re CRUDding dictionaries and then adding some querying
2. a more advanced ORM style where you’re create models, and have some sort of schema defined in a Python file.

For the former, gcloud.datastore has had the goal of covering this use case. For the latter, ndb is the latest way (supported) way of doing this – with others potentially existing, but ndb seems to be the clear leader.

We also have a unique situation where our code currently might have trouble running in App Engine – whereas ndb can’t run outside of App Engine. The layout sort of looks like this:

Looking forward

If our goals are …

1. We agree that both styles of interacting with Datastore matter (and both should exist)
2. gcloud.datastore and ndb are our choices for each style respectively
3. gcloud.datastore and ndb should both run in App Engine and non-App Engine runtimes
4. gcloud.datastore is where all the recommended Python stuff to talk to Datastore should live (it is the “official source of truth”)
5. People who want to write new Python libraries for Datastore can rely on code that exists in gcloud.datastore (and set gcloud as a Python dependency)

… then I’d like to suggest that we…

1. Port ndb over as gcloud.datastore.ndb (bringing with it datastore_rpc and datastore_query)
2. Rewrite gcloud.datastore to run on top of datastore_query
3. Rename gcloud.datastore to be a peer with ndb (using “simple” in this diagram, not set on that at all though).

which makes things look like this:

What do you guys think?

/cc @GoogleCloudPlatform/cloud-datastore @dhermes @tseaver @silvolu @proppy

[mattwarrenrnp]

I’d like to use ndb on a python 3.5 project in the flexible environment. would love to see some progress on making this possible.

[jgeewax]

@magoarcano : See @pcostell’s note from above, specifically https://github.com/GoogleCloudPlatform/datastore-ndb-python/blob/master/demo/task_list.py which is a demo app using NDB that runs inside GCE. Pay special attention to the configuration settings that turn things off.

Also, As Patrick says, the performance of this will be sub-par due to synchronous requests and no caching.

It’s important to note that this is an incredibly complex undertaking because the run time environments are significantly different and NDB was built on the premise that it would only ever run in App Engine (which is an “all or nothing” service with Datastore, Memcache, Task Queues, etc all available via RPC calls). In Flex (or GCE, or AWS, etc) things are very different so a majority of those assumptions don’t hold anymore.

This means that even though a port of NDB might work, there will be a huge number of “Oh, we didn’t realize NDB made that assumption!” moments, so we’re being extra cautious about what we release to the world. We don’t want to hand out code that works only in the right circumstances – it should work everywhere.