datastore: very slow queries, single record by key

Ubuntu 16.04.3 LTS Python 3.6.3 google-cloud-datastore>=1.5.0

repro code:

from google.cloud import datastore
from time import time
start = time() * 1000
ds = datastore.Client('project-name')
key = ds.key('TestPerf', 'c10d6806-e3d4-4694-a8be-38f481817c')
entity = ds.get(key)
end = time() * 1000
print("duration : ", (end - start))

We are just starting out with google cloud platform and chose datastore as the option to store data because it seems the simplest, however we are noticing very slow performance on queries. Most common query is by primary key (above code) or = on one of the fields.

We are getting about ~200/300ms on the above query with a single entity or many entities in the DB. The queries are done either from a Compute Engine instance or Kubernetes clusters (both same perf).

We are not sure what to do from here, short of switching to a different DB option on the GCP platform. Any help would be greatly appreciated. Thanks!

[david-gang]

I am happy to hear that someone is raising this point. We experience the slowness with the Java client. First of all datastore is slower than cloud sql. Secondly there are fluctuations between 50ms and a couple of hundred ms . This forces us to use intermediate caches instead of querying the datastore directly.

[natasha-aleksandrova]

Thank you for information.

Your results look much better than mine. What could account for such drastic difference? Avg of ~20ms is much better and something we could work with (although coming from PostgreSQL with 1-2ms queries by the PK, it is still a little high).

Also, a little bit about our use case. We are working on REST endpoints and using Datastore to store our core entities like User, Driver, Company etc. They are fairly simple data structures, and most commonly our endpoints will do a single query by the Key to get the entity. So the code snippet I provided in the issue I reported is pretty much the basis of the code, init DS client, then do a get by Key.

Given our use case, would SQL be more suitable for us? Or anything we can do to optimize Datastore for us?

Thanks! Natasha