Grpc.Core.RpcException: Status(StatusCode=Unauthenticated, Detail="Deadline Exceeded") ?

We are deploying asp.net core 2.0 project to app engine flex(while previous version is under load: about 200req/sec), and in some cases we are getting “Grpc.Core.RpcException: Status(StatusCode=Unauthenticated, Detail=“Deadline Exceeded”)” while trying to run query on datastore with standard datastore client, sometimes errors are gone and sometimes they are rising again, not sure if it depends on requests rate. So here is our app.yaml:

env: flex
runtime: aspnetcore
resources:
 cpu: 16
 memory_gb: 14.4

automatic_scaling:
  min_num_instances: 8
  max_num_instances: 20
  cool_down_period_sec: 180
  cpu_utilization:
    target_utilization: 0.5 

And here is full stack trace:

Grpc.Core.RpcException: Status(StatusCode=Unauthenticated, Detail="Deadline Exceeded")
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at Grpc.Core.Internal.AsyncCall`2.UnaryCall
at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse]
at Grpc.Core.Internal.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse]
at Google.Cloud.Datastore.V1.Datastore.DatastoreClient.RunQuery
at Google.Api.Gax.Grpc.ApiCall.<>c__DisplayClass0_0`2.<Create>b__1
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass1_0`2.<WithRetry>b__0
at Google.Cloud.Datastore.V1.QueryStreamer.<Sync>d__7.MoveNext
at System.Collections.Generic.List`1.AddEnumerable
at System.Linq.Enumerable.ToList[TSource]
at Google.Cloud.Datastore.V1.LazyDatastoreQuery.GetAllResults

@vuukle @jskeet

[jskeet]

At this point I suspect you really need a Datastore expert to help you. I’ll reach out internally to try to find someone who can help.

[jskeet]

I’m afraid I don’t know details about Datastore itself - I work on a lot of different APIs, so don’t know very much about each of them. I just make sure you can get requests from C# code to the server…

Here’s a small sample for you for the pooling. You’d create an instance on start-up, then keep that instance everywhere, calling var db = pool.GetNextInstance(); each time you wanted to make a request. If you have a chance to test that out and see if it improves your latency, that would be really interesting.

using Google.Apis.Auth.OAuth2;
using Google.Cloud.Datastore.V1;
using Grpc.Auth;
using Grpc.Core;
using System.Linq;

namespace DatastoreTest
{
    // Sample code only; needs more work for productionization.

    /// <summary>
    /// Pool of DatastoreDb instances which are returned in round-robin order (they're never
    /// "returned" to the pool). Assumptions: default credentials, a single DB, and no need to
    /// release channels.
    /// </summary>
    public class RoundRobinDatastoreDbPool
    {
        private readonly DatastoreDb[] _instances;
        private int _nextIndex = 0;
        private readonly object _lock = new object();

        public RoundRobinDatastoreDbPool(string projectId, string namespaceId, int count)
        {
            var credentials = GoogleCredential.GetApplicationDefault()
                .CreateScoped(DatastoreClient.DefaultScopes) // Not necessary as of Grpc.Core 1.8
                .ToChannelCredentials();
            string host = DatastoreClient.DefaultEndpoint.Host;
            int port = DatastoreClient.DefaultEndpoint.Port;
            _instances = Enumerable.Range(0, count)
                .Select(_ => new Channel(host, port, credentials))
                .Select(channel => DatastoreClient.Create(channel))
                .Select(client => DatastoreDb.Create(projectId, namespaceId, client))
                .ToArray();
        }

        public DatastoreDb GetNextInstance()
        {
            lock (_lock)
            {
                DatastoreDb instance = _instances[_nextIndex++];
                if (_nextIndex == _instances.Length)
                {
                    _nextIndex = 0;
                }
                return instance;
            }
        }
    }
}