Spanner sessions keep increasing

Our Spanner Sessions keep increasing until we restart the pods (we create a new database object on pod initialize). https://github.com/googleapis/google-cloud-node

We don’t see any impact on the pods’ memory and CPU. It looks like the only impact is on the max latency of the query-by-id and query-by-column

Environment details

• OS: alpine
• Node.js version: node:14
• npm version: 6.14.13
• @google-cloud/spanner version: 5.9.0

Steps to reproduce

We create a single database instance and we use that database for all queries and transactions. (please advise if that’s the right approach).

            const queryOptions = { optimizerVersion: 'latest' };
            this.db = instance.database(databaseId, undefined, queryOptions);

[daneshka]

@skuruppu Thanks for the follow-up. Unfortunately, we haven’t deployed that to prod yet. But our stage environment looks much better. I will post here after I get some data from prod.

[olavloite]

@daneshka

The query count is 33602 for that query in one hour, which is not a high number. What’s your suggestion on that?

That is indeed not a high number, unless those 33602 queries are all executed within a very short fraction of one hour (see below).

• Use more pods? This should not be the best option because the existing pods already have lots of free resources
• Create more database instances in each pod?
• any other solution?

I don’t think you should increase the number of pods or number of database instances in each pod based on what we are seeing here. The only configuration change that I think you should consider, is increasing the max number of sessions in the session pool for the databases instances that you are using. That would mean instantiating the database instances with something like this:

// 400 means max 400 sessions, this again means that you can execute at most 400 parallel queries and/or
// transactions using this database instance. Increase this number even further if your application has bursts
// that will cause even more queries to be executed in parallel.
const database = instance.database('some-database', {max: 400});

The database.run(query) statement should not cause any session leaks, so I really don’t think that is the real problem (unless the query is taking a long time to finish, which the above information indicates that it does not).

One possibility in this case could be that the query that you are seeing is one that comes in bursts. I noticed that the table that is being queried is called reminders. If that query for example is coming from some timed (background) task, and very many instances of that task can be executed at the same time, then that could be the trigger for the problem (and also the reason that you are seeing this query many times in the error message). So that could mean that:

1. There’s no real session leak here (or otherwise only a small one coming from something else, causing a slow build up of sessions)
2. The session pool is temporarily being exhausted by a large burst of read requests at the same time. The default session pool settings will allow a maximum of 100 sessions to be created for one database instance, which effectively means that a pod can execute at most 100 queries in parallel.