the database system is in recovery mode

• EdgeDB Version: 1-beta1
• OS Version: it’s the docker image

Steps to Reproduce:

I’m not entirely sure. It seems to happen when I open “too many” connections and use them in parallel. But it’s quite erratic. With 2 connections, it worked fine. With 5, it crashed. With 4 connections, it worked fine. With 2, it crashed. (but it worked fine earlier)

I also tried with transaction but it’s the same. Sometimes it works, sometimes it fails.

    async for tx in connection.retrying_transaction():
        async with tx:

INFO 1 2021-03-04 19:30:21,109 edb.server: EdgeDB server (1.0-beta.1+g2eee1afe4.d20210301) starting.
INFO 1 2021-03-04 19:30:45,280 edb.server: Serving on 0.0.0.0:5656
INFO 1 2021-03-04 19:30:45,281 edb.server: Serving on /run/edgedb/.s.EDGEDB.5656
INFO 1 2021-03-04 19:30:45,282 edb.server: Serving admin on /run/edgedb/.s.EDGEDB.admin.5656
---- Exception occurred: the database system is in recovery mode ----

1. edb.server.pgcon.errors.BackendError: the database system is in recovery mode

---- Traceback ----

    /usr/lib/x86_64-linux-gnu/edgedb-server-1-beta1/lib/python3.9/site-packages/edb/server/server.py, line 721, in reconnect
        > conn = await self._pg_connect(defines.EDGEDB_SYSTEM_DB)
    /usr/lib/x86_64-linux-gnu/edgedb-server-1-beta1/lib/python3.9/site-packages/edb/server/server.py, line 214, in _pg_connect
        > return await pgcon.connect(self._get_pgaddr(), dbname)
    edb/server/pgcon/pgcon.pyx, line 130, in connect
        > if (self.lineno and
    edb/server/pgcon/pgcon.pyx, line 1255, in connect
        

edb.server.pgcon.errors.BackendError: the database system is in recovery mode
ERROR 1 2021-03-04 19:40:21,325 asyncio: Task exception was never retrieved
future: <Task finished name='Task-133' coro=<Server._on_sys_pgcon_connection_lost.<locals>.reconnect() done, defined at /usr/lib/x86_64-linux-gnu/edgedb-server-1-beta1/lib/python3.9/site-packages/edb/server/server.py:720> exception=BackendError('the database system is in recovery mode')>

[elprans]

• if I open a connection to EdgeDB, there is or there is not a new connection to PG?

EdgeDB maintains an internal dynamic pool of PostgreSQL connections, which, by default starts from 10 connections and grows up to 100 connections. That pool is quite smart and is similar in function to how standalone Postgres poolers (e.g. pgbouncer) work. If you don’t have much concurrency in your connections to EdgeDB and don’t have many connections to different databases, then the pool should stay around 10 connections.

• This --max-backend-connections=10, does it get applied on EdgeDB or PG?

These are 10 connections from EdgeDB to Postgres.

(I removed it) because I couldn’t make it work

That seems like a bug in edgedb-docker, we’ll take a look.

[elprans]

Maybe my question sounds confusing, but I’m actually confused that Postgres could crash because of memory.

Well, the gist is that the database service consists of multiple processes, some of which are spawned by EdgeDB and most are Postgres processes. Currently, EdgeDB creates a minimum of 10 connections to Postgres, each being a separate Postgres backend process, so you are probably looking at something like at least 20 processes running in the container.

That said, the ability to run on a 2GiB machine isn’t an unreasonable requirement (many starter-level VMs have that or even less), so we’ll see if we can implement a “resource-constrained” mode in which EdgeDB would limit the overall number of processes to some small amount.