Serialization of RPC between main process and the compiler

Motivation

Currently, we have:

1. Main process and compiler process communicating over Unix socket
2. The protocol uses pickle as serialization
3. Compiler process connects to Postgres on its own

This isn’t going to work for the main process in Rust and supporting cloud:

1. While it’s possible to work with Pickle in Rust it isn’t a good idea
2. We want a many-to-many relationship between the processes in the cloud
3. Holding by a compiler onto a Postgres connection is inefficient (connections are costly in Postgres)

The Proposed Protocol

Framing

@tailhook is used to use WebSocket framing for everything nowadays:

1. It’s standard, ubiquitously supported
2. Sufficiently extensible
3. Exposing (Prometheus?) metrics over HTTP on the same port is a plus
4. It has a standard connection liveness protocol (Ping/Pong packets)

Alternative Framing

1. We could use HTTP for most things, but being able to find out connection liveness reliably is a very big advantage. And we’ll probably need some non-RPC messages, see below
2. Roll-your-own will require provisioning ping/pong and future extensions. While it isn’t particularly complex, I don’t think it’s justified.
3. Use postgres framing. The downsides are:
   • It’s just framing, we still need to specify how actual request/response, ping/pong and other packets look like
   • We still need to decide how to serialize or own data

Serialization

@tailhook is used to CBOR:

1. It’s standard
2. There is Concise data definition language (CDDL) to describe CBOR structures
3. It’s binary, fast and sufficiently extensible
4. It’s relatively compact. While preserving field names it’s still comparable to protobuf after gzipping (there is a standard compression extension in websocket protocol when we need one). We can also replace field names with 1-byte numbers or use arrays if we have lots of small objects.
5. There are fast encoders for every language including Python and Rust

Alternative Serialization

Lots of them:

1. Protobuf is not a starter.
2. Cap’n’proto is unnecessarily complex and bad at backward compatibility
3. JSON is bad at binary data and not compact enough
4. Postgres protocol: there are two different protocol serializations there: one for message structure and one for the data. The latter one requires describing types upfront and is suited only for tabular data, and also is not very compact. The former is not self-descriptive. Which makes debugging harder.

Can go over and over, have to just pick one.

Semantics

To make complier completely stateless, we move all communication to Postgres into the main process.

Compile To SQL

There are perhaps two API calls:

compile_to_sql(query_text, schema_hash) -> (compiled_query, Option<changes_to_schema>)
compile_to_sql_given_schema(query, schema) -> (compiled_query, Option<changes_to_schema>)

Note:

1. It’s stateless. If there is no schema with that hash compiler returns an error and the main process calls the second method.
2. The slow path here is when the actual schema changes, which is fine.
3. ❓ The DDLs probably don’t need any dedicated processes with these semantics
4. ❓ Schemas might not need to be tied to a user or a database. So multiple tutorial users can share the same schema if the hash matches.

There are few performance optimizations can be done on top:

1. The main process keeping track of which schemas are cached (i.e. compiler publishes hashes as it receives schemas)
2. Forcing preload of specific schemas when connecting to a compiler (this is probably only justified in the paid tier).

The obvious question here is: Is it fast enough to transfer schema like this?

1. Well, obviously we need to do it one way or another
2. For paid tier, it will always be preloaded anyway (but asserting on a hash is still useful)
3. For free tiers (i.e. try.edgedb.com) we should have a small limit on the schema size to avoid DoS (and hopefully we can reuse the same schema for most users)

Alternative Semantics

Well, there are a lot of different approaches and combinations on them. Please comment!

---

@tailhook: I’ve marked by “❓” my pure speculations that need more research.

Update 1: added notes about using postgres protocol Update 2: mention CDDL

[elprans]

So what is a problem with that

As long as it’s not worse than what we currently have, I’m fine with whatever approach. I’m NOT fine with having to hunt and kill orphaned subprocesses. In my workflow I restart the server very often, so it needs to start quickly and shut down cleanly. Docker and other virtualization/containerization is a non-starter for me due to long startup time.

[elprans]

Also, @tailhook, please remember that we are discussing an implementation of a single RPC interface with a handful of methods, which is not public, has a single client, and no requirement to be backward compatible.