Make datasource provider dynamic

⚠️ Update: The actually implemented solution can be found here: https://github.com/prisma/prisma/issues/1487#issuecomment-635999202

---

Context

It’s a common use case to use different but very similar databases for different environments (e.g. using SQLite for local development and using Postgres in production). Prisma is especially well suited to support this use case as every Prisma connector is aware of the database’s capabilities and can therefore safely (i.e. at schema validation / client generation time) determine whether two databases are compatible. It does so by reducing the supported database capabilities down to the “overlapping capabilities” (or in mathematical terms: “Lowest common denominator”). For example even though Postgres supporting scalar list fields, when a user wants to use Postgres and SQLite in the same project, the scalar list feature won’t be available i.e. the schema won’t validate and client operations won’t be generated.

In the past we tried to solve this by supporting an optional enabled: env("SOME_VAR") (see e.g. #1142) however this solution had multiple problems such as:

• It’s unintuitive how enabled works and how it’s supposed to be used. I.e. $SOME_VAR is casted to a boolean
• It’s also not clear in which cases the user should configure multiple datasources for multiple environments or just replace environment variables

Proposal

Here is a quick summary of what I’m proposing. More details afterwards.

1. Remove the provider: "postgresql" property from the datasource block as it’s redundant to the information contained in the url property.
2. This enables datasources to become fully dynamic as they can be set via environment variables (and even allows for different datasource types e.g. dynamically switching between SQLite and Postgres).
3. To still enable static schema validation, we’ll introduce a providers: ["postgresql", "sqlite"] property

Details

⚠️ Still WIP

Related issues

• This allows affects (and simplifies) #1480
• Further simplifies #1142
• Enables https://github.com/prisma/prisma-client-js/issues/184

[Errorname]

Hi there, just dropping by to say that I’m really looking for this feature to be implemented for my use cases (SQLite in dev and Postgres in prod & multi-tenancy accross different databases) 🙏

[mavilein]

We would like to move forward with this proposal now. I revisited it and would like to suggest a slightly different proposal for this change.

Summary

1. Extend the Prisma schema syntax to allow the provider property in datasource blocks to also accept an array of values, e.g. provider: ["postgresql", "sqlite"].
2. The active provider for the datasource is then determined by the provided url property.
3. The schema validation must determine the set of capabilities supported by all specified providers. Only capabilities that are supported by all providers can be then used. This requires the provider property to be always specified as opposed to the initial proposal.

Details

1. syntax changes

The current syntax for provider is still supported as is:

datasource db {
  provider = "sqlite"
  url      = env("DATABASE_URL")
}

Additionally we will allow to also provide an array of providers. This is contrary to the original proposal that suggested to add the plural form providers. By only extending the singular provider property makes the change simpler by not dealing with edge cases such as both properties being specified.

datasource db {
  provider = ["sqlite", "postgres"]
  url      = env("DATABASE_URL")
}

2. Process for determining the active provider

If multiple providers are specified the active provider needs to be determined in order to be able to interact with the underlying database. This is done by the following process:

1. Resolve the value for the url property of the datasource. Either it is a static value or an environment variable that needs to be resolved.
2. Loop over the providers specified in the provider property. The resolved URL is passed to each provider in a function called canHandleURL. The provider must return true or false depending on whether it can handle this URL. E.g. the postgres provider would return true for all URLs starting with postgres://.
3. The first provider that returns true becomes the active provider.

This logic implies that connection strings for different providers must not be overlapping. Consider for instance provider: ["cockroach", "postgres"] that uses Postgres and CockroachDB. Cockroach DB is wire compatible with Postgres and hence can deal with Postgres connection URLs like postgres://. Therefore the described logic would always determine the first provider in the array as active. That means that we need to be able to differentiate our provider URLs in such cases by e.g. adding prefixes like cockroach+postgres://.

3. Influence of capabilities on schema validation

If multiple providers are specified the Prisma schema needs to be compatible with all of them. For example when using provider: ["sqlite", "postgres"] the schema needs to be compatible with Postgres and SQLite. The consequence in that case is that scalar list fields, e.g. tags String[], would not be allowed as Postgres supports them but SQLite does not. This example illustrates that we need to determine the set of capabilities supported by all of the specified providers. A capability is only supported by a set of multiple providers if each supports that capability. Following is an overview of the capabilities that the current set of available providers exposes. For each capability it is noted whether a given connector supports it or not. The list of capabilities is going to grow when more providers or features (like custom types) are added.

capabilities	sqlite	postgres	mysql
scalar lists	no	yes	no
enums	no	yes	yes
json	no	yes	yes
relations over non unique fields	no	no	yes

Edge cases

We have to be aware that prisma introspect might yield invalid schemas with this proposal. Consider a schema using provider: ["sqlite", "postgres"]. Depending on the env var the postgres datasource might become active. The underlying database might use a JSON column. The introspection would then add the JSON field. Subsequent schema validations would fail because JSON is not part of the set of supported capabilities for those providers. But this can only happen when starting with an existing database. So that is probably a good trade off.