[SIP-54] Proposal to improve export/import of objects

[SIP-54] Proposal to improve export/import of objects

Motivation

Superset provides functionality for exporting and importing objects. It can be used to export:

• Datasets: metadata describing the dataset definition and semantic layer can be exported to YAML (example);
• Databases: metadata describing the database connection, as well as metadata describing contained datasets, can be exported to YAML (example);
• Dashboards: metadata describing the dashboard, contained charts and associated datasets can be exported to a single JSON file (example).

For import, currently only dashboards can be imported. The import assumes that the databases are identical in both instances, having the same tables. In theory databases can have different names, but as @mistercrunch said “the code in this area is not in a great shape”, with bugs (https://github.com/apache/incubator-superset/issues/11028) and unexpected side-effects (https://github.com/apache/incubator-superset/issues/10479). Additionally, since users need to specify a database when importing, dashboards with charts from multiple databases are currently not supported.

The main motivation for this SIP is to fix and improve the import/export functionality, proposing a well-defined format for serializing and deserializing collections of Superset objects. This includes databases, datasources, charts and dashboards. The format specification will be versioned, providing backwards compatibility.

Having a well-defined interchange format not only will prevent bugs but also allow us to build new functionality on top of it:

• We can potentially include the data in the download (assuming it’s smaller than a given threshold). This would allow users to, eg, download a dashboard from a blog post or repository, load it into their instance, and explore it.
• It would provide a foundation for file-based configuration of dashboards. Other tools like Grafana allow users to define dashboards in files and store them under version control, and having a well-defined format will make it easier to implement a similar storage mechanism in Superset.
• It would be easier to build tools that programmatically generate dashboards and charts.

Proposed Change

The implementation of this SIP will build upon work in progress introducing UUIDs to import/export mixins (https://github.com/apache/incubator-superset/pull/7829). Adding UUIDs to the Superset models will help prevent conflicts when importing/exporting, as well as allowing the import of dashboards that are powered from different databases.

This SIP introduces a specification for serializing Superset objects, focusing on backwards compatibility and readability. Objects (databases, datasources, charts and dashboards) will be serialized to YAML, one file per object. Files will be grouped into directories according their type, and zipped together into a single file.

For example, exporting the “Unicode Test” dashboard would result in a ZIP file with the following structure:

# dashboard_unicode-test_20200923T173845.zip
databases/examples.yaml
datasets/examples/unicode_test.yaml
charts/unicode_cloud.yaml
dashboards/unicode_test.yaml

Each object will be versioned, and use UUIDs for relationships:

# databases/examples.yaml
version: 1.0.0
id: e834a2be-439f-4cdf-bd55-0a1a32df7ceb
title: Examples
...

# datasets/examples/unicode_test.yaml
version: 1.0.0
id: 9e963850-e556-4c09-bc95-79d1c0c98724
database_id: e834a2be-439f-4cdf-bd55-0a1a32df7ceb
table_name: unicode_test
...

When importing, Superset will unzip the file and check if it can import the provided version. Exports in the current format (single YAML or JSON file) will continue being supported, even though they don’t declare a version.

On import, objects will be matched against existing objects based on UUID. In the case of a match, users will be presented with options to upsert (merge into existing object, with attributes on the file having precedence), overwrite or ignore. If there’s no match, the object will be created if possible.

For databases, sensitive information such as passwords and the secure_extra field would be omitted, requiring the administrator to manually provide them on import.

New or Changed Public Interfaces

The implementation of this SIP would deprecate the /import_dashboards endpoint, substituting it for a more generic /api/v1/import endpoint for any object type. The initial scope of this SIP is to replace the existing “Import Dashboards” menu entry under “Settings” in the header, but in the future we could have additional navigation paths, eg, having an “Import Database” option close to the button to add a new database.

The current import page is a CRUD UI generated by FAB, and it will be replaced with a React-based UI.

New dependencies

No new dependencies will be introduced.

Migration Plan and Compatibility

Implementing UUIDs on the ImportMixin would require adding a new column to the models to store the UUID, as well as populating existing objects with a new value. To simplify the export import/export process, the migration script would also add UUIDs to the position_json field in dashboards, pointing to the object UUID instead of its primary key.

Rejected Alternatives

Other serialization formats were considered, but since we want files to be human-readable YAML stood out, especially because it doesn’t introduce any new dependencies.

[CoryChaplin]

@betodealmeida would you see backing up / powering charts & dashboards from the github as a continuation of this effort ?

I think we can come up with nice abstractions when developing the import, so that they can be reused when developing a mechanism for filesystem-based configuration. But I’m not sure on the timeline for that, it’s not a high priority AFAIK.

I think there’s a nice use case for “official dashboards” that can be versioned, changes reviewed and deployed. And even more so for people running multiple instances of Superset, starting by dev/staging/prod or multi-team/multi-customer versions.

[bkyryliuk]

@betodealmeida would you see backing up / powering charts & dashboards from the github as a continuation of this effort ?