[SIP-71] API to pass metadata over to Celery workers

[SIP-71] Proposal for API to pass metadata over to Celery workers

Motivation

There are several types of metadata required to run analytical queries. Among them are user-impersonating authentication tokens and granular source information (e.g. dashboard or chart IDs). Unfortunately, this “global” information is not available on the Celery workers since they run remotely in a different Flask context. This proposal suggests to add Superset config API to pass required “global” data over so user code on Celery worker is able to utilize it.

Proposed Change

Add CELERY_FLASK_METADATA_EXTRACTOR config function allowing to save all necessary data from flask.g, flask.request and flask.context on the request side and CELERY_FLASK_METADATA_INITIALIZER config function restoring the data on the Celery worker side.

New or Changed Public Interfaces

Superset config will just have new CELERY_FLASK_METADATA_EXTRACTOR and CELERY_FLASK_METADATA_INITIALIZER config functions.

New dependencies

None

Migration Plan and Compatibility

The API is completely backward compatible

Rejected Alternatives

• I thought of just having CELERY_PASSOVER_METADATA config tuple with names of fields of flask.g to save/restore, but it would not allow to populate flask.g with custom fields (mb derived from flask.request or flask.session), and adding a new functionality to populate flask.g in the first place would become as complex as the proposal while being way less intuitive.

[craig-rueda]

LGTM as long as the new config options default to a no-op, which I’m sure you were planning anyways.

[cccs-RyanS]

Hey @rusackas would love to revive this one and the associated PR. Should I create a new issue for it / have this one reopened?