Unable to use create database via API: `The CSRF token is missing.`

Expected results

POST /api/v1/database endpoint should work because it’s in the documentation.

Actual results

When I use the Swagger client embedded into the application (/swagger/v1), the API throws the following exception:

The CSRF token is missing.

Here is a snippet that reproduces the same issue as well:

token_request = requests.get("{}/api/v1/security/csrf_token".format(self.superset_url),
                     headers={'Authorization': 'Bearer {}'.format(self.current_access_token)})
if token_request.status_code != 200:
       raise Exception("Unable to get CSRF token: {}".format(token_request.text))
csrf_token = token_request.json().get('result')

r = requests.post("{}/api/v1/database".format(self.superset_url),
                  json={
                      "allow_csv_upload": False,
                      "allow_ctas": True,
                      "allow_cvas": False,
                      "allow_dml": False,
                      "allow_multi_schema_metadata_fetch": True,
                      "allow_run_async": True,
                      "database_name": database_name,
                      "expose_in_sqllab": True,
                      "sqlalchemy_uri": "presto://.."
                  },
                  headers={'Authorization': 'Bearer {}'.format(self.current_access_token), "X-CSRFToken": csrf_token})

Screenshots

Environment:

superset version: superset version 1.2.0 python version: python --version: python-3.7.10 node.js version: node -v: not relevant

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven’t found one similar.

[buremba]

@youngsol Here is how it works in our case: https://github.com/metriql/metriql-superset/blob/main/metriql2superset/superset.py#L21

[lziosi]

In case this helps anyone else, the problem with the above code is that the CSRF token must be obtained in the same session as where it is used, because there is a session cookie added by the call that creates the CSRF token. This became evident after reading: https://github.com/wtforms/flask-wtf/blob/4b90067908f0ab42bf25f4bc580a4367e09e27dd/src/flask_wtf/csrf.py#L66 However, after fixing that, I got the new error: 400 Bad Request: The referrer header is missing. The fix for this second error must take into account the fact that Referrer header should be misspelled as Referer: https://flask.palletsprojects.com/en/2.1.x/api/#flask.Request.referrer

In the end this code worked fine:

import requests
import json
import os

BASE_URL = os.getenv("BASE_URL")
USERNAME = os.getenv("USERNAME")
PASSWORD = os.getenv('PASSWORD')
SUPERSET_DB_NAME = os.getenv('SUPERSET_DB_NAME')
DB_NAME = os.getenv('DB_NAME')
DB_PORT = os.getenv('DB_PORT')
DB_HOST = os.getenv('DB_HOST')
DB_USER = os.getenv('DB_USER')
DB_PASSWORD = os.getenv('DB_PASSWORD')

# https://levelup.gitconnected.com/solve-the-dreadful-certificate-issues-in-python-requests-module-2020d922c72f
# The CA_BUNDLE variable contains the absolute path of a file that has the root-ca, intermediate-ca and site-cert
# The 3 certs can be exported by Google Chrome as individual Base 64 encoded CER files
# Then manually chained into a single file
# When working in a test environment, you can set CA_BUNDLE to False
CA_BUNDLE=False

def login(base_url, username, password):
    url = base_url + 'api/v1/security/login'
    payload = {'password': password, 'provider': 'ldap', 'refresh': 'true', 'username': username}
    payload_json = json.dumps(payload)
    headers = {'Content-Type': 'application/json'}
    try:
        res = requests.post(url, data=payload_json,
                            verify=CA_BUNDLE, headers=headers)
        res.raise_for_status()
        access_token = res.json()['access_token']
        refresh_token = res.json()['refresh_token']
        return access_token, refresh_token

    except requests.exceptions.RequestException as err:
        print("Request Exception:", err)
    except requests.exceptions.HTTPError as errh:
        print("Http Error:", errh)
    except requests.exceptions.ConnectionError as errc:
        print("Error Connecting:", errc)
    except requests.exceptions.Timeout as errt:
        print("Timeout Error:", errt)


def create_database(base_url, access_token,
                    superset_database_name, database_name, database_port, database_host,
                    database_user, database_password):
    csrf_url = base_url + 'api/v1/security/csrf_token'
    # Construct the Authorization header of the form Bearer access_token
    headers = {'Authorization': 'Bearer ' + access_token}

    url = base_url + 'api/v1/database'

    payload = {
        "database_name": superset_database_name,
        "engine": "postgresql",
        "configuration_method": "sqlalchemy_form",
        "sqlalchemy_uri": "postgresql+psycopg2://{}:{}@{}:{}/{}".\
            format(database_user, database_password, database_host, database_port, database_name)
        }
    payload_json = json.dumps(payload)
    try:
        session = requests.Session()
        session.headers['Authorization'] = 'Bearer ' + access_token
        session.headers['Content-Type'] = 'application/json'
        # Note: it is mandatory that the csrf token be obtained in the same session where it will be used
        csrf_res = session.get(csrf_url, verify=CA_BUNDLE)
        # Note that Referrer must be misspelled as Referer, see:
        # https://flask.palletsprojects.com/en/2.1.x/api/#flask.Request.referrer
        # The Referer[sic] request - header field allows the client to specify,
        # for the server’s benefit, the address (URI) of the resource from which the Request-URI was obtained
        # (the “referrer”, although the header field is misspelled).
        session.headers['Referer']= csrf_url
        session.headers['X-CSRFToken'] = csrf_res.json()['result']
        res = session.post(url, data=payload_json, verify=CA_BUNDLE)
        res.raise_for_status()

    except requests.exceptions.RequestException as err:
        print("Request Exception:", err)
    except requests.exceptions.HTTPError as errh:
        print("Http Error:", errh)
    except requests.exceptions.ConnectionError as errc:
        print("Error Connecting:", errc)
    except requests.exceptions.Timeout as errt:
        print("Timeout Error:", errt)


if __name__ == '__main__':
    access_token, refresh_token = login(BASE_URL, USERNAME, PASSWORD)
    create_database(BASE_URL, access_token, SUPERSET_DB_NAME, DB_NAME, DB_PORT, DB_HOST,
                    DB_USER, DB_PASSWORD)