QHub Extension Mechanism

Description

The are many projects that we would like to integrate into QHub and additionally other projects that we may not have the time/resources to support in QHub. We need an extension mechanism to integrate projects that a loosely connected to Qhub. For me prefect and clearml are good examples of this. I would also like to see if this could be adopted within Qhub for some parts.

Suggestion

Features that the extension mechanism should have.

• allows us to expose the service at https://<custom-domain>/<custom-prefix>/
• optionally setup oauth2 client for service
• provide an “overrides” section so that we can easily control the helm chart
• helm chart information

Example

extensions:
  - name: Prefect
    chart: <path-to-chart-url>
    version: <chart-version>
    domain: <optional domain name>
    prefix: <prefix-url>
    oauth2_client: true/false
    overrides:
       ...

Lets start a discussion about this. We need to use approach as well for a few components.

[costrouc]

Let me talk in detail a bit more about these pieces.

OAuth2 Provisioning

We should provision an OAuth2 client per service with keycloak this is possible an encouraged.

Should take as inputs:

• callback_url which should be roughly calculated from domain/prefix.

This should return:

• endpoint
• user url information
• client id
• secret

This is complex … not sure how to expose to the helm chart. Possibly have some way to reference these variables within the overrides? Ideas welcome.

Ingress Provisioning

Typically helm charts already have ingress hooks. How do we make sure that we are integrating this properly. I’d prefer if we hook into the existing chart for this. Since traefik will do the work for provisioning certs. Need to think about this more.

Monitoring Integration

We should apply annotations that allow prometheus monitoring to easily connect.

[costrouc]

From the development of the stages in PR #1003 I would like to propose a general “extension” mechanism that makes all parts of Qhub an extension. Take for example the following from deploy.py

def provision_01_terraform_state(stage_outputs, config):
     ...
     elif config['provider'] == 'gcp':
        stage_outputs[directory] = terraform.deploy(
            terraform_import=True,
            directory=os.path.join(directory, config['provider']),
            input_vars={
                'name': config['project_name'],
                'namespace': config['namespace'],
                'region': config['google_cloud_platform']['region']
            },
            terraform_objects=[
                QHubGCPProvider(config),
            ],
            state_imports=[(
                "module.terraform-state.module.gcs.google_storage_bucket.static-site",
                f"{config['project_name']}-{config['namespace']}-terraform-state",
            )])
    ...     

Later when we look at stages/02-infrastructure/gcp we see a terraform module /directory that is called by this function. There is a lot being done in the terraform.deploy(...) function but there are important sections seen here.

1. input_vars how do values in qhub_config.yaml and outputs from prior stages stage_outputs map to imput variables to the terraform module.
2. state_imports these are ids/attrs that can be used such that terraform can “import” these resources if they already exist. This allows for qhub to be more stateless. We have used this in the past to import terraform-state in the stages/01-terraform-state.
3. terraform_objects is a hook to allow for “dynamic” terraform resources to be rendered to the stage. This should only be a function of the qhub_config.yaml and NOT previous stage_outputs so that the render command can be separated from the deploy command.

Right now this imformation is written redundantly in qhub/deploy.py and qhub/destroy.py instead I suggest that this information be included in the terraform module itself and qhub will search for it. My suggestion is config.py.

The structure I am suggesting. A minimal QHub extension would be as follows.

config.py
*.tf

config.py would be an optional file and if nothing is provided we would assume sane defaults:

• input_vars would be set to {"qhub_config": config, "stage_outputs": state_outputs}
• state_imports would be empty []
• terraform_outputs would be empty []

The optional functions within config.py would be

def input_vars(qhub_config, stage_outputs):
     return {
         'name': qhub_config['project_name']
    }

def state_imports(qhub_config, stage_outputs):
     return [(
                "module.terraform-state.module.spaces.digitalocean_spaces_bucket.main",
                f"{qhub_config['digital_ocean']['region']},{qhub_config['project_name']}-{qhub_config['namespace']}-terraform-state",
    )]

def terraform_ouputs(qhub_config):
    from qhub.render.terraform import QHubGCPProvider
    return [
           QHubGCPProvider(config),
    ]

terraform_outputs would be called within the render stage. While state_imports and input_vars would be called in the deploy stage. The advantage here is that we could fully adopt this model for all of our components making us fully use our extension mechanism within qhub.

Questions

How would you allow additional extensions easily in QHub with this model?

extensions:
   - path: github.com/path/to/repo
     config:
        a: "value"
   - path: ./my/custom/qhub/extension
     config:
         another: 1
         value: [1, 2, 3]

We would add an extensions key to qhub-config.yaml that takes a list of paths. These paths can be directories and github repo paths as well. After the core qhub stages have been run it will iterate through each extension using the config.py in the extension to determine the mapping of qhub-config.yaml to terraform input variables. Extensions can also assume that certain providers are already configured via environment variables e.g. keycloak and kubernetes.

The main reason I suggest this approach is that it allows for easy abstraction and making qhub-enterprise just another extension to qhub.