Allow for customized PrefectSecret for AWS/GCP/Azure/etc. Result providers

Current behavior

In order to use the new Results (such as S3Result) it requires that you have your credentials preserved in a PrefectSecret with a specific PrefectSecret name. https://docs.prefect.io/core/concepts/secrets.html#default-secrets

This prevents the ability to customize which credentials you want a particular Flow to run as. In our situation, we have multiple Agents running in multiple environments. The On-Prem Agent is configured to write their Results to an On-Prem MinIO server; while the AWS Agent is configured to write their results to an AWS S3 instance. The networks of each environment is locked down such that one cannot talk to the other.

The credentials between the On-Prem MinIO and AWS S3 are different, but as it is currently, we’re only allowed to configure one of those credentials into the AWS_CREDENTIALS PrefectSecret.

Proposed behavior

Expose the ability to specify which PrefectSecret the credentials are located in. This would be similar to the way the old S3ResultHandler worked with the aws_credentials_secret kwarg. Where you could specify the default of this parameter to AWS_CREDENTIALS if none is provided: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/utilities/aws.py#L37

Example

In this example, we’re introducing the aws_credentials_secret parameter, which would need to be passed on to get_boto_client, where it would attempt to get the prefect.context.secrets:

with Flow(
    name="custom AWS creds",
    result=S3Result(
        bucket='prefect-flow-results',
        aws_credentials_secret='MINO_CREDENTIALS',
        boto3_kwargs=dict(
            region_name='us-east-1',
            endpoint_url='https://minio.local',
        )
    )

[szelenka]

Took a slightly different approach, but it’s functioning.

For Pods that launch in a given cluster, I’m injecting that EnvVar when it spawns through a PodPreset, matching on the label for prefect.io/identifier:

---
apiVersion: redhatcop.redhat.io/v1alpha1
kind: PodPreset
metadata:
  name: prefect-aws-s3
  namespace: podpreset-webhook
spec:
  selector:
    matchExpressions:
      - key: prefect.io/identifier
        operator: Exists
  env:
    - name: PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS
      value: "{\"ACCESS_KEY\": \"value\", \"SECRET_ACCESS_KEY\": \"/value\"}"

This way, when the Flow Run Pod starts up, it’ll get the EnvVar with the local secret context set and it’s easier to rotate the token in the PodPreset manifest. The PodPresets also give us the flexibility to leverage the job_spec_yaml and other K8 labels to refine what context is set to which Flow Runs at run-time.

[cicdw]

Yup that’s correct