Unable to template target / location when mapped tasks are involved and deploying on a dask cluster

Description

I am using a static dask cluster deployment setup - (following this deployment recipe)

I am on prefect v0.11.4 and using a k8s agent and I am trying to template the location or target for a mapped task.

Please see two example code snippets below:

Using target templating

template = 'prefect-testing/{task_name}/{filename}_{map_index}.prefect'

s3_result = S3Result(
    bucket=os.environ["AWS_BUCKET"],
)

@task()
def gen_list():
    return [x for x in range(10)]


@task(
    target=template
)
def add(x, y):
    return x + y


@task(
    target=template
)
def multiply(x, y):
    return x * y

    with Flow(
        flow_name,
        environment=RemoteDaskEnvironment(address="tcp://dask-scheduler:8786"),
        storage=Docker(
            registry_url=registry_url,
            image_name=image_name,
            image_tag=image_tag,
            python_dependencies=[
                'boto3==1.13.14',
            ]
        ),
        result=s3_result
    ) as flow:
        x = gen_list()
        y = gen_list()
        added = add.map(x, y)
        multiply.map(added, added)

I get the following error

Unexpected error while reading from S3: KeyError('filename')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/s3_result.py", line 166, in exists
    self.client.get_object(Bucket=self.bucket, Key=location.format(**kwargs))
KeyError: 'filename'

Using Result.location templating

template = 'prefect-testing/{task_name}/{filename}_{map_index}.prefect'

s3_result = S3Result(
    bucket=os.environ["AWS_BUCKET"],
    location=template
)

@task
def gen_list():
    return [x for x in range(10)]


@task
def add(x, y):
    return x + y


@task
def multiply(x, y):
    return x * y

    with Flow(
        flow_name,
        environment=RemoteDaskEnvironment(address="tcp://dask-scheduler:8786"),
        storage=Docker(
            registry_url=registry_url,
            image_name=image_name,
            image_tag=image_tag,
            python_dependencies=[
                'boto3==1.13.14',
            ]
        ),
        result=s3_result
    ) as flow:
        x = gen_list()
        y = gen_list()
        added = add.map(x, y)
        multiply.map(added, added)

I get the following error after multiply is mapped - i.e. flow runs fine until it reaches multiply[0]

3 June 2020,04:40:57 	prefect.S3Result	DEBUG	Starting to download result from prefect-testing/{task_name}/{filename}_{map_index}.prefect...
3 June 2020,04:40:57 	prefect.S3Result	ERROR	Unexpected error while reading from result handler: ClientError('An error occurred (404) when calling the HeadObject operation: Not Found')

it fails to format the location before reading it because it says Starting to download result from prefect-testing/{task_name}/{filename}_{map_index}.prefect - for some reason the location formatting is not invoked for the second mapped task

[marwan116]

Hi @cicdw - just tested it out - Result.location templating is working fine now! Closing this issue as both templating approaches are working. Thanks!

[marwan116]

For the second issue, can you confirm that your dask workers have the appropriate configuration so they can authenticate with S3?

Yes - I don’t think authentication is the issue here because all previous tasks to multiply - i.e. gen_list and add’s results are being saved to S3 just fine