question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

task_run_name's kwargs referring to task inputs shadowed by Prefect's context instead

See original GitHub issue

Description

Recently, a task_run_name parameter was added to the task constructors, and documentation was written on how to dynamically set task Run Names dynamically based on inputs.

At runtime, Prefect automatically populates kwargs with raw inputs and also prefect context: https://github.com/PrefectHQ/prefect/blob/1bbefb71fa15cfc6bf6b6d766fe7591dafaa4717/src/prefect/engine/cloud/task_runner.py#L346

This leads to a hard-to-catch bug if you happen to use a task input variable under the same name as one of the keys in the Prefect Context. Common variable names like config or parameters or even map_index, if used in task input, would be shadowed by the correspondent context value and lead to crashes or unexpected task run names.

Expected Behavior

An exception could be raised if a duplicated name is detected… But most likely the existing behaviour just needs to be clearly explained in the documentation.

Reproduction

from prefect import task, Flow

@task
def generate_configs():
    return [{"value": 1}, {"value": 2}, {"value": 3}]


@task(task_run_name=lambda **kwargs: f"{kwargs['config']['value']}")
def parse_configs(config):
    print(config["value"])


with Flow("Task Run Names Bug") as flow:
    parse_configs.map(generate_configs())

Environment

Reproducible in any environment with Prefect version 0.13.10+, any Cloud deployment method.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
jcristcommented, Nov 3, 2020

IMO a library shouldn’t issue a warning (note you want warning.warn(...), not raise here) that can’t be disabled by the user making some change to their code. The only way for a user to fix this is to change the argument names to their function, which seems drastic given that most users probably won’t need to access config when generating a task name (most users only want the input arguments to the task). Forcing the user to change their argument names just to silence a warning that doesn’t affect them seems like bad UX to me.

I think we should stick with the simple solution, and maybe add a small note in the Task docstring for the task_run_name argument mentioning how conflicts are handled.

0reactions
mashun4ekcommented, Nov 9, 2020

PR #3622 closes the issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

The Anatomy of a Prefect Task
Everything you wanted to know about Prefect Tasks and more. Contents. The Basics; Running Tasks; Task Inputs and Outputs; Adding Tasks to Flows;...
Read more >
celery.app.task — Celery 5.2.7 documentation
Task implementation: request context and the task base class. class celery.app.task. ... kwargs (Dict) – Original keyword arguments for the task.
Read more >
How to inspect mapped tasks' inputs from reduce tasks in Prefect
Mapping over a list preserves the order. This is a property you can use to link inputs with the errors. Check the code...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found