question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Prefect Tasks do not seem to cache when invoked in a map

See original GitHub issue

Hi,

Trying to get the Prefect cache to work for a mapped task. It seems like the tasks are correctly cached, but something about the function signature prevents the cache from correctly being recalled. New to prefect, so definitely may just be operator error…

from prefect import Flow, task
from prefect.schedules import IntervalSchedule
import pendulum
from prefect.engine.cache_validators import all_inputs
import time

# Define flow run frequency
refresh_schedule = IntervalSchedule(
    start_date=pendulum.parse("2019-01-01"),
    interval=pendulum.duration(minutes=1)
)



@task
def a():
    return [1,2]

@task(
    cache_for=pendulum.duration(hours=1),
    cache_validator=all_inputs)
def b(a):
    time.sleep(2.5)
    
    

with Flow("test_flow", schedule=refresh_schedule) as test_flow:
    z = a()
    b.map(a=z)


test_flow.run()

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
jeremiahpslewiscommented, May 24, 2019

Just read your PR, was curious to see what changes you needed to make. As part of going through the code, I saw a couple of changes that might improve readability / take advantage of the any short-circuiting in python. Please let me know what you think. 😃 https://github.com/PrefectHQ/prefect/pull/1072

1reaction
jeremiahpslewiscommented, May 23, 2019

Great! Looking forward to a fix, love the lib. And thanks for the quick response!! 😄

Read more comments on GitHub >

github_iconTop Results From Across the Web

Does Prefect support caching on a flow level rather than only ...
My understanding is that the cache_for is used within the execution scope of one flow, so it knows the status/data of the tasks...
Read more >
Caching and Persisting Data - Prefect Docs
Prefect provides a few ways to work with cached data between tasks or flows. In-memory caching of task inputs is automatically applied by...
Read more >
States | Prefect Docs
States are the "currency" of Prefect. All information about tasks and flows is transmitted via rich State objects. While you don't need to...
Read more >
Tasks - Prefect 2 - Coordinating the world's dataflows
This is a task cache key implementation that hashes all inputs to the task using a JSON or cloudpickle serializer. If the task...
Read more >
Mapping | Prefect Docs
Even though the user didn't create them explicitly, the children tasks of a mapped task are first-class Prefect tasks. They can do anything...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found