Prefect Tasks do not seem to cache when invoked in a map
See original GitHub issueHi,
Trying to get the Prefect cache to work for a mapped task. It seems like the tasks are correctly cached, but something about the function signature prevents the cache from correctly being recalled. New to prefect, so definitely may just be operator error…
from prefect import Flow, task
from prefect.schedules import IntervalSchedule
import pendulum
from prefect.engine.cache_validators import all_inputs
import time
# Define flow run frequency
refresh_schedule = IntervalSchedule(
start_date=pendulum.parse("2019-01-01"),
interval=pendulum.duration(minutes=1)
)
@task
def a():
return [1,2]
@task(
cache_for=pendulum.duration(hours=1),
cache_validator=all_inputs)
def b(a):
time.sleep(2.5)
with Flow("test_flow", schedule=refresh_schedule) as test_flow:
z = a()
b.map(a=z)
test_flow.run()
Issue Analytics
- State:
- Created 4 years ago
- Comments:5 (2 by maintainers)
Top Results From Across the Web
Does Prefect support caching on a flow level rather than only ...
My understanding is that the cache_for is used within the execution scope of one flow, so it knows the status/data of the tasks...
Read more >Caching and Persisting Data - Prefect Docs
Prefect provides a few ways to work with cached data between tasks or flows. In-memory caching of task inputs is automatically applied by...
Read more >States | Prefect Docs
States are the "currency" of Prefect. All information about tasks and flows is transmitted via rich State objects. While you don't need to...
Read more >Tasks - Prefect 2 - Coordinating the world's dataflows
This is a task cache key implementation that hashes all inputs to the task using a JSON or cloudpickle serializer. If the task...
Read more >Mapping | Prefect Docs
Even though the user didn't create them explicitly, the children tasks of a mapped task are first-class Prefect tasks. They can do anything...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Just read your PR, was curious to see what changes you needed to make. As part of going through the code, I saw a couple of changes that might improve readability / take advantage of the
any
short-circuiting in python. Please let me know what you think. 😃 https://github.com/PrefectHQ/prefect/pull/1072Great! Looking forward to a fix, love the lib. And thanks for the quick response!! 😄