LocalModule Storage Option

Hi all,

This is a cross posting of discussion #3304. This implementation will allow for easier sharing of python workflow libraries that are installed via pip/conda in any type of environment, including LocalEnvironment.

Proposed behavior

LocalModule will serve just like Local with stored_as_script=True, where instead of a path=‘/path/to/my/flows/exampleflow.py’, you could use module_path=‘from myrepo.flows import exampleflow’ or ‘myrepo.flows.exampleflow’. Thus the flow is accessible in any environment where that module import works, not just the environments that share the same path structure.

In an example workflow library, workflows are separated into scripts and can thus be imported and used. One can easily do the following psuedo code…

from myrepo.workflows import exampleflow
exampleflow.flow.register(...)

or alternatively… While I import the entire script above, it may be easier to simply import the single instance (the flow variable).

from myrepo.workflows.exampleflow import flow
flow.register(...)

where the exampleflow.py looks like…

import prefect
from prefect import task, Flow

@task
def hello_task():
    logger = prefect.context.get("logger")
    logger.info("Hello, Cloud!")

flow = Flow("hello-flow", tasks=[hello_task])

from prefect.environments.storage import LocalModule
flow.storage = LocalModule(module_path=__name__) # __name__ will give 'myrepo.workflows.exampleflow'

The LocalModule Class

Just to start with implementation, LocalModule should be pretty simple to implement in terms of serialization and dynamic loading.

The serialization of a module is simply saving the path and version:

info = {'module': 'myrepo.workflows.exampleflow',
'version': 'v0.0.1',
'class_str': 'flow', # (optional) this is what I made my variable name for my Flow instance above
}

And dynamic loading can be done with…

module_path = info['module'] # grab the string from the serialized info
class_str = info['class_str'] # grab the name of the variable for Flow instance
# (optional) some version check -- maybe warn or raise error if there is a difference...?
# and now use the string to import
import importlib
module = importlib.import_module(module_path)
# and if the Flow is saved as a variable instance named 'flow', we can grab it using
flow = getattr(module, class_str)

[jacksund]

Sure thing! I’ll use the Local storage class as a starting point and give this an attempt.

[jacksund]

I won’t be able to continue this until next week (specifically until the 28th), but feel free to add comments to my changed files! If you’d like me to just add more comments to the code or clear up meaning in the doc strings, I can do that as well.