PIN 15: Cloud Client Refactor

PIN 15: Cloud Client Refactor

Date: January 10, 2020

Author: Josh Meek

Status

Proposed

Context

Interaction with the Prefect Cloud API should take place in the easiest and clearest way possible. Currently the Cloud API Client interaction looks something like this:

from prefect import Client

client = Client()

client.create_project(...)
client.create_flow_run(...)
client.get_flow_run_info(...)
client.set_flow_run_state(...)

Every query and mutation that would call Cloud’s GraphQL API exists at the top level of the Client. This has a few drawbacks:

• the client.py file has no order and is almost 1200 lines
• looking for a function on the Client yields a large list to sift through
• the feel of the Client is not functional in nature and has no separation of components

The current Client also only contains 15 functions (one of which is deprecated).

Proposal

This PIN suggests refactoring the Client to have a more natural separation of Cloud objects with a clear function-based aspect. The Client would now act similarly to other Cloud-based clients that engineers are used to.

The Client will follow a new client.OBJECT.ACTION pattern and it would look like this:

from prefect import Client
client = Client()

flow = client.flow(id="...", ...) # get a Cloud flow based on `id`
# `flow` is now an object which has actions that can be called

print(flow)                     # outputs serialized flow information

flow.update_project(...)        # update project
flow.set_schedule_active(...)   # set schedule active
flow.run(...)                   # run flow, creates flow run
flow.archive(...)               # archive
flow.delete(...)                # delete
# etc.

Some actions would immediately return usable objects:

flow_run = flow.run(...)        # returns a flow run client object that can be used

flow_run.state                  # current state
flow_run.cancel(...)            # cancel the run
#etc.

Calling object would act as a query, calling action would act as a mutation. Other mutations could be surfaced at the higher level if they are unrelated to Cloud objects (e.g. sendFeedback) but these will be added on an adhoc basis.

In order to perform simple queries with the Client (as it currently stands) users may find themselves having to manually write GraphQL:

client.graphql(query="query { flow (where: { id: { _eq: ... } } { name } }")

This can be take a more natural programmatic approach by using the new form:

client.flow(id="...").get("name")

Consequences

The proposed refactor has some beneficial consequences:

• Client usability would increase
• a large amount of new functionality would be added that currently does not exist
• a lot of the GraphQL written for some CLI commands could be replaced with small Client calls

Of course, there are other actionable consequences as well:

• the old Client methods would need to receive a deprecation warning (to be removed at a later release)
• documentation would need to be updated
• Cloud Runners / CLI / etc. would be updated to use the new paradigm

Actions

In order to realize this PIN the only actions needed are to add the new OBJECT.ACTION functionality and subsequent unit tests. Besides that there are supplemental actions which may happen at the same time or in a later Pull Request:

• add deprecation warnings to old Client methods
• update / add new Client documentation (promote new functionality)
• refactor Cloud Runners / CLI / etc. to use new client methods
• eventually remove old Client methods in a later release

[cicdw]

Yea, and maybe the query method should take arguments for the attributes to query (like half of a GraphQL payload). For example, flow.query("environment{ labels }")

[jlowin]

One recommendation stemming from Chris’ point: run is almost always a verb, so my expectation is that client.run() runs something, rather than creating a “run” noun.

I anticipate two call patterns, one for creating a new object and one for retrieving an existing one. Once created / retrieved, the object can be manipulated via methods.

f = client.create_flow(...) # returns a CloudFlow object (or whatever)
# OR
f = client.get_flow(id) # returns a CloudFlow object (or whatever)
# THEN
f.delete()
f.change_project()



fr = client.create_flow_run(...) # returns a CloudFlowRun object (or whatever)
# OR
fr = client.get_flow_run(...) # returns a CloudFlowRun object (or whatever)
# THEN
fr.get_state()
fr.delete()
fr.run()