A workflow engine

IBMQ needs a workflow engine to manage the execution and retrieval of jobs.

This would be a higher level construct than Job, which is the unit of interaction with the backends (from the API’s point of view). But jobs have arbitrary limitations that should be masked from the high-level user. For example each backend places its own limit on the number of experiments contained within a job. So currently you have to manually batch many circuits into some jobs, manually store the jobs into some datastructure, and manually retrieve/post-process the results. If one job fails, it can be difficult to recover the total results without manually resubmitting that job and replacing the entry in the database.

The workflow engine automates this process. It can be called a JobManager. At a high level, the user specifies which “tasks” they would like to run. Even if a task has a million circuits, the workflow engine will take care of splitting them, farming them out to the backends, reporting on the progress to the user, and joining the reuslts into a single Result for the task.

Simple pseudo-code (up for discussion):

from qiskit.providers.IBMQ import JobManager
jm = JobManager()

# build a million circuits
from qiskit.circuit.random import random_circuit
circs = []
for _ in range(1000000):
    circs.append(random_circuit())

# farm out the jobs.. 
from qiskit import IBMQ
p = IBMQ.load_account()
device = p.backends.tenerife()
jm.run(circs, backend=device)

# inquire about the status of my runs
jm.status()  # says how many jobs, and the status of each

# join result of (potentially) multiple jobs
res = jm.result()

[jyu00]

Implemented via the PRs mentioned in this issue.

[ajavadia]

Hi @jyu00

1- Yes I think partial Jobs/Results should be supported. Terra recently started to support partial results for cases when only a subset of circuits succeed in a job (https://github.com/Qiskit/qiskit-terra/pull/3217). This would be analogous to that. It would be nice if the jobmanager.report() could say something like circuits 200-300 failed to submit. Reason: no credit left. I think currently it says something about job failure, but doesn’t tell the user which circuits in the original huge list this corresponds to.

2- Hm, yeah I see. But that sort of defeats the purpose here because the point of this is to hide “API details” from a user who doesn’t care how jobs are split up, what the limits are, etc. They just want to execute 1000 circuits and look at the result.

So how about: when I do jobmanager.result(), maybe it could return a JobManagerResult class which is agnostic to how many jobs it took to create it? This would basically be a wrapper around the list of results you currently return, but would support methods like jm_result.get_counts(circuit). This would not have a single job_id associated with it (could have a list of job_id maybe).

3- I’m good with whatever you decide here.