Infinite job submissions when submitted jobs' status is error/cancelled/unknown status

Information

• Qiskit Terra version: lastest
• Python version:
• Operating system:

What is the current behavior?

The problem is in the run_qobj() function in qiskit/utils/run_circuits.py. When a submitted job to a non-simulator backend is in a JobStatus.ERROR or JobStatus.CANCELLED status, the job is submitted again. This happens infinitely many times if the job is continually canceled or in an error state.

if job_status == JobStatus.CANCELLED:
    logger.warning("FAILURE: Job id: %s is cancelled. Re-submit the Qobj.", job_id)
elif job_status == JobStatus.ERROR:
    logger.warning(
        "FAILURE: Job id: %s encounters the error. "
        "Error is : %s. Re-submit the Qobj.",
        job_id,
        job.error_message(),
    )
else:
    logging.warning(
        "FAILURE: Job id: %s. Unknown status: %s. " "Re-submit the Qobj.",
        job_id,
        job_status,
    )
job, job_id = _safe_submit_qobj(
    qobj, backend, backend_options, noise_config, skip_qobj_validation
)

This is especially dangerous when a the job was submitted successfully, but reading the status was wrong (due to a bug in the job class or at the backend side), possibly causing infinite job submissions each costing the user money.

Steps to reproduce the problem

Execute a job that always returns the status JobStatus.CANCELLED.

What is the expected behavior?

The job submission should stop after a finite amount of retries.

Suggested solutions

Do not use while True loops in the code. Instead use a finite, preferably small, amount of retries.

[TheGupta2012]

I was thinking that too but Mathew Treinish and Steve Wood said that it would be okay to halt and notify that the job set execution has failed. The basis was that execution through QuantumInstance is intended for execution of quantum algorithms and failing of even one job in the job set would mean a failed execution of the algorithm altogether.

Also, if the reviewers want I could still add some functionality in other commits but the current PR resolves the issue by raising a QiskitError if any check for execution or status retrieval fails.

[TheGupta2012]

Hey @shakal, is there any other advice for the PR I opened? I’m not sure if I should ask the reviewers specifically.