Running pipelines via the UI hangs and 500's are encountered.

Original user report: https://meltano.slack.com/archives/C01TCRBBJD7/p1655302068277449?thread_ts=1655235602.029959&cid=C01TCRBBJD7

The pipeline shows in the UI for me but when I try to run the pipeline, it's stuck and I get the below error
2022-06-15T14:04:31.879525Z [error    ] Exception on /api/v1/orchestrations/jobs/state [POST] 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/meltano/api/controllers/orchestrations.py", line 207, in job_state
    state_ids = poll_payload["state_ids"]
KeyError: 'state_ids'
2022-06-15T14:04:31.879843Z [info     ] Error: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

This exception and state_ids being missing isn’t the direct error, its a side effect, for some reason the client isn’t passing along a valid POST object as its missing state_ids (side note this should probably return a Bad Request or Pre-con failed, not a 5xx). We’ll need to double-check and see why state_ids is actually not being included.

Impact

This issue blocks users from running pipelines in the Meltano UI web GUI.

Workaround

As a workaround, users would need to run EL workloads using the CLI (meltano schedule run) or in Airflow.

[alexmarple]

@edgarrmondragon i’m always so suspicious when it’s that easy. ok i’ll make that correction and test it.

[alexmarple]

https://github.com/meltano/meltano/pull/6216

Here’s a PR but the issue persists after swapping out instances of jobId with stateId. Will continue to investigate in the morning.