Feature Request: cancel scheduling of failing query after X retries

I have a Redshift query that seems to result in an overflow error:

[2015-10-16 17:37:27,957: ERROR/MainProcess] Task redash.tasks.execute_query[f36ec80c-e9f2-4dec-bfb0-3d44a0920f5c] raised unexpected: Exception('Integer data overflow (multiplication)\nDETAIL:  \n  -----------------------------------------------\n  error:  Integer data overflow (multiplication)\n  code:      1058\n  context:   \n  query:     2268469\n  location:  numeric.hpp:325\n  process:   query1_64 [pid=26484]\n  -----------------------------------------------\n\n',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/redash/redash.0.8.0.stanhu/redash/tasks.py", line 24, in __call__
    return super(BaseTask, self).__call__(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/redash/redash.0.8.0.stanhu/redash/tasks.py", line 312, in execute_query
    raise Exception(error)
Exception: Integer data overflow (multiplication)
DETAIL:  
  -----------------------------------------------
  error:  Integer data overflow (multiplication)
  code:      1058
  context:   
  query:     2268469
  location:  numeric.hpp:325
  process:   query1_64 [pid=26484]
  -----------------------------------------------

It appears that this exception causes celery to retry the same query again and again, causing our database load to be sustained at a high load.

There ought to be a way in redash to alert for these kinds of exceptions and log failed attempts. This one was buried in a supervisor log, and I had to look in the Redshift query log to see that it was trying over and over.

Also, this seems to suggest there are cases where retries are not desirable. I can see the usefulness of retrying due to a DB connection issue, but exceptions caused by a bad query should not warrant a retry.

[Ralnoc]

This is good to see getting added to the current versions. Is there details on how to clean up a scheduled query that is in this state? (i.e. if we have a query in this endless retry state, how do we stop it?)

[arikfr]

With #1584 and #1774 I think it’s safe to close this one.