Task.delay() hang forever when rabbitmq is down

Checklist

• I have verified that the issue exists against the master branch of Celery.
• This has already been asked to the discussion group first.
• I have read the relevant section in the contribution guide on reporting bugs.
• I have checked the issues list for similar or identical bug reports.
• I have checked the pull requests list for existing proposed fixes.
• I have checked the commit log to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).
• I have verified that the issue exists against the master branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or result backend.
• I have tried reproducing the issue on more than one version of the message broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries, ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version:

celery report Output:

Steps to Reproduce

Required Dependencies

• Minimal Python Version: 5.1.0
• Minimal Celery Version: 5.1.2
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

celery==5.1.2
django-celery-beat==2.2.1
kombu==5.1.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Start rabbitmq, then execute task.delay() => ok
Stop rabbitmq, then execute task.delay() again => hang 5 mins, then raise exception 
kombu.exceptions.OperationalError: failed to resolve broker hostname
I try with config: CELERY_BROKER_TRANSPORT_OPTIONS = {"max_retries": 3, "interval_start": 0, "interval_step": 0.2, "interval_max": 0.5} but it does not work.

Expected Behavior

Raise exception in some seconds

Actual Behavior

Hang for several minutes.

[thedrow]

This is a bug with our implementation. I’d need to try this myself to reproduce the bug.

[matusvalo]

Technically you can offload this to different thread but you need also to understand all details what does it mean - e.g.

1. you need to handle blocked thread - there is no easy way to kill blocked thread.
2. you need to manage/set daemon thread not to block main process during termination of main process etc.

As mentioned before if you don’t like blocking behaviour you can set retries and timeouts accordingly.