Background Task - Kafka Consumer not getting closed on Application Shutdown Ctrl+C

I have added a Kafka Consumer in an API call and when I try to close the application via Ctrl+C - I get below INFO message but the application never shuts down and I can see the PID attached to Kafka on port 9092 with uvicorn threads. The impact on multiple times hitting of Ctrl_C also leads to nothing and my code for shutdown event is never being called.

^CINFO: Shutting down INFO: Waiting for background tasks to complete. (CTRL+C to force quit)

Below is my code -

from fastapi import FastAPI, BackgroundTasks
from kafka import KafkaConsumer
from singleton import my_single_instance

consumer = KafkaConsumer('mitsuki',bootstrap_servers='localhost:9092',group_id = 'series-arc')

def consume_messages():
    for msg in consumer:
        print(msg)

app = FastAPI()

@app.on_event("shutdown")
def close_consumer():
    print('App Down')
    print(consumer.close())

@app.get('/start-consuming-kafka')
async def cool(bt:BackgroundTasks):
    if not my_single_instance.consumer_created:
        bt.add_task(consume_messages)
        my_single_instance.consumer_created = True
    return {'message':'Hello World 123'}

Background tasks are running even after hitting Ctrl+C. I waited for 30 mins to see the Kafka Consumer goes down but it never happened.

Is it the expected behavior or I have to go with some alternative approach.

My end goal is this -

I want to add more tasks to increase the parallelism for same Kafka Consumer as I have multiple partitions. Rule for Kafka Consumer - Number of Partition = Number of Consumer with same Consumer group ID.

So I have 20 partitions for my Kafka Topic and I want to add 5 Background tasks. I will spin up 4 replicas of same application instance in Kubernetes and then all 20 partitions will be covered up by background task consumer thread.

Kindly advise.

[n8sty]

The above still isn’t the correct use of background tasks. Background tasks shouldn’t run for the lifetime of an application, they should be used for tasks that need to happen asynchronously to a HTTP request. Like I said above, try using some other method to create and then safely stop an additional process. One such way is asyncio.loop.run_in_executor, but there are plenty of others (eg: threads, processes, cron). If you want to use an HTTP request to control a longer running background process you can have the HTTP request synchronously set some flag on app.state, and the background process can act according to the value(s) of that flag. I’ll repeat just to emphasize, background tasks are meant to be ephemeral and should not run for the lifetime of a FastAPI application (or Starlette) instance.

I think this issue should be closed since there’s no actual FastAPI specific component to the question.

[tiangolo]

Thanks for the help here everyone! 👏 🙇

Thanks for reporting back and closing the issue 👍