[QUESTION] Using MQTT with FastAPI

First check

• I used the GitHub search to find a similar issue and didn’t find it
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google “How to X in FastAPI” and didn’t find any information.

Description: How can I receive MQTT with fastAPI?

MQTT is used often in IoT. It is much lighter than HTTP and is build on top of TCP. It can also use a WebSocket transport layer.

I can think of the following options and would be interested to hear what you think.

1. Use Flask middleware

Flask has support for MQTT routes, so I could follow the instructions here to implement Flask middleware, https://fastapi.tiangolo.com/advanced/wsgi/#using-wsgimiddleware, however I don’t know if this works in general for any protocol (HTTP, MQTT, WS etc.) of if it only routes HTTP.

2. Use RabbitMQ

RabbitMQ has support for MQTT. But I’m unsure how I would integrate this with FastAPI. It would not be appropriate to use a background task, https://fastapi.tiangolo.com/tutorial/background-tasks/ because these are tasks that fired off after returning a response. Probably I would need to use an asyncio.Queue and call API methods manually when an event arrives.

3. Use MQTT over Websocket

paho-mqtt can use WebSocket transport. Again I’m unsure how to integrate this with FastAPI. Could I use the @websocket decorator. Probably not because that would just be the transport layer, not the protocol layer!

@app.websocket("/sent_data")  #  PUT called by external client to send data here
async def recieved_data_mqtt_over_ws():
    pass

Would it be possible to …?

Eventually something like this might be possible?

@app.mqtt("/sent_data")  #  PUT called by external client to send data here
async def recieved_data_mqtt_over_ws():
    pass

I have not yet started to write code for this part of the project yet, but it’s on the horizon and thought I would try and get some advice sooner rather than later.

[sabuhish]

Hello there, I know the issue is closed, but we have good news, we have worked on it, created an extension for FastApi. FastApi-MQTT is now released with its features. It is a wrapper around gmqtt, we documented every single point. We think are progressed, but if you face any issues pls let me know. Feel free to open issues or submit PR.

fastapi-mqtt: https://github.com/sabuhish/fastapi-mqtt

[sabuhish]

@sabuhish i think it’s a difficult problem, to integrate the event loop with an existing non-async library. I see various attempts at asyncio + MQTT, with varying levels of success. Take a look here https://pypi.org/project/asyncio-mqtt/ and at the list of similar projects at the bottom. Maybe one of them might already have what you need?

@danieljfarrell Sorry for being late. Thank you for your question, it took me back in the time when I first met with MQTT 😃 Last year in the summer, I got a project, data transfer from the web application to device and receiving data from it (STM-32). Paho is the most popular library which is written by Eclipse Foundation and Eclipse has many more libraries for MQTT across many languages. I needed to something in real-time with background scheduler tasks. When we got introduced to MQTT, we dived into many different libraries in python language for django framework. We analyzed paho, hbmqtt, aiomqtt, gmqtt, and others. Among them, we mostly emphasized on paho and tested worked fine. We developed django with paho, celery, and django-channles together. We were succeded. The reason why we chose paho, is because paho was up to date and maintained, had fewer issues in comparison. At that time I did really intend to dive more into asyncio, at the same I followed gmqtt itself to see what is happening. Paho was working quite nicely with more than 100 devices and receiving data in every 3 seconds from each device. Gmqtt was interesting because it had its own implementations and was built on the asyncio library it was in the early stages of development at that time. On the other hand, it was one of the Python libraries which early support MQTT 5.0. From the development team, Elena had talked about the client at Pycon Belarus. It is in the Russian language, unfortunately, I believe translation is possible. PyConBy: Elena Nikolaichik, MQTT with Python. Regarding to benchmarking results for clients, it showed up to the library was almost the same results with paho, and leaves behind others. Benchmarking. This year we took a look again at the libraries checked analyzed all possible solutions. There were around five libraries we took into account. We started with hmbqtt, realized has many issues, and outdated skipped it. Aiomqtt could not integrate paho’s event loop into asyncio. Asyncio-mqtt was not reliable had some critics issues and it was based on an example on paho’s repo. The reason why we were keen on it, because it was built on asyncio libray. Gmqtt was working already with fewer issues and maintained. As Elana said in production the library currently operates with million of payloads transferred between clients.

In regards to your answer, coroutines for full async: Methods that we wrote does not need to internally have to be fully async if operations are not I/O bound and they are simple operations we don’t require to make async. The main library that we use, is developed on top of asyncio. We also simplified it by writing the fastapi-mqtt. The reason for moving two methods into run_in_executer method is because for fastapi users could be easily used inside async-await functions.