Performance issue on outgoing requests

We have an API developed using DRF but recently we switched to FastAPI. We are loving it a lot. This is one of the greatest software that we have ever come across.

Now getting back to the point, our upcoming version of API is built using FastAPI. When we deploy it to a Docker Swarm cluster, regular endpoints (that just return JSON responses without making any computation) perform really great. In our small cluster, we achieved 40K+ requests per second in wrk benchmarks.

But when we call the endpoints where we are making an outgoing request, the performance dramatically decreases. We barely achieve 100 requests per second in our benchmarking tests.

The endpoints where the performance issue is present make outgoing requests and we return data in real-time. We are using a proxy service too so latency issues are there. But the performance difference between the normal endpoints and the ones that get external data and return to the user is really really huge.

We have tried using Gunicorn + Uvicorn worker and Uvicorn alone too. We also tried applying an API gateway like Kong but the performance issue remains there. On the nodes where the API is running CPU usage rises while the API gateway doesn’t go under any load.

Is this a CPU-intensive case or threads are locked when external requests response is awaited? We are using httpx and each API call results in a single external request. Any help will be appreciated greatly.

[STeveShary]

HTTP connection pooling doesn’t work when a rotating proxy is used and the TCP connection needs to be reestablished every time the IP changes.

“Layer of abstraction”: The cause of and solution to all your problems!

[rehmatworks]

@STeveShary Thank you so much for replying. Looks like you have pointed me in the right direction. I am modifying my code and I’ll run some benchmarking tests again.

• I’ll give locust.io a try
• Yes, the endpoints are HTTPS. I am now using HTTP endpoints as well as I am trying pooling
• I am using HTTPX
• I am using Sentry but haven’t looked in the logs for this particular issue yet

Thank you for all your help. I’ll be running benchmarking tests again and I’ll share the results.

Update: Connection pooling does work and I achieved 8K+ requests per second and the results were around ~100 requests per second previously. This is a huge difference. But when a rotating proxy is used, the performance degrades again. HTTP connection pooling doesn’t work when a rotating proxy is used and the TCP connection needs to be reestablished every time the IP changes.