[QUESTION] aiohttp throws RuntimeError

Description

I’m trying to build an async web scraper using aiohttp library. I have an aiohttp.ClientSession instance in a module that I import and share among various scrapers. A typical scraper is like this:

# app/http.py
import aiohttp
session = aiohttp.ClientSession()


# app/services.py
from app.http import session

async def translate(phrase: str) -> typing.List[str]:
    # ...
    res = await session.get(...)
    html = await res.text()
    # ...
    return translations

My path functions look like this:

# app/__init__.py

@app.get('/api/translation', response_model=typing.List[str])
async def get_translations(
        phrase=Query(...),
):
    return await translate(phrase)

When I send a GET request to /api/translation?phrase=bewegen, I get the following error:

# ...
INFO: ('127.0.0.1', 1549) - "GET /api/translation HTTP/1.1" 500
ERROR: Exception in ASGI application

  # ...

  File "D:\Development\python\langu\venv\lib\site-packages\aiohttp\helpers.py", line 568, in __enter__
    raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task

On the other hand, if I manually run the scraper, it runs just fine:

if __name__ == '__main__':
    async def main():
        return await translate('komm schon')

    results = asyncio.run(main())
    print(results)

Creating a new session for each request also works:

@app.get('/test')
async def test():
    session = aiohttp.ClientSession()
    async with session:
        res = await session.get('https://httpbin.org/get')
    return await res.json()

According to this page:

• https://stackoverflow.com/questions/52232177/runtimeerror-timeout-context-manager-should-be-used-inside-a-task

I need to share the same loop between aiohttp.ClientSession and FastAPI/uvicorn instances, but I’m not sure if that’s the best solution or if it’s possible at all.

Additional context Python version: Python 3.7.3

Installed packages

aiodns==2.0.0
aiofiles==0.4.0
aiohttp==3.5.4
async-timeout==3.0.1
attrs==19.1.0
beautifulsoup4==4.7.1
cffi==1.12.3
chardet==3.0.4
Click==7.0
fastapi==0.29.0
h11==0.8.1
idna==2.8
multidict==4.5.2
pycares==3.0.0
pycparser==2.19
pydantic==0.26
soupsieve==1.9.1
starlette==0.12.0
uvicorn==0.7.2
websockets==7.0
yarl==1.3.0

[ghost]

I think the issue is that you instantiate your session outside of an async def function. And when you instantiate it no loop is running.

If you want to keep the session at module level you can have something like this (I didn’t test it):

# app/http.py
import aiohttp

_session = None

async def get_session():
   global _session
   if _session is None:
       _session =  aiohttp.ClientSession()
   return _session

# app/services.py
from app.http import get_session

async def translate(phrase: str) -> typing.List[str]:
   # ...
   session = await get_session()
   res = await session.get(...)
   html = await res.text()
   # ...
   return translations

[zzztimbo]

@tiangolo @abdusco

I’m trying to implement the code suggested by @romeo1m.

 loop = asyncio.get_event_loop()
 loop.run_until_complete(foo())

gives me the following error: RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-0_0'.

asyncio.run(foo())

succeeds on the first call, but on subsequent calls gives me the follow error: RuntimeError: Event loop is closed

can someone point me to some code that shows how to implement a shared aiohttp.ClientSession() in fastapi?