Finesse database backends

Follow-ons from #366

• Change the interfaces so that fetchone and fetchall return Record instances.
• Cleanup the transaction and connection state management.
• ~Add SQLite support.~ - See #283 instead

[tomchristie]

You might find this helpful: https://github.com/encode/starlette/pull/390

1. Yes. I’ve already done that in the docs now: https://www.starlette.io/database/
2. No. Individual connections lifetimes are handled for you automatically. (Short version: They’ll autocommit as soon as possible unless they’re inside a transaction.)

You will want to connect/disconnect the database connection pool on startup/shutdown.

@app.on_event("startup")
async def startup():
    await database.connect()

@app.on_event("shutdown")
async def shutdown():
    await database.disconnect()

1. It’d be like this…

@app.route("/notes", methods=["GET"])
async def list_notes(request):
    query = notes.select()
    results = await database.fetch_all(query)
    return JSONResponse(content)

1. Nope, you don’t call .connect() and .disconnect(). Connections are managed for you, and you can wrap up operations in a transaction if you want to isolate a set of operations. Contection/Transaction management is dealt with using task-local storage, so databases is always able to find out whatever the current connection is.

2. “On the other side, I want to have docs of how to use “standard” SQLAlchemy ORM models in FastAPI” - You can’t use SQLAlchemy ORM with any async web framework. Or at least, you can, but you need to run the operations within a threadpool. If you’re using Starlette and you just write an endpoint as a standard function (non-async) then it’ll do that for you and you can use SQLAlchemy from there just fine. You’re then in a standard threaded context.

3. Yes exactly. I think you’d need to explicitly start and close an SQLAlchemy “session” once you’re in the running-in-a-threadpool block.

4. An SQLAlchemy middleware would be really good, and would be something we’d certainly be able to recommend in Starlette. Thinking about it a bit more, the session setup/close might have to run inside the same thread as the endpoint, since I’m pretty sure SQLAlchemy ORM must use thread-local storage. If that’s the case then you might need to start out implementing it as a decorator. (Perhaps have a look if there’s anything like that already, that wraps a function up inside a single SQLAlchemy session) - It’d be really good to also be able to recommend a good integration for working with SQLAlchemy/sync-views, as an alternative to the low-level databases/async-views option. (Or indeed GINO or whatever else)

[jpic]

Thank you so much for the discussion, also to @succerror for the example that got me started. I have a couple of tricks to share:

• do this in an ASGI Middleware and you also get WebSocket endpoint support
• use contextvars to magically abstract the SQLA session away and use models more like Django

Here is an example, using SQLA 1.4b1 which supports async:

import contextvars    
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine    

engine = create_async_engine('postgresql://localhost/starmvc', echo=True)
async_session = contextvars.ContextVar('async_session')    

starlette = Starlette(**yourstuff)

async def app(scope, receive, send):                                                           
    async with AsyncSession(engine) as session:                                                
        token = async_session.set(session)                                                     
        try:                                                                                   
            async with session.begin():                                                        
                response = await starlette(scope, receive, send)                               
            await session.commit()                                                             
        except:                                                                                
            await session.rollback()                                                           
            raise                                                                              
        finally:                                                                               
            await session.close()                                                              
            async_session.reset(token)                                                         
        return response     

Note that you don’t need the async_session contextvars, it’s only useful if you want to abstract away the SQLA session object like so:

class AsyncSessionDescriptor:    
    def __get__(self, obj, objtype):      
        session = async_session.get(False)                                                     
        assert session, 'starmvc.orm.async_session was not set'
        # TODO: do something better than the above line ^^
        return session                                                                         
                                                                                               
                                                                                               
class QueryDescriptor:                                                                         
    def __get__(self, obj, objtype):                                                           
        return objtype._session.sync_session.query(objtype)                                    
                                                                                               
                                                                                               
class ModelBase:                                                                               
    _session = AsyncSessionDescriptor()                                                        
    objects = QueryDescriptor()                                                                
                                                                                               
    @classmethod                                                                               
    def create(cls, **kwargs):                                                                 
        obj = cls(**kwargs)                                                                    
        obj.save()                                                                             
        return obj                                                                             
                                                                                               
    def delete(self):                                                                          
        self._session.delete(self)

    def save(self):
        self._session.add(self)


Model = declarative_base(cls=ModelBase)

I’m sorry for not presenting code that is as polished as yours, but I hope this can still be useful to others who are trying to get started.

This has not been tested in production. However, here is an additional trick you can take for free: pytest support

    @pytest.fixture(autouse=True)
    async def orm_async_session():
        async with AsyncSession(engine) as session:
            async_session.set(session)
            yield session
            await session.rollback()

    @pytest.fixture(scope='session', autouse=True)    
    async def migrate():    
        async with create_your_engine().begin() as conn:    
            await conn.run_sync(YourModel.metadata.create_all)    
        yield    
        async with engine.begin() as conn:    
            await conn.run_sync(YourModel.metadata.drop_all)    

    @pytest.fixture(scope='session')
    def event_loop(request):
        """Fix for the following error:

        ScopeMismatch: You tried to access the 'function' scoped fixture
        'event_loop' with a 'session' scoped request object, involved factories
        """
        loop = asyncio.get_event_loop_policy().new_event_loop()
        yield loop
        loop.close()


    # monkey patch the event loop yay
    from pytest_asyncio import plugin
    plugin.event_loop = event_loop

However, this will require a patched version of pytest-asyncio which maintains the contextvars for a test. But this will allow the User model to work outside of the requests, which allows you to test like with Django:

@pytest.mark.asyncio
async def test_user_create(client):
    response = await client.post(
        '/user/create',
        data=dict(
            email='[e-mail]',
        ),
    )
    assert response.status_code == 200
    assert User.objects.where(User.email == '[e-mail]').count()