Support PostgreSQL's "WITH HOLD" cursor option

Migrated issue, originally created by Scott Milliken (@smilliken)

http://www.postgresql.org/docs/current/static/sql-declare.html

It’s currently not possible to specify the cursor option “WITH HOLD”. This option is useful when you’d like to commit between reads from a cursor.

In particular, in enables one to do this:

cur = session.query(Foo).yield_per(1)
for foo in cur:
    print(foo)
    # maybe do some writes..
    session.commit()

Currently, this causes an error:

Traceback (most recent call last):
  File "withold_test.py", line 9, in <module>
    for q in c:
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 86, in instances
    util.raise_from_cause(err)
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 63, in instances
    fetch = cursor.fetchmany(query._yield_per)
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 997, in fetchmany
    self.cursor, self.context)
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 990, in fetchmany
    l = self.process_rows(self._fetchmany_impl(size))
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 1149, in _fetchmany_impl
    row = self._fetchone_impl()
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 1139, in _fetchone_impl
    self.__buffer_rows()
  File "SQLAlchemy-1.0.11/lib/python2.7/site-packages/sqlalchemy/engine/result.py", line 1126, in __buffer_rows
    self.__rowbuffer = collections.deque(self.cursor.fetchmany(size))
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) named cursor isn't valid anymore

I attached a patch for your review. Example:

cur = session.query(Foo).yield_per(1, with_hold=True)
for foo in cur:
    print(foo)
    # maybe do some writes..
    session.commit()

Happy to submit a pull request if you’re prefer.

As an aside, I’d also like to support the “SCROLL” and “NO SCROLL” options, but to be useful we’d need to add APIs to support the direction clause in FETCH and MOVE statements (http://www.postgresql.org/docs/current/static/plpgsql-cursors.html). Do you have an opinion about how you’d like to implement this, or should I just give it a shot?

---

Attachments: cursor_with_hold.patch

[zzzeek]

I’m voting to close this as a wontfix because if we go to the original use case:

cur = session.query(Foo).yield_per(1)
for foo in cur:
    print(foo)
    # maybe do some writes..
    session.commit()

right there, that is not supportable. when session.commit() is called, the Connection is returned to the pool where it may be closed fully, the transaction is closed, and it is absolutely not appropriate for there to be a cursor still linked to that connection. I see the original patch doesn’t seem to take this into account so not sure how that patch actually was able to work.

the use case of “WITH HOLD” seems to create an extremely different kind of flow that would be very difficult to support as it would likely have lots of non-working cases based on the very strong assumptions within SQLAlchemy that cursors are usable within the scope of a connection being checked out and running in a transaction. IMO this goes into the same area as scrollable cursors, multiple result sets and all that. these are valid use cases but they are very unusual, are almost never requested, and are difficult to generalize around, they remain available by using direct cursor access.

[sqlalchemy-bot]

Changes by Michael Bayer (@zzzeek):

• set milestone to “1.x.xx”