chunk iterator support for results

It would be helpful to be able to ‘automatically’ iterate through fetchmany, similar to pd.read_sql(chunksize) or as suggested here: http://code.activestate.com/recipes/137270-use-generators-for-fetching-large-db-record-sets/

So for example instead of writing the following in this example:

    with engine.connect() as conn:
        result = conn.execution_options(stream_results=True).execute(
            Customer.__table__.select().limit(n)
        )
        while True:
            chunk = result.fetchmany(10000)
            if not chunk:
                break
            for row in chunk:
                row["id"], row["name"], row["description"]

you could write something like:

    with engine.connect() as conn:
        result = conn.execution_options(stream_results=True, arraysize=10000).execute(
            Customer.__table__.select().limit(n)
        )
   for chunk in result:
          for row in chunk:
                row["id"], row["name"], row["description"]

I’m using arraysize to be in accordance with pep249 dbapi cursor.arraysize but perhaps chunksize might be better

[zzzeek]

the activestate recipe is wrong. with the vast majority of DBAPIs, it doesn’t matter if you call fetchall() or fetchmany(), the rows are already fetched into memory fully after execute() is called. so I think this feature is kind of misleading since fetchmany() is already misleading. The option that should be considered here is stream_results: https://docs.sqlalchemy.org/en/13/core/connections.html?highlight=stream_results#sqlalchemy.engine.Connection.execution_options.params.stream_results only works with three DBAPis right now - without it, there are no memory savings to fetchmany().

[sumau]

Very clear 😃