Stream all results for arbitrary SQL and canned queries

I think that there is a difficulty with canned queries.

When I want to stream all results of a canned query TwoDays I get only first 1.000 records.

Example: http://myserver/history_sample/two_days.csv?_stream=on

returns only first 1.000 records.

If I do the same with the whole database i.e. http://myserver/history_sample/database.csv?_stream=on

I get correctly all records.

Any ideas?

[simonw]

It’s interesting to note WHY the time limit works against this so well.

The time limit as-implemented looks like this:

https://github.com/simonw/datasette/blob/5f9f567acbc58c9fcd88af440e68034510fb5d2b/datasette/utils/__init__.py#L181-L201

The key here is conn.set_progress_handler(handler, n) - which specifies that the handler function should be called every n SQLite operations.

The handler function then checks to see if too much time has transpired and conditionally cancels the query.

This also doubles up as a “maximum number of operations” guard, which is what’s happening when you attempt to fetch an infinite number of rows from an infinite table.

That limit code could even be extended to say “exit the query after either 5s or 50,000,000 operations”.

I don’t think that’s necessary though.

To be honest I’m having trouble with the idea of dropping max_returned_rows mainly because what Datasette does (allow arbitrary untrusted SQL queries) is dangerous, so I’ve designed in multiple redundant defence-in-depth mechanisms right from the start.

[simonw]

I should research how much overhead creating a new connection costs - it may be that an easy way to solve this is to create A dedicated connection for the query and then close that connection at the end.