peewee.OperationalError: too many SQL variables PLUS

Hi there:

I’m using peewee to insert files into SQLite, I meet this error: peewee.OperationalError: too many SQL variables . I tried Google, some said it’s because the SQLITE_MAX_VARIABLE_NUMBER.

So i searched and using this code to get the SQLITE_MAX_VARIABLE_NUMBER.

def max_sql_variables():
	"""Get the maximum number of arguments allowed in a query by the current
	sqlite3 implementation. Based on `this question
	`_

	Returns
	-------
	int
		inferred SQLITE_MAX_VARIABLE_NUMBER
	"""
	import sqlite3
	db = sqlite3.connect(':memory:')
	cur = db.cursor()
	cur.execute('CREATE TABLE t (test)')
	low, high = 0, 100000
	while (high - 1) > low: 
		guess = (high + low) // 2
		query = 'INSERT INTO t VALUES ' + ','.join(['(?)' for _ in range(guess)])
		args = [str(i) for i in range(guess)]
		try:
			cur.execute(query, args)
		except sqlite3.OperationalError as e:
			if "too many SQL variables" in str(e):
				high = guess
			else:
				raise
		else:
			low = guess
	cur.close()
	db.close()
	return low

SQLITE_MAX_VARIABLE_NUMBER = max_sql_variables()

And i get the value is 999. Then I tried the following steps:

... code omit...
for _fileList in _list:
	print "Insert %s Records" % len( _fileList )
	FileLite.fromFileList( _fileList )

(1) I split my file list to 400, But it still said peewee.OperationalError: too many SQL variables (2) I split my files list to 100,it still said peewee.OperationalError: too many SQL variables (3) I split my files list into 80, It’s OK! (4) I increase the value to 84, It seems OK and insert some data, but when the loops goes on, the peewee.OperationalError: too many SQL variables throws, HOW COULD THIS HAPPEN?

[vadimkantorov]

It would be great if peewee could automatically batch up inserts upon the insert_many call (peewee is in a better position to count accurately the number of used SQL variables)

[coleifer]

As far as I can tell this is not a Peewee issue, but running into limits imposed by SQLite.