Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

sqlite-utils index-foreign-keys fails due to pre-existing index

See original GitHub issue

While running the command:

sqlite-utils index-foreign-keys $SQLITE_DIR/pudl.sqlite

I got the following error:

Traceback (most recent call last):
  File "/home/zane/miniconda3/envs/pudl-dev/bin/sqlite-utils", line 8, in <module>
    sys.exit(cli())
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/cli.py", line 454, in index_foreign_keys
    db.index_foreign_keys()
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py", line 902, in index_foreign_keys
    table.create_index([fk.column])
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py", line 1563, in create_index
    self.db.execute(sql)
  File "/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py", line 421, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: index idx_generators_eia860_report_date already exists

This DB was created with the foreign key constraint PRAGMA enabled and a bunch of column-level CHECK constraints. Is this an expected behavior? Should one not try to index foreign keys if FK constraints are already being enforced within the DB?

I’m also noticing that the size of the DB after FK indexes have been added went from 483MB to 835MB, which seems like a much bigger jump than when I’ve done this previously.

Software versions…

sqlite-utils 3.17.1
sqlite 3.36.0
SQLAlchemy 1.4.26 (used to create the DB)

Issue Analytics

State:
Created 2 years ago
Comments:11 (11 by maintainers)

Top GitHub Comments

1reaction

simonwcommented, Nov 14, 2021

OK, this should fix it.

0reactions

simonwcommented, Nov 14, 2021

create_index(..., find_unique_name=) is good. Default to false. index_foreign_keys can set it to true.

Top Results From Across the Web

Changelog - sqlite-utils

This means that index-foreign-keys will work even if one of the indexes it ... Previously such columns would be rejected with an error....

sqlite-utils(1) - Debian Manpages

This library and command-line utility helps create SQLite databases from an existing collection of data. Most of the functionality is available as either...

sqlite-utils - PyPI

CLI tool and Python utility functions for manipulating SQLite databases. ... Extract columns into separate tables to better normalize your existing data.

Weeknotes: Datasette, sqlite-utils, Datasette Desktop

A neat thing about Pluggy is that you can add new arguments to existing plugin hooks without breaking plugins that already use them....

SQL

you have already developed database applications using the SQL language, ... What people often forget is that “always index foreign keys” is a...