Rendition constraint exceeds maximum MySQL index key length

Issue Summary

The change to image rendition model constraints in the 1.8 release causes problems with MySQL index key lengths. Specifically, adding this constraint:

unique_together = (
    ('image', 'filter_spec', 'focal_point_key'),
)

creates this MySQL constraint command:

ALTER TABLE `renditionstable` ADD CONSTRAINT `indexname_uniq` UNIQUE (`image_id`, `filter_spec`, `focal_point_key`);

on a table like this:

mysql> DESCRIBE renditionstable;
+-----------------+--------------+------+-----+---------+----------------+
| Field           | Type         | Null | Key | Default | Extra          |
+-----------------+--------------+------+-----+---------+----------------+
| image_id        | int(11)      | NO   | MUL | NULL    |                |
| filter_spec     | varchar(255) | NO   | MUL | NULL    |                |
| focal_point_key | varchar(255) | NO   |     | NULL    |                |
| ...             |              |      |     |         |                |
+-----------------+--------------+------+-----+---------+----------------+

This fails on both a new database and an existing one with

ERROR 1071 (42000): Specified key was too long; max key length is 1000 bytes

This constraint is trying to create a UNIQUE index on three columns, two of which are type varchar(255). When using the utf8 encoding, these columns use 3 bytes per character (for uf8mb4, it’s 4 bytes per).

When creating an index in MySQL, “key values are formed by concatenating the values of the given columns.” This means that the index width would have to be (11) + (255 * 3) + (255 *3), which exceeds the maximum key length of 1000 bytes (the limit for the MyISAM storage engine). In Wagtail 1.7 and earlier because the constraint was on filter_id instead, the index size would have been (11) + (11) + (255 *3), which fit okay under the limit.

Using the MySQL InnoDB storage engine (instead of MyISAM) this limit is 767 bytes by default although it can be increased to 3072 using --innodb-large-prefix. This would likely satisfy this requirement, but it would be nice to be able to use MyISAM as well somehow.

One possible fix/workaround for this would have been to define AbstractRendition.filter_spec and .focal_point_key as something like max_length=64 versus the current 255. Assuming those columns could live with the narrower widths, this change would get this constraint under both the 767 and 1000 byte limits. On the other hand, I can see why it might be good to standardize/future-proof these fields, and it might be hard to retrofit this change given that 1.8 has already shipped.

Beyond that idea I’m not sure a good way to workaround this, short of moving to InnoDB (not preferred). Another idea might be to avoid use of AbstractRendition at all, and just duplicate needed functionality in a custom rendition class, with narrower columns as suggested above. Any thoughts?

(This migration was also mentioned by @bkfox in #2953, but as that issue is more generally about index lengths, I thought it better to create a more specific one here. #2925 is also somewhat related.)

Steps to Reproduce

1. Run a local MySQL instance using the MyISAM storage engine (SET default_storage_engine=MYISAM).
2. Create a Django project pointing to your MySQL, and then run migrations against Wagtail 1.8.
3. You should see this error when wagtailimages.0016_deprecate_rendition_filter_relation is run.

Technical details

• Python version: 2.7.11
• Django version: 1.8.15
• Wagtail version: 1.8

[gasman]

Have updated the PRs to “step down” the max_length as proposed - seems to do the job.

Tested with the following configuration:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'utf8test',
        'USER': 'root',
        'PASSWORD': '...',
        'OPTIONS': {
            'charset': 'utf8',
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'; SET default_storage_engine=MYISAM; SET NAMES utf8;",
        }
    }
}

and a database created with mysql -uroot -p -e "CREATE DATABASE utf8test CHARACTER SET utf8 COLLATE utf8_general_ci". Performed the following tests:

1. Starting from a fresh install on 1.7, upgrade to the fix/mysql-index-length-1.8 branch, run ./manage.py migrate. Upgrade to the fix/mysql-index-length-1.9 branch, run ./manage.py migrate
2. Starting from a fresh install on fix/mysql-index-length-1.8, upgrade to the fix/mysql-index-length-1.9 branch, run ./manage.py migrate
3. Bring up a fresh install on fix/mysql-index-length-1.9
4. Starting from a fresh install on 1.7, upgrade to the fix/mysql-index-length-1.9 branch, run ./manage.py migrate

Also tested the upgrade from 1.8 -> fix/mysql-index-length-1.8 -> fix/mysql-index-length-1.9 and 1.8 -> fix/mysql-index-length-1.9, on a Postgres instance.

[gasman]

@chosak Thanks for the confirmation!

Now merged into 1.8.x and master.