Illegal mix of collations (utf8mb4_unicode_ci,IMPLICIT) and (utf8mb4_general_ci,IMPLICIT) in 0075_populate_latest_revision_and_revision_object_str

Issue Summary

When migrating a django project (on a fresh db) that uses mariadb:10 with --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci, the migration 0075_populate_latest_revision_and_revision_object_str of wagtailcore fails with the following traceback. I unfortunately failed to find out which two were the offending fields. I also couldn’t find any db settings to make the migration pass anyway

File "/app/.venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute                                                                                             
    return self.cursor.execute(sql, params)                                                                                                                                                   
  File "/app/.venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 75, in execute                                                                                         
    return self.cursor.execute(query, args)                                                                                                                                                   
  File "/app/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute                                                                                                      
    res = self._query(query)                                                                                                                                                                  
  File "/app/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query                                                                                                       
    db.query(q)                                                                                                                                                                               
  File "/app/.venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query                                                                                                    
    _mysql.connection.query(self, query)                                                                                                                                                      
MySQLdb.OperationalError: (1267, "Illegal mix of collations (utf8mb4_unicode_ci,IMPLICIT) and (utf8mb4_general_ci,IMPLICIT) for operation '='")                                               

The above exception was the direct cause of the following exception:
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 27, in <module>
    main()
  File "./manage.py", line 23, in main
    execute_from_command_line(sys.argv)
  File "/app/.venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/app/.venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/app/.venv/lib/python3.8/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/app/.venv/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/app/.venv/lib/python3.8/site-packages/django/core/management/base.py", line 96, in wrapped
    res = handle_func(*args, **kwargs)
  File "/app/.venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 349, in handle
    post_migrate_state = executor.migrate(
  File "/app/.venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/app/.venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/app/.venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/app/.venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 125, in apply
    operation.database_forwards(
  File "/app/.venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/app/.venv/lib/python3.8/site-packages/wagtail/migrations/0075_populate_latest_revision_and_revision_object_str.py", line 18, in populate_latest_revision
    Page.objects.all().update(latest_revision_id=latest_revision_id)
  File "/app/.venv/lib/python3.8/site-packages/django/db/models/query.py", line 1190, in update
    rows = query.get_compiler(self.db).execute_sql(CURSOR)
  File "/app/.venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/app/.venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql
    cursor.execute(sql, params)
  File "/app/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 562, in execute
    return real_execute(self, sql, params)
  File "/app/.venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute                                                                                             
    return self._execute_with_wrappers(
  File "/app/.venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers                                                                              
    return executor(sql, params, many, context)
  File "/app/.venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute                                                                                            
    return self.cursor.execute(sql, params)
  File "/app/.venv/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/app/.venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute                                                                                            
    return self.cursor.execute(sql, params)
  File "/app/.venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 75, in execute                                                                                        
    return self.cursor.execute(query, args)
  File "/app/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/app/.venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/app/.venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1267, "Illegal mix of collations (utf8mb4_unicode_ci,IMPLICIT) and (utf8mb4_general_ci,IMPLICIT) for operation '='")                                      

Steps to Reproduce

1. Start a mariadb:10 docker container with --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
2. git clone https://github.com/meine-stadt-transparent/meine-stadt-transparent
3. cd meine-stadt-transparent
4. git checkout 7af8a4ee514c8d19fc7b54ca41ae372c8fe8eede
5. poetry install # tested with poetry==1.2.2
6. ./manage.py migrate wagtailcore

Any other relevant information. For example, why do you consider this a bug and what did you expect to happen instead?

• I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: Unfortunately, I failed to reproduce this with a fresh project. I think it needs the previous migrations to cause the bug. It happens with a completely fresh database when only migrating wagtailcore though

Technical details

• Python version: Run python --version. python:3.8 docker container
• Django version: Look in your requirements.txt, or run pip show django | grep Version. 4.1.2 (in poetry.lock)
• Wagtail version: Look at the bottom of the Settings menu in the Wagtail admin, or run pip show wagtail | grep Version:. 4.0.2 (in poetry.lock)
• Browser version: n/a

[vsalvino]

I don’t think any action is necessary in Wagtail. Wagtail is using legal Django/SQL syntax.

However it would be worth adding a note to the docs, if and when mysqlclient accepts the change. I am planning to add such a note to both Wagtail and Django docs whenever the patch is released in mysqlclient.

It may be useful to temporarily add a note to the Wagtail 4 upgrade considerations, as this behavior will affect a large portion of MySQL users.

[vsalvino]

I don’t think setting db_collation would help. In this case, we have two tables/columns which are both using the exact same collation. However the connection is using an entire different and incorrect collation. The CAST statement uses the connection’s collation, not the table’s/column’s. This is a documented behavior of MySQL and MariaDB. The Django ORM does not add custom collation or charset args to the CAST statement, even if you set db_collation on the fields.

The only workaround is to not use a Cast statement in Django, or to use my patch to mysqlclient which correctly sets the collation of the connection. If you have the rare setup where your tables/columns are intentionally using different collations, then there would be no possible way to use the Cast statement in Django between those columns.

I have not been able to get a response from the maintainer of mysqlclient.