(1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x80\\xF0\\x9F...' for column 'name' at row 1")

Hi, I can’t make it to post emojis to the database when I use my-app.appspot.com but when I run it locally python manage.py runserverwith the same libraries on GAE everything works perfectly and I can post and retrieve emojis .

here is my settings .py

import os
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine'):
    # Running on production App Engine, so use a Google Cloud SQL database.
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'HOST': '/cloudsql/my-app:us-central1:my-app-mysql',
            'NAME': '********',
            'USER': 'root',
            'PASSWORD': '*********',
        }
    }
else:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': '*******',
            'USER': 'root',
            'PASSWORD': '*********',
            'HOST': '**********',
            'PORT': '3306',
            'OPTIONS': {
                'charset': 'utf8mb4',
            }
        }
    }

here is the charset when using cloud shell

mysql> SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';
+--------------------------+--------------------+
| Variable_name            | Value              |
+--------------------------+--------------------+
| character_set_client     | utf8               |
| character_set_connection | utf8               |
| character_set_database   | utf8mb4            |
| character_set_filesystem | binary             |
| character_set_results    | utf8               |
| character_set_server     | utf8mb4            |
| character_set_system     | utf8               |
| collation_connection     | utf8_general_ci    |
| collation_database       | utf8mb4_general_ci |
| collation_server         | utf8mb4_general_ci |
+--------------------------+--------------------+
10 rows in set (0.15 sec)

and here is the charset when I connect using the IP of the database from another client

Variable_name                        Value
character_set_client                utf8
character_set_connection            utf8mb4
character_set_database              utf8mb4
character_set_filesystem            binary
character_set_results               utf8
character_set_server                utf8mb4
character_set_system                utf8
collation_connection                utf8mb4_unicode_ci
collation_database                  utf8mb4_general_ci
collation_server                    utf8mb4_general_ci

I’m I missing something ?!!

how do I make work?

thanks

[thebatlab]

As a final followup, this appears to be working now with no changes to Django code needed!

My database settings now has: 'OPTIONS': {'charset': 'utf8mb4'} And I changed the table I needed emoji support in to the proper character set via this command: ALTER TABLE my_table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

And emojis are working just fine. I put the SQL in a migration file so it’s a part of the project with no need for extra database configuration manually, so it’s all seamless.

Thanks, @myelin , presuming it was you who deployed the fix 😃

[myelin]

Aforementioned internal engineer here 😃

TL;DR: Cloud SQL and App Engine support emojis and 4-byte UTF-8, but ‘OPTIONS’: {‘charset’: ‘utf8mb4’} in your Django settings file will result in “Can’t initialize character set utf8mb4”.

The issue is in the C code that we use to talk to MySQL. It doesn’t support utf8mb4 itself, so when it makes a connection to MySQL, it tells the server to use “UTF8”, which in MySQL means UTF-8 minus the 4-byte characters… and all your emojis get mangled.

However, Python and Cloud SQL both support 4-byte UTF-8 just fine, so if you follow up with a “SET NAMES utf8mb4” command, as @waprin explained above, it’ll tell the Cloud SQL server that it’s safe to send 4-byte UTF-8, and everything will work.

As such, ‘OPTIONS’: {‘charset’: ‘utf8mb4’} doesn’t work on App Engine. To get utf8mb4 in Django:

• Either use ‘OPTIONS’: {‘charset’: ‘utf8’}, or just leave the ‘charset’ option off entirely.

• Edit your base.py and change get_new_connection so that it sends the “SET NAMES utf8mb4” command.

After making these two changes, it should work on both localhost (using dev_appserver.py) and App Engine.

Likewise if you’re using MySQLdb without Django, you need to initialize it like this:

conn = MySQLdb.connect(unix_socket=‘/cloudsql/<your app details>’, user=‘<user>’, passwd=‘<password>’, db=‘<db>’, charset=‘utf8’) conn.execute(“SET NAMES utf8mb4”)

This should all stop being a problem sometime in the next few months (we have a fix, but it’s waiting on a ton of stuff to get deployed and tested before it can go out).

Here’s a Stack Overflow thread with some more details: http://stackoverflow.com/questions/36144026/unable-to-use-utf8mb4-character-set-with-cloudsql-on-appengine-python