Non english characters are replaced with question marks sometimes (mysql)

We found out today, that some non english charcacter doesn’t get saved properly to the database. Some of them get replaced by a questionmark, some by two, sometimes it works fine. This only happens if you use a mysql backend

I was able to figure out an example that always fails. Set up an etherpad lite instance with mysql, copy paste this string привет into a pad, restart etherpad lite and reload the pad. You will see that your string got replaced by questionmarks

This results in ueberDB in this sql command

REPLACE INTO `store` VALUES ('pad:test3', '{\"atext\":{\"text\":\"Welcome to Etherpad Lite!\\n\\nThis pad text is syncприветhronized as you type, приветso that everyone viewing this page sees the same text. This allows you to collaborate seamlessly on documents!\\n\\nEtherpad Lite on Github: http://j.mp/ep-lite\\n\\n\",\"attribs\":\"|2+r+l*0+6+m*0+6|4+4e\"},\"pool\":{\"numToAttrib\":{\"0\":[\"author\",\"a.L7J3iR4yRjzJJJVc\"]},\"nextNum\":1},\"head\":2,\"chatHead\":-1,\"publicStatus\":false,\"passwordHash\":null}'),('pad:test3:revs:2', '{\"changeset\":\"Z:6i>6|2=r=l*0+6$привет\",\"meta\":{\"author\":\"a.L7J3iR4yRjzJJJVc\",\"timestamp\":1330620479659}}');

If I paste this sql command into phpmyadmin, the special chars gets replaced with question marks too. It looks like the node mysql driver doesn’t escape this chars properly, or the database scheme can not work with such chars

Any help to solve that problem is appreciated

[jhollinger]

In mysql, each table has its own character set, and it usually defaults to latin1. This is probably causing the issue. (The charset can also be set for each connection. They should match.) The charset should probably be UTF-8. I think something like this should fix the db:

ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

[marcelklehr]

Not taking into account the possible additional magic that encodeURIComponent performs, this might be a solution.