Remove option breaks main slugify functionality

Reproduction script:

const slugify = require('slugify');

const fileName = decodeURIComponent('a%CC%8Aa%CC%88o%CC%88-123'); // åäö-123
const fileName2 = decodeURIComponent('%0A%C3%A5%C3%A4%C3%B6-123'); // åäö-123

const withoutRemove = slugify(fileName);

const withRemove = slugify(fileName, {
  remove: /[*+~.()'"!:@]/g,
});

const withRemoveAlternativeEncoding = slugify(fileName2, {
  remove: /[*+~.()'"!:@]/g,
});

console.log(withoutRemove);
console.log(withRemove);
console.log(withRemoveAlternativeEncoding);

Result

aao-123
åäö-123 # Broken
aao-123

I can’t reproduce this when typing the “åäö” characters on my keyboard, however if I paste them from my console or a browser URL then this behaviour can be reproduced. I suspect there is a problem with different code points similar to this: https://github.com/ipython/ipython/issues/522

[khromov]

👋 @simov

I understand the problem. It appears this is called a “Decomposed character sequence” and is an alternate way of writing certain characters.

There is a good visual example here: https://superpixel.ch/bugreport/safari-diacritic/

I saw some discussion regarding this on SO, pointing out Chrome / Safari as the culprit.

It feels like the current solution will work just fine in a majority of cases, since it separates the main character (which won’t be filtered) from all the diacritics and such (that will be filtered). But it would be nice if it could somehow work with remove.

I noticed there’s a String.normalize function that seems to do what we want. Maybe one option could be to apply it before any other code, but it would be hard to assess the implication of it on existing codebases.

Example:

Before normalization (å character only):

console.log(Array.from(decodeURIComponent('a%CC%8A'))
  .map((v) => v.codePointAt(0).toString(16))
  .map((hex) => "\\u" + "0000".substring(0, 4 - hex.length) + hex));

> Array [ "\\u0061", "\\u030a" ] // a,  ̊

After normalization:

console.log(Array.from(decodeURIComponent('a%CC%8A').normalize())
  .map((v) => v.codePointAt(0).toString(16))
  .map((hex) => "\\u" + "0000".substring(0, 4 - hex.length) + hex));

> Array [ "\\u00e5" ] // å - great success!

[simov]

Published in v1.5.0 https://github.com/simov/slugify/commit/2df63f965aaf63844a51d3c3fab7704d83a31877