Certin ASCII text lines not sorting properly.

Description

Mostly, sorting works, but when I sort certain sequences of text, they don’t sort in an expected order.

To Reproduce

Steps to reproduce the behavior:

1. Create a new scratch file in Intellij
2. Insert the following four lines:

ThemeData accent properties
The forgetChild method must call super
The RenderEditable needs to be laid out before hit testing
The Route and Navigator Refactoring

1. Select all four lines, and go to Context Menu>String Manipulation>Sort…>Sort Lines
2. Select “Case Insensitive”, “Ascending”, “Natural Order Comparator”
3. Select OK.

The result is this:

The forgetChild method must call super
ThemeData accent properties
The RenderEditable needs to be laid out before hit testing
The Route and Navigator Refactoring

The same result occurs with the “Collator” comparator (with en_US), with case sensitive sorting or not, with “ignore leading whitespace” on or off.

If I pare it down to this set of lines:

The f
Theme
The R
The R

It still happens.

If I remove the last character from each of those lines:

The 
The 
The 
Them

Then it works properly, and this is the order I would expect.

Expected behavior

Depending on the sort order for the space character (and I checked, they’re all actual 0x20 space characters), I would expect either that the line beginning with “ThemeData” to be sorted after (most likely, that’s what the Unix sort command does), or before, but not in the middle of the list.

Environment:

IntelliJ IDEA 2020.1 (Community Edition) Build #IC-201.6668.121, built on April 8, 2020 Runtime version: 11.0.6+8-b765.25 x86_64 VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o macOS 10.15.4 GC: G1 Young Generation, G1 Old Generation Memory: 5120M Cores: 16 Registry: core.pooled.threads=72, undo.documentUndoLimit=1000, vfs.use.nio-based.local.refresh.worker.parallelism=30, undo.globalUndoLimit=100, ide.index.image.max.size=0, ide.balloon.shadow.size=0, caches.indexerThreadsCount=72 Non-Bundled Plugins: Batch Scripts Support, CMD Support, String Manipulation, com.andrewbrookins.wrap_to_column, Git Scope, GrepConsole, com.marlboro.gitbar, com.paperetto.dash, mobi.hsz.idea.gitignore, Dart, io.flutter, Docker, PythonCore

[krasa]

Wow. That sucks.

I started using a new version of https://github.com/paour/natorder but it seems to be broken, and the collator is broken in a different way. In your example it looked the same, but here you can see how it behaves:

no comparator:
The A
The X
The a
The x
Theme
---------
collator:
The a
The A
Theme
The x
The X
---------
NaturalOrderComparator:
The A
The X
The a
The x
Theme
---------
Paour.NaturalOrderComparator:
The A
The X
The a
Theme
The x
---------

I will return to a previous implementation, and add an option for using no comparator.

[krasa]

FYI collator is excluding special characters (that is, common whitespace characters [0009-000D, 0020] and rule syntax characters [0021-002F, 003A-0040, 005B-0060, 007B-007E]). https://docs.oracle.com/javase/6/docs/api/java/text/RuleBasedCollator.html

So I will have to add them all to the rules like so: https://stackoverflow.com/a/16567963/685796