problem with search in russian with capital letters

Search in Russian language is case-sensitive (unlike English), but search string normalize query to lower-case. So, if i want to find word like Привет in Привет, Мир!, i’ll get an empty result. Reproduced on fresh install self-hosted, ubuntu 20.04, last zulip version in https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz - 3.2, Install zulip with this man: https://zulip.readthedocs.io/en/latest/production/install.html Enabling multi language search: https://zulip.readthedocs.io/en/latest/subsystems/full-text-search.html

[kou]

Ah, we also need to specify the normalizer in the highlight function we used: https://pgroonga.github.io/reference/functions/pgroonga-match-positions-character.html

It doesn’t have a feature to customize normalizer. I’ll improve PGroonga and release a new version. Then we can add support for highlighting Russian texts.

[kou]

I’ve implemented it and released PGroonga 2.2.8.

The following change will fix it:

diff --git a/zerver/views/message_fetch.py b/zerver/views/message_fetch.py
index 2b6a0a6894..9a2d519aef 100644
--- a/zerver/views/message_fetch.py
+++ b/zerver/views/message_fetch.py
@@ -471,10 +471,13 @@ class NarrowBuilder:
         query_extract_keywords = func.pgroonga_query_extract_keywords
         operand_escaped = func.escape_html(operand)
         keywords = query_extract_keywords(operand_escaped)
+        index_name = "zerver_message_search_pgroonga"
         query = query.column(match_positions_character(column("rendered_content", Text),
-                                                       keywords).label("content_matches"))
+                                                       keywords,
+                                                       index_name).label("content_matches"))
         query = query.column(match_positions_character(func.escape_html(topic_column_sa()),
-                                                       keywords).label("topic_matches"))
+                                                       keywords,
+                                                       index_name).label("topic_matches"))
         condition = column("search_pgroonga").op("&@~")(operand_escaped)
         return query.where(maybe_negate(condition))

But we also need to run ALTER EXTENSION pgroonga UPDATE; as superuser after we upgrade installed PGroonga. But we don’t have the feature in our Puppet configuration yet…