Technical design: content localization and internationalization in 3.x

This ticket is concerned with the localization and internationalization of user-editable content in Apostrophe 3.x sites. For static text, see: https://github.com/apostrophecms/apostrophe/issues/3204

Module Design

In 3.x, content localization will be a standard feature in a forthcoming minor release update of 3.x core. It will not be a separate module as in 2.x. This is because the feature is significantly simpler in 3.x, already “baked into” the database design, and essentially mandatory for website creation in certain countries, i.e. Canada. Thus it meets the standard of “things you must have to have a website” that we try to apply to core apostrophe features. Of course this does not mean you must configure any locales.

Configuration

Configuration of the available locales will take place in a @apostrophecms/i18n module, supporting a nested format as in 2.x:

options: {
  locales: [
    {
      name: 'default',
      label: 'Default',
      private: true,
      children: [
        {
          name: 'en-gb',
          label: 'England'
        },
        {
          name: 'en-us',
          label: 'United States'
        },
        {
          name: 'fr-fr',
          label: 'France'
        }
      ]
    },
  ],
  defaultLocale: 'en-gb'
}

More than one locale at the root level is permitted. The hierarchy is mainly for user interface purposes when selecting locales as an editor, however it is also applied to auto-replication as described below.

The defaultLocale is that which is assumed if the locale cannot be inferred from the request and it does not have to be a root locale.

Private locales

Private locales can never be viewed by logged-out users. To simplify what was complex in 2.x permissions, all users with the contributor role or better can view private locales.

Permissions

In the core open source release of Apostrophe, all users with the contributor role or better can edit across locales, although they may not be able to publish as appropriate to their role. Per-locale permissions will be considered an enterprise feature. Note that the contributor role in 3.x solves most problems that would otherwise arise from this.

Prefixes and hostnames

Locales will continue to support prefix and hostname options as in 2.x. The defaultLocaleByHostname option will also be supported as in 2.x.

The current locale will not be stored in a server-side session variable. When necessary, i.e. because prefixes and hostnames were not set for development and early content creation, it will be kept in a cookie on the browser side. Otherwise it will be inferred from prefixes and hostnames as in 2.x.

Database structure

Unlike Apostrophe 2.x, Apostrophe 3.x has begun its life with a significant portion of this feature already implemented. That is because, in 3.x, a distinction between “draft” and “published” content always exists. This is implemented through the use of distinct MongoDB documents that are related to one another by a shared aposDocId property.

Also unlike 2.x, in Apostrophe 3.x the _id property has a constant aposDocId part, followed by parts that identify the “locale” and the “mode.” Although 3.x does not yet have a UI for localization, it already sets a fixed “locale,” en, on new documents in anticipation of this feature.

Thus in 3.x a typical page has the following properties relevant to localization:

_id: "xyz:en:draft",
aposDocId: "xyz",
aposLocale: "en:draft",
aposMode: "draft"

Note that the _id property includes all of the information present in the other three, however here is some denormalization (duplication) for convenience and for efficient queries.

If a document exists in two language locales, en and fr, then there will be four copies, with the following _id values:

"xyz:en:draft"
"xyz:en:published"
"xyz:fr:draft"
"xyz:fr:published"

Replication

In A3, there is no guarantee that a document exists in multiple locales, except for the following documents which are always replicated across all locales:

• The home page.
• Other parked pages.
• The @apostrophecms/global document, which contains global settings.
• Documents directly related to the above three categories of documents. “Directly related” means that the schema of the document itself or the schema of its own widgets contain a relationship field pointing to those documents.
• Auto-replication to a new locale is based on the nearest parent available in the hierarchy, according to the same algorithm used in 2.x when replicateAcrossLocales: false is set.

The above approach is also available in 2.x if the replicateAcrossLocales: false option is set. In 3.x this is the default and only behavior, to prevent databases from becoming unreasonably large due to mass replication when much of the content is locale specific.

Exceptions

Never-published drafts and archived documents

A document that has never been published will exist in the draft mode but not the published mode. lastPublishedAt will be nullish.

A document that is in the archive will also exist in the draft mode only, and lastPublishedAt will again be nullish.

localized: false

Piece types with the localized: false flag are not localized at all. They have no aposLocale or aposMode. Their aposDocId is identical to their _id. This feature is typically reserved for user accounts and other content types for which localization and drafts do not make sense. Page types cannot use localized: false. Piece types that require translation (localization) should never use localized: false.

autopublish: true

Piece types with the autopublish: true flag do have locales and modes, and both draft and published modes will exist. However, at the user interface level, Apostrophe will guarantee that when such pieces are updated the changes are immediately published. This is useful when localization is appropriate but approvals are not. Images are a good example: approving images is usually a waste of effort because they do not appear on pages unless selected for that purpose.

A strength of autopublish: true is that it leaves open the possibility of changing your mind and setting it to false later for that type. This is not currently possible with localized: false.

Operations across locales

In 2.x, operations across locales included:

• “Replicate,” i.e. the feature which invites you to create a document in “fr” that already exists in “en” at the time you first try to switch to “fr” as an editor;
• “Export,” which attempts to copy only the changes made in the latest commit; and
• “Force Export,” which copies the entire document’s current draft mode content to another locale’s current draft mode.
• “Force Export Related,” which also copies “related” documents such as images when copying a document.

In 3.x, this feature set is simplified based on the actual experiences of several enterprise customers:

• “Replicate” will continue to be available.
• The new “Export” feature, now called “Localize” will be replaced with the behavior of the old “Force Export” feature. That is, when a user “exports” from locale A to locale B, Apostrophe will always copy the entire document’s content to locale B.
• The old “Export” behavior will be removed.
• “Force Export Related” would continue to be available, under a new name, as part of “Localize.” However based on user experiences it will definitely always ask which related types you want to localize, and will probably have a provision to always exclude certain types. For instance, if your page has an explicit relationship to the home page, you still don’t want the home page exported also every time you export that page. Pages in general will always be excluded.

Regarding the removal of the old "Export " feature, this change is being made because it was only possible to use it well if the document’s changes always began in a “main” locale and were then exported gradually to child locales. Any deviation from this pattern makes it impossible for Apostrophe to detect the relevance of the changes to the other locale, and so they were not exported. End users eventually just used the “Force Export” button. So we are taking that simpler approach only in 3.x.

However, 3.x does have a forthcoming “copy and paste” feature for widgets, which should be a very practical alternative when only changes to one widget need to be copied to another locale.

REST API

The localize operation will be available via a distinct REST API endpoint, i.e. POST /:_id/localize. While it can technically be reproduced using only the existing POST and PATCH end points, it is better to have operations which provide a way to track the origin of a document, i.e. which locale it was most recently localized from.

Relationships (joins) across locales

Relationships (formerly called joins in 2.x) had limitations in 2.x. It was not possible to have a relationship from a type exempt from workflow to a type included in workflow. This is because every locale of each document had a distinct, basically random _id.

In 3.x, relationships are always resolved based on the aposDocId, not the entire _id. As a result, there is no problem when resolving relationships. Piece types marked localized: false may have relationships to others, and vice versa.

[boutell]

Localized pages and pieces need to be allowed to have the same or divergent slugs, as these are business requirements for various clients.

In our experience using cookies to hold the locale is a bad idea except for content creators and developers who are testing. In production the distinction should always be made by domain or by prefix, as you mention. But, supporting the cookie strategy is necessary for dev / staging / preprod scenarios, particularly when the goal is not to use prefixes but rather different domains. Devs are not likely to set up subdomains for all of their dev and staging testing of locales.

On Mon, Jul 12, 2021 at 1:24 PM Miro Yovchev @.***> wrote:

I was about to mention that “changing my mind” on locale support/auto publish is a big deal for me. I’m currently in re-iteration which may brings some changes to our “autopublish” pieces to become manually publishable. The opt-in/opt-out at any level of development must be supported by those core features IMHO.

I’m also thinking about the unique checks, mostly referring to the slug here. Currently aposMode shares slug, which means draft and published with same slug co-exist (which makes great sense). When it comes to locale it becomes more hairy.

• Cookie based locale detection would mean shadowed page A in en and fr with same slug, very bad SEO and UX
• Locale detection based on URL part - path or domain - doesn’t suffer this issue because it’s namespaced by request. I can imagine some companies would like to translate their slugs (e.g. /about and /за-нас for en and bg respectively), but I personally (most probably) would stick with paths (hopefully possible) like /en/about and /bg/about and it’s a legit use case. And in this case we most probably don’t need /en to be part of the /about slug, because apos already knows the locale of the page and can prepend it for us, while the slug is shared among those pages.

So, to summarize - in my book it makes sense localized page versions to share unique fields as slug in the same way aposMode versions do but it brings some questions when it comes to client locale detection. Do you have any clear strategy in this direction?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/apostrophecms/apostrophe/issues/3205#issuecomment-878457449, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAH27NUVXJ2DBWL3YPIVR3TXMQOFANCNFSM47RBVLOQ .

–

THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his

[stuartromanek]

I also want to caution against referencing ‘buttons’ that activate or enable features before the design process has gotten underway.