Make Zulip's permissions model much more flexible using groups

Please use this chat.zulip.org thread for substantive feedback on this project/design/etc; because of all the clutter for GitHub links and the like, this page is unlikely to be a useful place for discussion).

---

This is the master issue for an extended project that we’ll be doing to extend Zulip’s permissions model to support using groups for every meaningful permission in the project, with a reasonable sequencing that will allow us to do this migration reasonably efficiently over time. The high level concept to make user groups more powerful and be able to contain each other, turn our existing roles into implicitly maintained user groups, and then change every permissions setting to have the value be a user group (rather than a role). We’ll likely split various chunks of this into independent issues as we work on it.

• We extend the UserGroup data model in Zulip to support allowing a group to contain an array of other groups in addition to the existing array of users functionality. @andersk is taking point on doing this since there are tricky issues involving cycles and providing a transitive closure table to make queries of the form “Is user X in group Y” efficient. This likely involves adding a group_members ManyToManyField to the UserGroup model.
• Write optimized and well-tested methods in the backend for important core queries like is_user_in_group(user_id, group_id, recursive=True) in zerver/lib/user_groups.py.
• We add an is_system_group boolean field on UserGroup. Its main purpose will be that users cannot directly administer such groups (rename, add/remove members, etc. should all be rejected). We may be able to delete this later, but it’ll be important for the transition strategy discussed below. I think structurally this means access_user_group_by_id always reject with these.
• We add is_system_group to API endpoints that send groups.
• We add temporary UI code to avoid displaying is_system_group user groups in UI showing the user groups in a realm, just to avoid user-facing transitions while we do the below.
• Create implicitly maintained UserGroup objects for the existing roles in Zulip, by adding/removing members in a transaction in do_change_user_role. E.g. a moderators system group group with a new is_system_group: True flag would contain all moderators individually, plus the admins and owners system groups, and do_change_user_role would transactionally call the methods to remove the user group their old user group and add them to the new one. Might require some careful investigation of the LDAP sync code path as well, since that uses the @is_realm_admin.setter setter method (i.e. user_profile.is_realm_admin = True, which will need to change to call do_change_user_role). We probably need an everyone system group and possibly a nobody system group is a good idea too (might be preferable to dealing with null being possible everywhere).
• This will likely require a function modify the Full members user group that is called when changing the “time limit for full members” setting as well as in a daily cron job, since that one is a function of time; it would just swap users between the members and full_members user groups.

API design work

We need to make roughly the following API changes, some of which we can start working on quite early.

• Extend the /register and GET /user_groups API endpoints to include the group_members list of IDs in the responses (i.e. which groups are direct members of other groups).
• Add events to notify clients when a group is added/removed from another user group, and write the web app code to handle these.
• Do JS frontend work to be able to provide functions like is_user_in_group, which would walk the data structures and be able to determine at least whether the current user is in a given group. If needed these functions can call the API when called with another user group, but it’s essential for rendering UI to know if the current user can do things without a server round trip.
• Add a GET /user_groups/{id}/members/{id|"me"} endpoint to check if the user ID in question is within the provided user group; this may be a necessary helper for simple clients to decide how to render details.
• Add a GET /user_groups/{id}/members endpoint (for API completeness); it should probably support a recursive option that provides the transitive closure (i.e. just user members).
• Open issues for the mobile and terminal client apps pointing to the new documentation with advice for correctly implementing these.
• Add support for syncing direct membership of user groups from LDAP (#9957) and SAML. @mateuszmandera I think this is probably a self-contained independent project from the rest that we should make sure makes the same release as the main feature.
• We should look at adjusting the permissions for user groups to allow bots to be members and to access them (https://github.com/zulip/zulip/issues/15465).

Permissions UI elements

Once the above is done, we can now start migrating realm permissions to the new model. Structurally, a setting UI widget that presently supports our COMMON_POLICY_TYPES as a dropdown would become a dropdown_list_widget for selecting a user group. Below I have notes for how to convert a single setting; we may want to design a strategy to do many of these together efficiently once we’ve prototyped it for one.

• Write a database migration to replace e.g. create_stream_policy with a create_stream_group ForeignKey pointed at UserGroup. This migration would convert the policy value to the corresponding system group to avoid changing anything for users when we cut it over.
• To do just the above, we don’t require any user-facing UI changes: The UI could still be a dropdown with the web app converting a common_policy_types value into the corresponding system group to send to the server via the API.
• Once we’ve done that migration, we can change the UI to a dropdown_list_widget allowing the user to select any user group.
• We should also offer a “Create group” button that lets the user make a new group in a modal without leaving the “Realm permissions” screen, and some mechanism for navigating to the “user groups” management UI and look at the target stream.
• We do the above for all of the individual organization-level permissions.

We’ll want to do the same for stream-level permissions as well, most importantly “Who can post to this stream”; it’s possible we actually want to do that one as our sample project before migrating any of the realm ones just because it’s unique and very useful to be group-based.

Once we’ve migrated the realm-level settings, we’ll want to add new settings (like https://github.com/zulip/zulip/issues/18491) using this model.

User group management UI

I’ve opened https://github.com/zulip/zulip/issues/19526 as a focused issue to track the rewrite of “User groups” described here.

Being able to use the above features will require changes to our “User groups” management UI: making it a less simple/messy component, providing more settings, and allowing the addition of user groups to other user groups. While we might go with an intermediate state where we just let you add user_group pills to user groups, but I think structurally we will want the user group management UI to be more like the “Streams management” UI, with the list of groups in the left sidebar and tabs for membership, settings, and eventually other things like a “permissions” list showing what the group can do. The dropdowns I expect us to add as part of this process are roughly these:

• Group that can join this group themselves (default: nobody)
• Group that can leave this group from Zulip UI (default: group itself). (Important for LDAP sync, for example)
• Group that can add members to this group (default: group itself)
• Group that can remove members from this group (default: group itself)
• Group that can edit settings for this group (default: group itself)
• Groups that can mention this group (https://github.com/zulip/zulip/issues/19029) (default: everyone)
• Groups that can send private messages (#13656) (default: everyone)
• Whether the group should be advertised as an option during account creation (likely replacing https://github.com/zulip/zulip/issues/13670 along with stream-level features).
• Rendered markdown in group descriptions (like stream descriptions); relatively low priority.
• Custom badges for user groups #19751

(There would likely be separate realm-level settings for which groups can do each of these actions for all groups)

User invitations

We will want to extend the systems for creating and displaying invitations for users to allow specifying a set of initial user groups for a user.

New stream settings

Once the above infrastructure work has happened, there’s a half-dozen permissions settings that we’ll want to create a stream-level version of. A not complete list of things of this form we’ll want is as follows:

• Group that is automatically a member of this stream. (Might want 2 versions: One for “added on account creation” and another for “cannot unsubscribe”)
• Group that can send messages to the stream (the one stream permission setting we have with roles today).
• Group that can see the stream’s existence when not subscribed (effectively “public stream” is “all members”, private stream is “administrator role” today).
• Users who can join the stream when not subscribed (effectively “public stream” is “all members”, “private stream” is nobody today; this would allow things like making a “support team” group that can join any stream associated with that team freely, which results in a decent number of unnecessary muted stream subscriptions to provide occasional access)
• Group that can edit topics in the stream.
• Group that can delete messages they themselves sent in this stream.
• Group that can delete all messages sent in this stream.
• Group that can resolve topics in the stream even if they can’t edit topics normally.
• Groups that can create new topics (#19642).
• Groups that can unsubscribe from the stream (#19661).
• Etc. – structurally everything originally imagined for https://github.com/zulip/zulip/issues/3783 belongs here.

The theory is that all of these will be straightforward to add once we have the infrastructure in place; I’ll wait to open dedicated issues until we have the infrastructure/API support in place to actually add one of these.

[timabbott]

Just adding a note that I think this is really important and we plan to prioritize pushing forward as though it’s a release goal, but there’s enough here that we’re unlikely to finish everything in time for the 6.0 major release, so I’m not adding the release goal label at this time.

[alya]

@yringler it’s currently covered under the “etc.” bullet; we don’t need a separate issue.

Etc. – structurally everything originally imagined for https://github.com/zulip/zulip/issues/3783 belongs here.

Thanks for checking!