User restrictions

From an admin perspective it may sometimes be useful to support restrictions on user resource usage. A few examples:

• Maximum number of workers per user
• Maximum resources per user
• Maximum number of clusters per user

We should probably provide a common metrics, but will likely also need a generic hook for operations an admin may want to restrict. This may look like:

def can_start_cluster(user):
    """Check if a user can start a cluster.
    
    Returns
    -------
    allowed : bool
        True if the user can start a cluster, False otherwise
    msg : str or None
        A user-facing message explaining why the operation is currently forbidden.
    """
    pass

def can_start_worker(user, cluster):
    """Check if a user can start a worker.
    
    Returns
    -------
    allowed : bool
        True if the user can start a worker, False otherwise
    msg : str or None
        A user-facing message explaining why the operation is currently forbidden.
    """
    pass

[ericdill]

I agree that group-level restriction sets would be what we’d want if we supported anything other than uniform restrictions.

As long we don’t engineer ourselves into a corner with respect to this at some point in the future, I’d suggest punting on it for now

[martindurant]

This is a massive improvement in the usefulness of Gateway, wholly in favour! I can perhaps help. I agree with erring for situation 1. . For 2. it is indeed tricky, I think I favour an error still (“please try again later”!), but obviously autoscaling needs to know that scaling can fail.

To @ericdill 's comment: can_start_worker, etc., would be pluggable, and the org can implement how they make this decision themselves?