Editing annotation access through API (or viewing them through UI) sometimes deletes annotation elements [HIGH PRIORITY]

The is a bug that causes the dangerous side effect of deleting annotation elements when the access permissions are edited!

This behavior happens CONSISTENTLY when we APPEND to the existing users or groups, and OCCASIONALLY when we replace the users or groups with new ones. Until this bug is fixed, try to avoid changing annotation access permissions, and if you really do have to do it, make SURE you replace existing users/groups as opposed to appending to them.

To help track down the bug here are some details:

• Here is a video showing this behavior in real-time: https://youtu.be/8jcH_9uX5m8
• This only happens when the annotation access is edited through the girder client API, but not through the UI.
• One particular folder and all its subfolders show the consistent behavior of deleting the annotation elements even when we fully replace the existing list of users or groups. The other folders only show this behavior when we append to the list of users as opposed to fully replacing them.
• The particular DSA instance we are using may not be the last version (or it may be, @dgutman can comment on that). I know for sure that the DSA instance version we are using uses https, and therefore /gider needs to be added in the middle of the link when a DSA slide is opened in the HistomicsTK UI. See this video capture: https://youtu.be/iyX2MXCYMEk
• This is the code snippet I am using to edit the annotation access:

def update_permissions_for_annotation(
        gc, annotation_id=None, annotation=None,
        groups_to_add=[], replace_original_groups=True,
        users_to_add=[], replace_original_users=True):
    """Update permissions for a single annotation.
    Parameters
    ----------
    gc : gider_client.GirderClient
        authenticated girder client instance
    annotation_id : str
        girder id of annotation
    annotation : dict
        overrides annotation_id if given
    groups_to_add : list
        each entry is a dict containing the information about user groups
        to add and their permission levels. A sample entry must have the
        following keys
        - level, int -> 0 (view), 1 (edit) or 2 (owner)
        - name, str -> name of user group
        - id, st -> girder id of user group
    replace_original_groups : bool
        whether to replace original groups or append to them
    users_to_add : list
        each entry is a dict containing the information about user
        to add and their permission levels. A sample entry must have the
        following keys
        - level, int -> 0 (view), 1 (edit) or 2 (owner)
        - login, str -> username of user
        - id, st -> girder id of user
    replace_original_users
        whether to replace original users or append to them
    Returns
    -------
    dict
        server response
    """
    # TODO -- this is an especially damaging bug! Fix me!
    GIRDERBUG = """
    We've discovered a bug that causes the dangerous side effect of 
    deleting annotation elements whent he access permissions are edited!
    This behavior happens CONSISTENTLY when we APPEND to the existing
    users of groups, and OCCASIONALLY when we replace the users or
    groups with new ones. Until this bug is fixed, try to avoid 
    changing annotation access permissions, and if you really do
    have to do it, make SURE you replace existing users/groups
    as opposed to appending to them."""
    warnings.warn(GIRDERBUG, RuntimeWarning)

    global ASKTOCONTINUE
    if ASKTOCONTINUE:
        warnings.warn("""
        > Press any key to continue at your own risk, or.. 
        > Press Control+C to stop.""")
        input("")
        ASKTOCONTINUE = False

    if annotation is not None:
        annotation_id = annotation['_id']
    elif annotation_id is None:
        raise Exception(
            "You must provide either the annotation or its girder id.")

    # get current permissions
    current = gc.get('/annotation/%s/access' % annotation_id)

    # add or replace as needed
    if replace_original_groups:
        current['groups'] = []
    else:
        raise Exception(GIRDERBUG)

    if replace_original_users:
        current['users'] = []
    else:
        raise Exception(GIRDERBUG)

    for group in groups_to_add:
        current['groups'].append(group)

    for user in users_to_add:
        current['users'].append(user)

    # now update accordingly
    # OLD WAY (BAD!!)
    # return gc.put('/annotation/%s/access?access=%s' % (
    #        annotation_id, json.dumps(current)))
    # PROPER WAY
    return gc.put('/annotation/%s/access' % annotation_id, data={
        'access': json.dumps(current)})

• This is how I managed to revert the annotation and get back the elements:

def revert_annotation(
        gc, annotation_id=None, annotation=None, version=None,
        revert_to_nonempty_elements=False, only_revert_if_empty=True):
    """Revert an annotation to a previous version.
    Parameters
    ----------
    gc : girder_client.GirderClient
        authenticated girder client
    annotation_id : str
        girder id of annotation
    annotation : dict
        overrides annotation_id if given
    version : int
        versoin number for annotation. If None, and
        not revert_to_nonempty_elements
        the default behavior of the endpoint is evoked, which reverts the
        annotation if it was deleted and if not, reverts to the last version.
    revert_to_nonempty_elements : bool
        if true, reverts to the most recent version of the annotation
        with non-empty elements.
    Returns
    -------
    dict
        server response
    """
    if annotation is not None:
        annotation_id = annotation['_id']
    elif annotation_id is None:
        raise Exception(
            "You must provide either the annotation or its girder id.")

    history = gc.get("/annotation/%s/history" % annotation_id)

    # no need to revert if empty
    if only_revert_if_empty and len(history[0]["groups"]) > 0:
        return dict()

    if (version is None) and revert_to_nonempty_elements:

        # NOTE: even though the "history" may show
        # the elements as empty, the "groups" attribute is really the
        # indication if the annotation version actually has some elements.
        # TODO -- This is likely a bug (?); fix me!!!
        for ver in history:
            if len(ver["groups"]) > 0:
                version = ver['_version']
                break

    ver = "" if version is None else "?version=%d" % version

    if version is None:
        print("    Reverting ...")
    else:
        print("    Reverting to version %d" % version)

    return gc.put("/annotation/%s/history/revert%s" % (annotation_id, ver))

• Even though the “history” may show the elements as empty for a particular annotation version, the “groups” attribute is really the indication if the annotation version actually has some elements. Is this a bug??

• We’ve experience some of these things before and always were baffled as they weren’t so reproducible. Now, it seems this is a reproducible bug, especially on that particular server and collection. Annotations are showing up in the wrong place and observing an annotation causes it to disappear! I recorded a video of this behavior: https://youtu.be/Q1Xj7yBuZFQ March 12, 2020 meeting update: Here is the history of the moved anntoation …

[
  {
    "_id": "5d9cc27c46929b04713bfdb8",
    "_version": 331490,
    "_versionId": "5e65255c7632342dac6a7a32",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5d9624c946929b0471d03ae3",
          "level": 1
        },
        {
          "flags": [],
          "id": "5e44a2e0ddda5f8398785304",
          "level": 0
        }
      ],
      "users": []
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [
      "(+)tumor",
      "(-)tumor",
      "roi",
      "unknown"
    ],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-12T19:27:08.188000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e6a8d0c5ba2ecde741a6110",
    "_version": 331478,
    "_versionId": "5e620c7634679044bdaa77ac",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5d9624c946929b0471d03ae3",
          "level": 1
        }
      ],
      "users": []
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T09:01:02.763000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e65255c7632342dac6a7a32",
    "_version": 331036,
    "_versionId": "5e620c7634679044bdaa77ac",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5d9624c946929b0471d03ae3",
          "level": 1
        },
        {
          "flags": [],
          "id": "5e44a2e0ddda5f8398785304",
          "level": 0
        }
      ],
      "users": []
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [
      "(+)tumor",
      "(-)tumor",
      "roi",
      "unknown"
    ],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T09:01:02.763000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e62114e34679044bdaadac4",
    "_version": 330821,
    "_versionId": "5e62096a34679044bda9a06e",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5e61330034679044bda81c85",
          "level": 1
        }
      ],
      "users": [
        {
          "flags": [],
          "id": "5956b83b92ca9a000d9c0488",
          "level": 2
        },
        {
          "flags": [],
          "id": "59bc677892ca9a0017c2e855",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbadf2e629140075e2b9f4",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbae92e629140075e2b9fa",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbaed5e629140075e2b9fd",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbaefde629140075e2ba00",
          "level": 2
        }
      ]
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T08:39:40.078000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e620c7634679044bdaa77ac",
    "_version": 330757,
    "_versionId": "5e62096a34679044bda9a06e",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5d9624c946929b0471d03ae3",
          "level": 1
        },
        {
          "flags": [],
          "id": "5e44a2e0ddda5f8398785304",
          "level": 0
        }
      ],
      "users": []
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [
      "(+)tumor",
      "(-)tumor",
      "roi",
      "unknown"
    ],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T08:39:40.078000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e620c4c34679044bdaa60cd",
    "_version": 330106,
    "_versionId": "5e62014b34679044bda8d6d0",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5e61330034679044bda81c85",
          "level": 1
        }
      ],
      "users": [
        {
          "flags": [],
          "id": "5956b83b92ca9a000d9c0488",
          "level": 2
        },
        {
          "flags": [],
          "id": "59bc677892ca9a0017c2e855",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbadf2e629140075e2b9f4",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbae92e629140075e2b9fa",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbaed5e629140075e2b9fd",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbaefde629140075e2ba00",
          "level": 2
        }
      ]
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T08:26:39.178000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e62096a34679044bda9a06e",
    "_version": 330044,
    "_versionId": "5e62014b34679044bda8d6d0",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5d9624c946929b0471d03ae3",
          "level": 1
        },
        {
          "flags": [],
          "id": "5e44a2e0ddda5f8398785304",
          "level": 0
        }
      ],
      "users": []
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [
      "(+)tumor",
      "(-)tumor",
      "roi",
      "unknown"
    ],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T08:26:39.178000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e62093f34679044bda98b81",
    "_version": 329343,
    "_versionId": "5e6135ed34679044bda81df4",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5e61330034679044bda81c85",
          "level": 1
        }
      ],
      "users": [
        {
          "flags": [],
          "id": "5956b83b92ca9a000d9c0488",
          "level": 2
        },
        {
          "flags": [],
          "id": "59bc677892ca9a0017c2e855",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbadf2e629140075e2b9f4",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbae92e629140075e2b9fa",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbaed5e629140075e2b9fd",
          "level": 2
        },
        {
          "flags": [],
          "id": "5ccbaefde629140075e2ba00",
          "level": 2
        }
      ]
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T07:51:53.819000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  },
  {
    "_active": false,
    "_annotationId": "5d9cc27c46929b04713bfdb8",
    "_id": "5e62014b34679044bda8d6d0",
    "_version": 329281,
    "_versionId": "5e6135ed34679044bda81df4",
    "access": {
      "groups": [
        {
          "flags": [],
          "id": "5d9624c946929b0471d03ae3",
          "level": 1
        },
        {
          "flags": [],
          "id": "5e44a2e0ddda5f8398785304",
          "level": 0
        }
      ],
      "users": []
    },
    "annotation": {},
    "created": "2019-10-08T17:08:12.243000+00:00",
    "creatorId": "5ccbaefde629140075e2ba00",
    "groups": [
      "(+)tumor",
      "(-)tumor",
      "roi",
      "unknown"
    ],
    "itemId": "5d94313746929b0471cf4d81",
    "public": false,
    "updated": "2020-03-06T07:51:53.819000+00:00",
    "updatedId": "5cc9bbd2e629140075e2b3fa"
  }
]

• March 12, 2020 meeting update: This works even while the database is stale and is not being simultaneously backed up, so ignore the point below.

• On a related note, when a database is being backed up or if the admin is doing anything on the database, it seems any activity corrupts the data. It may be the case that this weird behavior I recorded is precisely due to the fact that a database backup operation is happening while I am querying it. This should absolutely not be the case. Background operations should either: A. Block everyone from accessing the database, or B. Not impact things at all. Note that this may or may not explain the previous times we’ve seen this happen in the past and were perplexed about what happened.

• March 12, 2020 meeting update:

  • Any single annotation permission viewed through the Histomics UI causes its elements to disappear.
  • The bug is replicable through the DSA interface (as well as HistomicsUI where it was first discovered)
  • Sending a JSON should be in the BODY of the request, as opposed to being encoded in the URL string itself (see comment from David M below).

Some discussion can also be found here: https://github.com/DigitalSlideArchive/HistomicsTK/pull/821

[manthey]

This should be fixed by https://github.com/girder/large_image/pull/428

[kheffah]

@manthey OK I checked and the PR does fix this bug. Fantastic!