Improved Handling of Active and Archived Recordings

Brainstorming ideas for V2+ API handlers and how we want to deal with active and archived recordings, in ways that will support more flexible client use-cases and therefore better web-client UX. These may also include changes to related classes like SerializableRecordingDescriptor.

Idea 1

Merged active/archive queries with state filtering by query param, supporting negation

• GET /api/v2/targets/:targetId/recordings response:

{
  "meta": {
    "status": "OK",
    "type": "application/json"
  },
  "data": [
    {
        "id": 1,
        "name": "myRecording",
        "state": "RUNNING",
        "startTime": 12345,
        "duration": 0,
        "continuous": true,
        "toDisk": true,
        "maxSize": 0,
        "maxAge": 0
    },
    {
        "name": "myRecording.1",
        "state": "ARCHIVED",
        "startTime": 12345,
        "duration": 30000,
        "continuous": true,
        "toDisk": true,
        "maxSize": 0,
        "maxAge": 0
    }
  ]
}

Here, active recordings for the :targetId are included in the response, just like GET /api/v1/targets/:targetId/recordings - however, a state: ARCHIVED recording is also included in the response. This represents an archived copy of that recording that was taken 30000ms after the source recording was first started. It does not have a unique id of its own, and it is named in a way that also indicates it is archived and which source recording it came from. Other recording options, like continuous and maxSize, from the source recording are also preserved and included in the data about this archived recording.

Implementation details: we already have archives that are subdivided by target, so it should be relatively easy to create an API handler that queries the JVM for its active recordings, queries the filesystem for archived recordings from that target, and joins the results. The recording options can be stored in the filesystem alongside the archived recording - here, the archived recording would be at myRecording.1.jfr in the archive, and the continuous/toDisk/maxSize/maxAge/duration data could be stored in myRecording.1.[properties|json] in the same directory.

• GET /api/v2/targets/:targetId/recordings?state=ARCHIVED response:

{
  "meta": {
    "status": "OK",
    "type": "application/json"
  },
  "data": [
    {
        "name": "myRecording.1",
        "state": "ARCHIVED",
        "startTime": 12345,
        "duration": 30000,
        "continuous": true,
        "toDisk": true,
        "maxSize": 0,
        "maxAge": 0
    }
  ]
}

• GET /api/v2/targets/:targetId/recordings?state=!%3DARCHIVED response:

{
  "meta": {
    "status": "OK",
    "type": "application/json"
  },
  "data": [
    {
        "id": 1,
        "name": "myRecording",
        "state": "RUNNING",
        "startTime": 12345,
        "duration": 0,
        "continuous": true,
        "toDisk": true,
        "maxSize": 0,
        "maxAge": 0
    }
  ]
}

Idea 2

Notes and Labels for recordings

It would be useful if users could add notes and/or labels to recordings. Labels could be used as a query parameter filter and would be simply key-value pairs, while notes would be a general free-form text field. Labels might also include additional metadata applied by Cryostat. Labels would be included in query responses:

{
  "meta": {
    "status": "OK",
    "type": "application/json"
  },
  "data": [
    {
        "name": "myRecording",
        "state": "ARCHIVED",
        "startTime": 12345,
        "duration": 30000,
        "continuous": true,
        "toDisk": true,
        "maxSize": 0,
        "maxAge": 0,
        "labels": [
            "user/reason": "service-outage",
            "cryostat/datetime": "2021-10-08T14:56:17-04:00"
        ],
        "notes": "blah blah"
    }
  ]
}

Notes and Labels would be optionally applied to a recording when it is created. Users should also have a way to modify the notes and labels of a recording, whether active or archived, after the fact. This would most likely be a PATCH request against the recording resource.

Implementation notes: labels could be stored again as a .properties file in some location where Cryostat can correlate the properties to a recording. For active recordings, care would need to be taken to remove the properties file if we reconnect to a target and observe that the matching recording no longer exists (by matching name and start time?). The freeform notes would have some large upper limit on size and could also be stored in the filesystem. Eventually, both the notes and labels as well as the additional metadata properties mentioned in Idea 1 should probably be stored in a database, rather than the filesystem.

Web-client Motivations

Idea 2 would clearly add some capabilities to the web-client - the recording creation form would be enhanced with fields for notes and labels. The recording tables would display notes and labels, probably in the row expansion. There would also be some easy way to edit the notes and labels when they are visible.

Idea 1 would clean up the current Recordings view. As it is, the Recordings view is split between an Active and an Archived tab, but the Archived tab displays recordings from any target, not only the currently selected one. After Idea 1 is implemented, the web-client would simply have one Recordings view for a target, which would display both active and archived recordings together, and only the archived recordings for that selected target. There could be a check box or other toolbar filters to narrow down the results displayed in the table, including a filter for archived-only, or active-only, or running-only, etc. The web-client could then have an entirely separate view to display only archived recordings from all targets, which would display the data currently in the Archived Recordings tab of the Recordings view. This would be separated from the new Recordings view and its merged results table since the Archived view would now be non-target-specific.

[andrewazores]

Sure. It’s simply because the labels or other metadata we attach to recordings should have a similar lifecycle to the recording itself. If a recording lives in a target JVM’s memory then we shouldn’t lose the labels a user adds to that recording just because Cryostat was restarted while that recording is still ongoing in the target. Likewise, if it’s an archived recording that has some labels or metadata attached, that metadata should be preserved with the archived recording and should be retrievable by Cryostat if it’s restarted and reads the same archives. This would also be important for the future if we continue developing more microservices like -reports around Cryostat - if this metadata can be stored in something like a database that all the services have a connection to, then they can all share the same metadata about recordings. This would also be useful if in the future we ever have a deployment scenario where there are multiple Cryostat instances sharing work within the same namespace, for example - maybe they each run the same automated rules but they somehow agree on a way to partition the targets so that they aren’t all colliding and making duplicate requests.

It doesn’t necessarily need to be a database as in a SQL instance, it could also be a NoSQL DB or something else entirely like a simple key-value store. I don’t think we really have enough technical requirement laid out at this point to have much preference in these solutions. The important aspect is that the data store needs to be “durable”, so it should somehow be storing information to disk so that the Cryostat instance or the data store can both be restarted without losing data. Otherwise at this point the requirements seem pretty lax because we only have a few very simple use cases where basically anything will fit.

[andrewazores]

Idea 1 here is covered by the GraphQL API already, and Idea 2 is covered by the new metadata/labels that Janelles has implemented. I don’t think there is a compelling case to implement a new specific V2.1 API handler, or set of handlers, to delineate archived recordings by target, or have a way to combine those results with active recordings, when this is already exactly the kind of thing that GraphQL excels at and provides for us. Closing this.