Resource Info Request

This is a bit of a follow on to https://github.com/jupyter/jupyter/issues/212, addressing the computational context section of the jupyter spark roadmap.

/cc @holdenk @minrk @yuvipanda @rdblue

---

When working in a jupyter environment, it would be helpful to see current resource usage

• CPU(s)
• Memory of the system
• Memory of the current kernel

Beyond that, people need information from libraries. Taking Spark as an example, they want to know several fields:

• Spark UI (string - URL)
• Memory on executors
• State of executors (?)
• Spark Version (string)
• Hadoop Version (string)

Min, Yuvi and I have been talking about a message, similar to the kernel_info_request called resource_info_request and resource_info_reply to get information about the system.

Message type: resource_info_request:

content = {
}

Message type: resource_info_reply:

content = {
  resources: [
    {
      type: "kernel",
      metrics: [
        { value: x, max: y, name: "cpu" },
        { value: x, max: y, name: "memory" }
      ]
    }
  ]
};

If the kernel supports it (and the frontend wants to handle it), other types / libraries can provide their own information on resources (in Python, they’d have a registration API).

{
  type: "spark",
  metrics: [
    { value: x, name: "num_executors", display_name: "Executors" },
    ...
  ]
}

Optionally, if the kernel sends the system level data:

    {
      type: "system",
      metrics: [
        { value: x, max: y, name: "cpu" },
        { value: x, max: y, name: "memory" }
      ]
    },

I’m not hard set on any of the messaging here, so I’d love to hear more about naming and extensibility on top of this.

---

[rgbkrk]

Live at JupyterCon, a standing ovation occurred for this protocol change.

[ivanov]

I’ve got a prototype implementation of this that I’m developing with the help of the Jovyan Popular Front.