ENH: New interface for Brain

⚠️ Requires pyvista>=0.24 ⚠️

This PR summarizes the ideas to implement in Brain, previously called _TimeViewer (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569378110):

Milestone 0.20

• Get it working for VectorSourceEstimates (added in #7060)
• Improve the color cycle of point picking to allow point color re-use (#7516)
• Check that the latest version 0.24 of pyvista is used (added in https://github.com/mne-tools/mne-python/pull/7506)
• Put time_viewer='auto' and show_traces='auto' as the default for stc.plot / plot_source_estimates (added in https://github.com/mne-tools/mne-python/pull/7380)
• Use the latest version 0.24 of pyvista (added in https://github.com/mne-tools/mne-python/pull/7458)
• Use interpolation nearest as the default (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-589447242, added in https://github.com/mne-tools/mne-python/pull/7257, reverted in 95ad0eec8a2e1e9c7c74e25852157e650aea7df0, added in https://github.com/mne-tools/mne-python/pull/7469)
• Improve performance of interpolation (added in https://github.com/mne-tools/mne-python/pull/7257, reverted in 95ad0eec8a2e1e9c7c74e25852157e650aea7df0, added in https://github.com/mne-tools/mne-python/pull/7469)
• Display current time with a line in MplCanvas (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-582899553, added in https://github.com/mne-tools/mne-python/pull/7423)
• Link the time plot of MplCanvas with the time slider (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-582899553, added in https://github.com/mne-tools/mne-python/pull/7423)
• Add a way to clear all the added points in show_traces=True mode (added in #7386)

Milestone 0.21

• Set PyVista as the default 3d backend (done in https://github.com/mne-tools/mne-python/pull/8220)
• Add reset key that resets camera all views and time to what they were at the beginning / viz init (suggested in https://github.com/mne-tools/mne-python/pull/8082#issuecomment-672948671, done in https://github.com/mne-tools/mne-python/pull/8125)
• Link brains picking infos (suggested in https://github.com/mne-tools/mne-python/issues/7959, done in https://github.com/mne-tools/mne-python/pull/8112)
• Link brains colorbar (suggested in https://github.com/mne-tools/mne-python/pull/7227#issuecomment-577361561, done in https://github.com/mne-tools/mne-python/pull/8126)
• Link brains camera (suggested in https://github.com/mne-tools/mne-python/issues/7959, done in https://github.com/mne-tools/mne-python/pull/7962)
• Remove _MNE_BRAIN_TRACES_AUTO=false and build the frontpage after https://github.com/mne-tools/mne-python/pull/7927 and https://github.com/mne-tools/mne-python/pull/7791 (suggested in https://github.com/mne-tools/mne-python/pull/7925#issuecomment-649584156; in progress in #7927)
• Scrape traces (done in #7927)
• Get picking infos with get_picked_points() (suggested in https://github.com/mne-tools/mne-python/issues/7959, done in https://github.com/mne-tools/mne-python/pull/7963)
• Add remove_labels (suggested in https://github.com/mne-tools/mne-python/issues/7959, done in https://github.com/mne-tools/mne-python/pull/7964)
• Add support for pinch-to-zoom (suggested in https://github.com/mne-tools/mne-python/issues/7162#issuecomment-604403711; done in https://github.com/pyvista/pyvistaqt/pull/26)
• Fix sizing when TimeViewer is present (added in #7775)
• Add mne.surface in the public_modules (suggested in https://github.com/mne-tools/mne-python/pull/7289#discussion_r375273059, added in https://github.com/mne-tools/mne-python/pull/7930)
• Make the internal matplotlib figure to match foreground/background colors with the render view (suggested in https://github.com/mne-tools/mne-python/pull/7843#issuecomment-635384571; done in #7925)
• Remove PySurfer from CircleCI (probably after changing plot_parcellation.py to just iterate over labels, see #7275, added in https://github.com/mne-tools/mne-python/pull/7659, https://github.com/mne-tools/mne-python/pull/7695, https://github.com/mne-tools/mne-python/pull/7702)
• Improve toolbar icon consistency and resolution (suggested in https://github.com/mne-tools/mne-python/pull/7612#issuecomment-614703740, added in https://github.com/mne-tools/mne-python/pull/7698)
• Use button widgets (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569346780, merged upstream, proof of concept in https://github.com/mne-tools/mne-python/pull/7200, limitations encountered in https://github.com/mne-tools/mne-python/pull/7200#issuecomment-574273958) or vtkPlaybackWidget for Play/Pause control of the playback, added in https://github.com/mne-tools/mne-python/pull/7589)
• Add a way to create a movie/GIF (added in https://github.com/mne-tools/mne-python/pull/7257, reverted in 95ad0eec8a2e1e9c7c74e25852157e650aea7df0, added in https://github.com/mne-tools/mne-python/pull/7574)
• Update latest.inc and plot_visualize_stc to mention save_movie() (suggested in https://github.com/mne-tools/mne-python/pull/7257#pullrequestreview-374179244, added in https://github.com/mne-tools/mne-python/pull/7257, reverted in 95ad0eec8a2e1e9c7c74e25852157e650aea7df0, added in in https://github.com/mne-tools/mne-python/pull/7574)
• Use a progress bar during movie saving (suggested in https://github.com/mne-tools/mne-python/pull/7257#discussion_r389260208, added in https://github.com/mne-tools/mne-python/pull/7257, reverted in 95ad0eec8a2e1e9c7c74e25852157e650aea7df0, added in https://github.com/mne-tools/mne-python/pull/7612)
• Improve performance by using directly the widget representation instead of widget name or use pass_widget option to widget callback (added in https://github.com/mne-tools/mne-python/pull/7227)

Milestone 0.22

• Fix colorbar representation with transparent=True, should alpha blend fmin->fmid with brain_color (added in https://github.com/mne-tools/mne-python/pull/8643)
• Fix memory problems (#8399, #8379, #8427, #8575)
• Merge “_TimeViewer” with _Brain (added in https://github.com/mne-tools/mne-python/pull/8352)

Milestone 0.23

• Get jupyter notebook integration working properly with multi-views and time course plotting/clicking (partially solved by https://github.com/mne-tools/mne-python/pull/7758, added in in https://github.com/mne-tools/mne-python/pull/8503)
• Add brain silhouette (added in https://github.com/mne-tools/mne-python/pull/8771)
• Add better intructions to integrate in Report (added in https://github.com/mne-tools/mne-python/pull/8730)
• Add shorcuts to rotate the camera and shift time (added in https://github.com/mne-tools/mne-python/pull/8792)
• Widget migration from VTK to Qt/Jupyter Widgets (added in https://github.com/mne-tools/mne-python/pull/8862)
• Light/Dark theme switcher (suggested in https://github.com/mne-tools/mne-python/pull/9000#issuecomment-791557878, added in https://github.com/mne-tools/mne-python/pull/9149)

Milestone 1.0

Milestone 1.1

• add a color parameter to add_sensors() to set channels color (suggested in mne.discourse.group)
• Interactive sensor-level field patterns browser (suggested in https://github.com/mne-tools/mne-python/issues/8382#event-3892037680, work in progress in https://github.com/mne-tools/mne-python/pull/8803)
• Interface for drawing ROIs using the mouse
• Move the vertexId/MNI from the legend to the status bar (suggested in https://github.com/mne-tools/mne-python/pull/8792#issuecomment-767898353)
• Fix ctrl-C pressing during generation of movies – should abort generation, currently sometimes just stops a given VTK render (caught by vtkRenderWindowInteractor) sometimes stops the screenshot generation at our end but still saves a movie, etc.
• Fix multi-window memory problems (#8241, #8808), macos failures (https://github.com/mne-tools/mne-python/pull/8611#issuecomment-739806982, #8619)
• Change the theme with QEvent.PaletteChange (suggested in https://github.com/mne-tools/mne-python/issues/9182)
• Add support for shadows (suggested in https://github.com/mne-tools/mne-python/issues/7162#issuecomment-813504371)

Done

• Unnecessary updates of Brain (reported in https://github.com/mne-tools/mne-python/issues/8986; fixed in #8246)
• Reduce pyvista viz test time (suggested in https://github.com/mne-tools/mne-python/pull/7726#issuecomment-624046382, https://github.com/mne-tools/mne-python/issues/8242; done in #8246)
• Lower the CPU usage (done in #7517)
• Refresh slider style (done in https://github.com/mne-tools/mne-python/pull/7509)
• Avoid the confusion between the default pyvista cell picking and the in _TimeViewer (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-589599734)
• Vertex picking -> time course (suggested in #7235, added in #7247)
• Tag a stable version of PyVista to use with _TimeViewer (added in https://github.com/mne-tools/mne-python/pull/7318))
• Add a legend/help (suggested in https://github.com/mne-tools/mne-python/issues/7268#issuecomment-580158431, added in https://github.com/mne-tools/mne-python/pull/7305)
• Switch from time_actor to time_slider_title to allow scaling with the window (added in https://github.com/mne-tools/mne-python/pull/7263)
• Use a contextmanager for tests using widgets (suggested in https://github.com/mne-tools/mne-python/pull/7276#issuecomment-581346281, added in #7280)
• Improve documentation about hemi (suggested in https://github.com/mne-tools/mne-python/issues/7250#issuecomment-578122285, added in https://github.com/mne-tools/mne-python/pull/7256)
• Allow fmin, fmid, and fmax to take on all data values, and bump the scalar range while interacting with the slider as necessary (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-571636885, added in https://github.com/mne-tools/mne-python/pull/7188)
• Add a fscale slider (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569002405, added in 068654f5a4967d9505163f480792cf5ee5022709)
• Add support for smoothing=nearest (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569455048, added in https://github.com/mne-tools/mne-python/pull/7191)
• Use vertical orientation for the scalarbar (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-568758879, added in 3bbc6c29f67f70e594b8fbbf8a0be2f0a30fc9bd)
• Use text slider widget (merged upstream, added in https://github.com/mne-tools/mne-python/pull/7186)
• Use PySurfer-like time dilation (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569346365, added in https://github.com/mne-tools/mne-python/pull/7200)
• Bring colorbar values auto-scaling (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569261106, added in https://github.com/mne-tools/mne-python/pull/7218, fixed in https://github.com/mne-tools/mne-python/pull/7226)
• Add support for hemi=split (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569346365, added in https://github.com/mne-tools/mne-python/pull/7219)
• Link brains timeline (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569415964, added in https://github.com/mne-tools/mne-python/pull/7227)

Bug

• The window title bar does not follow the theme (reported in https://github.com/mne-tools/mne-python/issues/9182)
• Unexpected behaviour with shortcuts (reported in https://github.com/mne-tools/mne-python/pull/8792#issuecomment-770160715)
• The main window opens in the background in jupyter qtconsole (reported in https://github.com/mne-tools/mne-python/pull/7305#issuecomment-585748906, https://github.com/mne-tools/mne-python/issues/7162#issuecomment-585889205)
• Detect no-opengl to avoid crash (reported in https://github.com/mne-tools/mne-python/issues/9333, fixed in https://github.com/pyvista/pyvista/pull/2250)
• Fix off-by-one in screenshot(time_viewer=True) (reported in https://github.com/mne-tools/mne-python/issues/9336, https://github.com/mne-tools/mne-python/pull/9351)
• Flatmap are not compatible with hemi=both (reported in https://github.com/mne-tools/mne-python/issues/9306, #9315)
• Unify sliders and combo boxes to use period instead of comma (reported in https://github.com/mne-tools/mne-python/pull/9126#issuecomment-801725029, fixed in https://github.com/mne-tools/mne-python/pull/9179)
• Wrong color with add_label (reported in https://github.com/mne-tools/mne-python/issues/8815#issue-797593479, fixed in https://github.com/mne-tools/mne-python/pull/8820)
• There are visual artifacts on overlapping surfaces on the brain (reported in https://github.com/mne-tools/mne-python/issues/7599, fixed in https://github.com/mne-tools/mne-python/pull/8576)
• The camera is reset too often causing the brain to “bounce” (fixed by https://github.com/mne-tools/mne-python/pull/8572)
• The glyphs colormap are not updated interactively (reported in https://github.com/mne-tools/mne-python/pull/7843#issuecomment-635395395; done in #7934)
• The parameter foreground is not supported in _Brain (reported in https://github.com/mne-tools/mne-python/issues/7842; done in https://github.com/mne-tools/mne-python/pull/7843)
• There is an unnecessary window title decoration bar on macOS (reported in https://github.com/mne-tools/mne-python/issues/7316)
• The focalpoint when hemi=both should be between the hemispheres (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-582899553, added in https://github.com/mne-tools/mne-python/pull/7390)
• The title parameter is not taken into account (reported in https://github.com/mne-tools/mne-python/issues/7162#issuecomment-594536145, fixed in https://github.com/mne-tools/mne-python/pull/7382 and https://github.com/mne-tools/mne-python/pull/7396)
• The slider handles when hemi=both overlap (reported in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-588279423, fixed in https://github.com/mne-tools/mne-python/pull/7383)
• The initial_time is not taken into account (reported in https://github.com/mne-tools/mne-python/issues/7379 added in https://github.com/mne-tools/mne-python/pull/7381)
• dorsal view causes the _Brain object to disappear (reported in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-568735444, fixed in https://github.com/mne-tools/mne-python/pull/7187)
• The colorbar labels are not updated when the range is modified (fixed in a70b992f4a1b1de0a753f7e36bd9656d1ea4f1df)

HARD

• Use slider with multiple handles (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-571665347 and https://github.com/mne-tools/mne-python/pull/7153#issuecomment-571667909, prototype available on gist)
• Improve help window UX (suggested in https://github.com/mne-tools/mne-python/pull/7305#issuecomment-585814805)
• Put label of the orientation slider above the bar (suggested in https://github.com/mne-tools/mne-python/pull/7153#issuecomment-572004994, prototype available on https://github.com/mne-tools/mne-python/pull/7211)

Mockup

From https://github.com/mne-tools/mne-python/pull/7153#issuecomment-569315991:

Progress

From plot_visualize_stc.py:

https://user-images.githubusercontent.com/18143289/103798907-a7ec1100-504a-11eb-8d45-f775836cbb9c.mp4

Current status

https://user-images.githubusercontent.com/18143289/116316751-af2e5a80-a7b2-11eb-9f83-bf81cc07a3e6.mp4

[larsoner]

@GuillaumeFavelier can you update the top comment with what you want to do for 0.24 and then for 1.0 based on your conversations with @agramfort ? Let’s leave this as milestoned for 0.24 until all 0.24 boxes are ticked (or bumped to 1.0) so we don’t forget anything

[akaszynski]

Feel free to add to the list above or give it a shot if you agree @GuillaumeFavelier . At some point we will want to check that shadows + EDL + SSAO all get along, too.

I’m going to work on getting the PR working, but after that I’ll work on getting multi-pass working according to MultiPass Rendering.