NNI 2021 Oct~Nov Iteration Plan

This is the plan for the iteration in 2021 Nov~Dec Iteration, it’s a 6 weeks iteration.

Release Plan for upcoming release

• Release manager: @liuzhe-lz
• Feature freeze date: 11-19 11-26 (last delay)
• Code freeze and demo date: 12-3 12-8 12-10 (last delay)
• Branch cut and next release planning date: 12-7 12-24
• Release date: 12-10 12-29

Documentation Update

• @ultmaster Check statics
  • @QuanluZhang Get Ni Yan’s account
• Decide which ones to keep Chinese doc
• Update docs for current release
• Deal with Chinese doc @ultmaster (12.1) #4345
• HPO doc hierarchy @ zebin (11.23)

NAS

• end2end example for NAS with Retiarii (target starting date 11/17)
  • P0 - @JiahangXu evaluate current SPOS support https://nni.readthedocs.io/en/stable/NAS/SPOS.html #4322
  • P0 - @JiahangXu evaluate current ProxylessNAS support https://nni.readthedocs.io/en/stable/NAS/Proxylessnas.html (11.19) (Jiahang: the example works, refine its doc in PR #4321 )
• P0 - @ultmaster Serialization (11.5 11.12 11.19 11.24) #4337
• P1 - @QuanluZhang Fix webui chart
  • survey (11.3)
  • fix (11.10) #4304
  • further design
• P1 - @Lijiaoa Trial parameters (11.23) #4314
• P1 - @QuanluZhang option of not exit experiment after done #4340

Model Compression

• P0 - @J-shang Add doc for v2 config list
• Pruner Improvement
  • @J-shang Support pruning by step (movement pruning, taylor, activation…). (11.12) #4308
• Migrate Pruner from v1
  • @J-shang AutoCompress Pruner. (11.5) #4280 review: @QuanluZhang @Fiascolsy
  • P0 - @Fiascolsy AMC Pruner. (11.5 11.12 11.19) #4320
• P2 - @Fiascolsy Fix AutoCompressPruner GPU memory leak.
• Doc
  • @J-shang Guide on how to customize pruner in v2.
  • P1 - @Fiascolsy Report performance of v2 compare with paper / v1. #4365
  • Need add: fine-grained with speed up in iteration will case issue. how to write config_list.
  • review doc & update the outdated doc.
    • P1 - @Fiascolsy pruning
    • @linbinskn quantize
  • P1 - @J-shang review overview and tutorial hirachey
• P2 - @Fiascolsy More v2 Example #4305
• @J-shang pass optimizer to pruner redesign (11.26) #4332

HPO

• @QuanluZhang Survey optuna, onedb, sagemaker
• @QuanluZhang fix dispatcher error, experiment status
• P1 - (zebin) improve tuner doc
• @ultmaster Survey ConfigSpace
• P1 - @liuzhe-lz TPE (PR #4239)
  • doc #4339
• Nested search space
  • P1 - @liuzhe-lz Grid search (11.5 11.10 11.19) #4324

nnictl

• P0 - @liuzhe-lz Deprecate config v1 (11.1)
  • ConfigBase class #4303
  • local training service #4313
  • All training services #4318
  • launcher #4330
  • update nni manager #4355
  • fix annotation (11.26)
  • run ITs
• P1 - @liuzhe-lz Raise error for setting invalid field on ConfigBase
• P0 - @liuzhe-lz add validation for trialGpuNumber/maxTrialNumberPerGpu combination

NNI manager & training service

• P1 - @SparkSnail Pipeline for reuse training services (11.5 11.8 11.10) #4310
• P1 - @SparkSnail Hybrid for all reuse training services (11.26) #4343

WebUI

• Overview page: blank at the bottom (11.16)
• Update readthedoc index.html (11.4 11.10)
• About trial duration question https://github.com/microsoft/nni/issues/3012

Other

• readthedocs dependency
• doc translation

Deferred

NAS

• P1 - @JiahangXu a TBD end2end example for NAS with Retiarii
• P1 - @QuanluZhang Support tuner_no_more_trial
• P1 - @JiahangXu review graph engine
• P? - @ultmaster Update mnasnet and example refactor (plan: #4249)
• P2 - (zebin) More choice type design
• P2 - (zebin) Survey naslib
• P2 - (zebin) Port Houwen’s algorithm
• P2 - @liuzhe-lz Merge RetiariiExperiment back to Experiment
• P3 - Visualization nasbench
• P3 - Support HPO in Retiarii
• P3 - Review one-shot, refactor with lightning

Compression

• Pruning Speed Up (check with ningxin)
• P1 - Fine-grained iterative speedup logic.
• P1 - NetAdapt Pruner
• @J-shang Transformer Pruner. (11.5 11.8) #4180
• @zheng-ningxin Function replacement
• P1 - Sensitivity Pruner.
• @J-shang ADMM: support more metrics.
• P2 - Support min_sparsity_per_layer in global mode.
• P2 - Mixed sparse pattern.
• P2 - Support mask output/input.
• P2 - Pruning with constraints (remain 4x channels).
• P2 - @Fiascolsy Iterative pruning starts from any sparsity ratio.
• P2 - KD in pruning scheduler (maybe use original model as teacher).
• @zheng-ningxin Memory usage optimization
• Refactor of model graph generation (as an independent component)
• Speed up with special/customized layer.
• Quantization V2

HPO

• P1 - fix len(arr) > 0 problem
• P1 - (zebin) Improve hyperband & PBT with shared storage
• P3 - Multi-objective
• P2 - (zebin) transferable HPO
• P2 - Evolution nested search space

nnictl

• P2 - @J-shang Improve nnictl experiment export
• P2 - @liuzhe-lz auto-completion
• P2 - @liuzhe-lz early validation

NNI Manager

• P2 - Distributed trial
• P1 - @acured Project back-end

Web UI

• Align issue in overview page
• Support searching customized string in the search box
• Experiment list could not open experiment with prefix https://github.com/microsoft/nni/issues/3746 (need backend)

[chenbohua3]

Hi @scarlett2018 I want to add these items for this iteration:

• About quantization:

  • Support dtype & qscheme customization for lsq quantizer and observer quantizer (P0)
  • upgrade lsq quantizer to lsq+ quantizer, which can used for asymmetric quantization (P0)
  • add model converter for mnn backend (P1)

• About architecture for quantization V2:

  • Analyze the architecture of the PyTorch quantization tool in the open source community, and summarize the advantages and disadvantages of each tool.(P0), with @linbinskn

• A demo case for quantizing YOLOX and deploy it on PyTorch backend (P1)

[JiahangXu]

NAS

• Refactor NAS examples of ProxylessNAS and SPOS.
• Support latency constraint on edge device for ProxylessNAS based on nn-Meter.