run=False in CLI prevents some config keys from being accepted

🐛 Bug

Instantiating LightningCLI with run=False prevents some config keys (namely seed_everything and ckpt_path) from being accepted. This seems to be because these config keys are defined by a subcommand parser, and these are not initialized when run=False is passed.

This makes it impossible to load config files created by Lightning during training since they contain the seed_everything ckpt_path key.

To Reproduce

Boring model

error: 'Configuration check failed :: No action for destination key "ckpt_path" to check its value.'

Expected behavior

Should run without error.

Environment

• CUDA:
  • GPU:
    • Tesla K80
  • available: True
  • version: 11.1
• Packages:
  • numpy: 1.21.5
  • pyTorch_debug: False
  • pyTorch_version: 1.10.0+cu111
  • pytorch-lightning: 1.5.10
  • tqdm: 4.63.0
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.7.12
  • version: # 1 SMP Tue Dec 7 09:58:10 PST 2021

cc @carmocca @mauvilsa

[mauvilsa]

LightningCLI is a class intended to ease the implementation of command line tools. Any change in the python source code in general is a new CLI. For more context also read lightning_cli.html#notes-related-to-reproducibility.

With run=False only the classes are instantiated. It is up to the developer to implement which and how the Trainer methods are called. And to add to the argument parser any additional needed parameters.

[JiangWenPL]

I would say loading a model from a configuration file in jupyter notebook is a quite common scenario for many research projects. For example, we may want to create colab demo for our research projects or run some interactive visualization or experiment on downstream tasks with our model. A simple workaround for current version could be something like this:

class NBCLI(LightningCLI):
    def fit(self, *args, **kwargs):
        pass

You just need to instantiate this CLI instead of the CLI you are currently using. But still I think it would be good if those scenarios can be addressed by the LightningCLI or have some documentations or examples to make the LightningCLI much easier to use for researchers. I would like to help if you think instantiating the model with CLI follows the philosophy of this library.