[Feature Request] allow for dashed overrides

🚀 Feature Request

It would be nice if one could use dashed overrides as well as the current non-dashed version

arg1: default1
arg2: default2

python app.py arg1=override1 arg2=override2
python app.py arg1=override1 --arg2=override2
python app.py arg1=override1 --arg3=unknown_arg1  # this still fails

Some scripts assume that arguments have the preceding dashes and thus breaks if you call hydra scripts. A notable example is the pytorch distributed launch script. The pytorch example basically invokes multiple runs of the given command with --local_rank={local_rank} attached to the command. So even if you have local_rank as an expected variable in your hydra config, it fails because of the formatting of the variable.

Motivation

I want to use hydra with pytorch distributed training and specifically something like python -m torch.distributed.launch --nproc_per_node=2 app.py arg1=override1

Pitch

I haven’t done extensive testing, but I think the following diff on hydra/_internal/utils.py would enable this feature.

diff --git a/hydra/_internal/utils.py b/hydra/_internal/utils.py
index 7eb38bf0..f23ac64f 100644
--- a/hydra/_internal/utils.py
+++ b/hydra/_internal/utils.py
@@ -148,6 +148,15 @@ def create_config_search_path(search_path_dir: Optional[str]) -> ConfigSearchPat
     return search_path
 
 
+def _convert_dashed_args(unknown_args: List[str]) -> List[str]:
+    new_args = []
+    for arg in unknown_args:
+        while arg.startswith("-"):
+            arg = arg[1:]
+        new_args.append(arg)
+    return new_args
+
+
 def run_hydra(
     args_parser: argparse.ArgumentParser,
     task_function: TaskFunction,
@@ -172,7 +181,8 @@ def run_hydra(
         task_name=task_name, config_search_path=search_path, strict=strict
     )
     try:
-        args = args_parser.parse_args()
+        args, unknown_args = args_parser.parse_known_args()
+        args.overrides += _convert_dashed_args(unknown_args)
         if args.help:
             hydra.app_help(config_name=config_name, args_parser=args_parser, args=args)
             sys.exit(0)
@@ -276,7 +286,9 @@ def get_args_parser() -> argparse.ArgumentParser:
 
 
 def get_args(args: Optional[Sequence[str]] = None) -> Any:
-    return get_args_parser().parse_args(args=args)
+    known_args, unknown_args = get_args_parser().parse_known_args(args=args)
+    known_args.overrides += _convert_dashed_args(unknown_args)
+    return known_args
 
 
 def _strict_mode_strategy(strict: Optional[bool], config_name: Optional[str]) -> bool:

[reactivetype]

Another use case where – syntax would be nice to have is for use with some editors’s launch configuration. For e.g., with MS Visual studio code, we would pass a list of args as follows:

args = [ ''--param_1=value1", "--param2=value2"]

[omry]

@reactivetype :

Another use case where – syntax would be nice to have is for use with some editors’s launch configuration. For e.g., with MS Visual studio code, we would pass a list of args as follows:

args = [ ''--param_1=value1", "--param2=value2"]

I am not using VSC myself, by traditionally IDEs does not assume getopt style and allow you to pass anything to the app.