command line arg parsing for Wav2vec example on TPU

A) wav2vec example when executed on TPU via using fairseq cli arguments leads to the following error:

Traceback (most recent call last):
  File "/home/sivaibhav/.local/lib/python3.8/site-packages/hydra/core/override_parser/overrides_visitor.py", line 302, in visitFunction
    return self.functions.eval(function)
  File "/home/sivaibhav/.local/lib/python3.8/site-packages/hydra/_internal/grammar/functions.py", line 34, in eval
    raise HydraException(
hydra.errors.HydraException: Unknown function 'InferredW2vConfig'
Available: bool,choice,float,glob,int,interval,range,shuffle,sort,str,tag
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "./train.py", line 14, in <module>
    cli_main()
  File "/home/sivaibhav/fairseq-dev/fairseq_cli/train.py", line 496, in cli_main
    cfg = convert_namespace_to_omegaconf(args)
  File "/home/sivaibhav/fairseq-dev/fairseq/dataclass/utils.py", line 389, in convert_namespace_to_omegaconf
    composed_cfg = compose("config", overrides=overrides, strict=False)
  File "/home/sivaibhav/.local/lib/python3.8/site-packages/hydra/experimental/compose.py", line 31, in compose
    cfg = gh.hydra.compose_config(
  File "/home/sivaibhav/.local/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 507, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/home/sivaibhav/.local/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 151, in load_configuration
    return self._load_configuration(
  File "/home/sivaibhav/.local/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 180, in _load_configuration
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "/home/sivaibhav/.local/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 95, in parse_overrides
    raise OverrideParseException(
hydra.errors.OverrideParseException: Error parsing override 'task.inferred_w2v_config=InferredW2vConfig(mask_length='${model.mask_length}', mask_prob='${model.mask_prob}', mask_selecti
on='${model.mask_selection}', mask_other='${model.mask_other}', no_mask_overlap='${model.no_mask_overlap}', mask_min_space='${model.mask_min_space}', mask_channel_length='${model.mask_
channel_length}', mask_channel_prob='${model.mask_channel_prob}', mask_channel_selection='${model.mask_channel_selection}', mask_channel_other='${model.mask_channel_other}', no_mask_ch
annel_overlap='${model.no_mask_channel_overlap}', mask_channel_min_space='${model.mask_channel_min_space}', conv_feature_layers='${model.conv_feature_layers}', encoder_embed_dim='${mod
el.encoder_embed_dim}')'
HydraException while evaluating 'InferredW2vConfig(mask_length='${model.mask_length}', mask_prob='${model.mask_prob}', mask_selection='${model.mask_selection}', mask_other='${model.mas
k_other}', no_mask_overlap='${model.no_mask_overlap}', mask_min_space='${model.mask_min_space}', mask_channel_length='${model.mask_channel_length}', mask_channel_prob='${model.mask_cha
nnel_prob}', mask_channel_selection='${model.mask_channel_selection}', mask_channel_other='${model.mask_channel_other}', no_mask_channel_overlap='${model.no_mask_channel_overlap}', mas
k_channel_min_space='${model.mask_channel_min_space}', conv_feature_layers='${model.conv_feature_layers}', encoder_embed_dim='${model.encoder_embed_dim}')': Unknown function 'InferredW
2vConfig'
Available: bool,choice,float,glob,int,interval,range,shuffle,sort,str,tag

Commandline used is:

export OMP_NUM_THREADS=1
    python3 ./train.py \
       /home/sivaibhav/manifest/ \
       --num-batch-buckets 3 \
       --tpu \
       --max-sentences 4 \
       --max-sentences-valid 4 \
       --required-batch-size-multiple 4 \
       --distributed-world-size 8 \
       --distributed-port 12597 \
       --update-freq 1 \
       --enable-padding \
       --log-interval 20 \
       --num-workers 6 \
       --task audio_pretraining \
       --criterion wav2vec \
       --arch wav2vec2 \
       --log-keys  "['prob_perplexity','code_perplexity','temp']" \
       --quantize-targets \
       --extractor-mode default \
       --conv-feature-layers '[(512, 10, 5)] + [(512, 3, 2)] * 4 + [(512,2,2)] * 2' \
       --final-dim 256 \
       --latent-vars 320 \
       --latent-groups 2 \
       --latent-temp '(2,0.5,0.999995)' \
       --infonce \
       --optimizer adam \
       --adam-betas '(0.9,0.98)' \
       --adam-eps 1e-06 \
       --lr-scheduler polynomial_decay \
       --total-num-update 400000 \
       --lr 0.0005 \
       --warmup-updates 32000 \
       --encoder-layerdrop 0 \
       --dropout-input 0.0 \
       --dropout-features 0.0 \
       --feature-grad-mult 0.1 \
       --loss-weights '[0.1, 10]' \
       --conv-pos 128 \
       --conv-pos-groups 16 \
       --num-negatives 100 \
       --cross-sample-negatives 0 \
       --max-sample-size 250000 \
       --min-sample-size 32000 \
       --dropout 0.0 \
       --attention-dropout 0.0 \
       --weight-decay 0.01 \
       --max-tokens 1400000 \
       --skip-invalid-size-inputs-valid-test \
       --ddp-backend no_c10d \
       --log-format simple \

B) When using hydra config file:

We see the following error:

Traceback (most recent call last):
  File "/home/sivaibhav/fairseq/fairseq_cli/hydra_train.py", line 45, in hydra_main
    distributed_utils.call_main(cfg, pre_main)
  File "/home/sivaibhav/fairseq/fairseq/distributed/utils.py", line 369, in call_main
    main(cfg, **kwargs)
  File "/home/sivaibhav/fairseq/fairseq_cli/train.py", line 124, in main
    task.load_dataset(valid_sub_split, combine=False, epoch=1)
  File "/home/sivaibhav/fairseq/fairseq/tasks/audio_pretraining.py", line 245, in load_dataset
    if self.cfg.tpu and task_cfg["mask_channel_prob"] == 0.0:
omegaconf.errors.ConfigKeyError: Key 'mask_channel_prob' not in 'AudioPretrainingConfig'
        full_key: mask_channel_prob
        reference_type=Optional[AudioPretrainingConfig]
        object_type=AudioPretrainingConfig

This can be reproduced using:

OMP_NUM_THREADS=1 fairseq-hydra-train   task.data=/home/sivaibhav/manifest   --config-dir ./examples/wav2vec/config/pretraining   --config-name wav2vec2_large_librivox_tpu.yaml

[kaleko]

Any update on this issue? I have a similar issue, though when trying to run fairseq.checkpoint_utils.load_model_ensemble_and_task on a wav2vec model that I fine tuned myself with fairseq-hydra-train. My issue looks like this:

omegaconf.errors.ConfigKeyError: Key 'eval_wer' not in 'AudioPretrainingConfig'
        full_key: eval_wer
        reference_type=Optional[AudioPretrainingConfig]
        object_type=AudioPretrainingConfig

[xjtupanda]

@KiriKoppelgaard I suppose not since I was just trying to extract features using pretrained models. But to work around this, in the end I used transformers package and loaded the pretrained model from faecbook’s Hugging Face pages, and it worked just fine without any error or warning.