An empty get_module() returning NotImplementedError will skip all unit tests!

When a model in TorchBenchmark raises NotImplementedError in the get_module() function, it will skip all the unit tests!

This happens because test.py will accept and skip the exception, NotImplementedError. These are the reasons why every test is skipped:

1. check_example
2. check_device
   • check_example and check_device directly calls model.get_module(), which may raise NotImplementedError.
3. train
4. eval
   • train and eval unit tests call set_train() and set_eval() respectively before running the method
   • set_train/eval() will call _set_mode(), where (model, _) = self.get_module() is called and may raise NotImplementedError.

For 1 and 2, it may make sense to skip them since their functionality requires get_module(). However for 3 and 4, we may choose:

• Overwrite _set_mode() to something that works (or pass).
• Skip _set_mode() and directly run train and eval.
• Require every model to implement get_module(). (This would also fix 1 and 2).

[jansel]

Here are a couple examples of get_module() being used to check correctness of custom transformations:

• https://github.com/pytorch/pytorch/pull/68563
• https://github.com/jansel/torchdynamo/blob/main/torchbench.py

Accomplishing the same thing with train()/eval() would not be possible with the current design. Especially the custom sync logic in the first example.

“is it of people’s interest to test the custom transformations on all the models?”

Yes this is the primary thing I use torchbench for.

[xuzhao9]

Here are a couple examples of get_module() being used to check correctness of custom transformations:

• Add lazy_bench.py to measure trace overhead and compute efficiency pytorch#68563
• https://github.com/jansel/torchdynamo/blob/main/torchbench.py

Accomplishing the same thing with train()/eval() would not be possible with the current design. Especially the custom sync logic in the first example.

“is it of people’s interest to test the custom transformations on all the models?”

Yes this is the primary thing I use torchbench for.

Thank you for the explanation. Could you please help add get_module() interface to the models that are missing (listed by @aaronenyeshi in the comment above)? After that, we could fix a unit test that makes sure every model needs to implement this interface.