Doubts about NAS+TypeError: tuple is not a Module subclass

Describe the issue: I am using your framework to perform a NAS experiment on my use case. While using LayerChoice I get the below error. TypeError: tuple is not a Module subclass

Apart from the above issue, I have a few more questions:

1. If I understood correctly, right now with one-shot NAS you only support LayerChoice and InputChoice, right? In my use case I would like to use input choice and also repeat. Is it possible to do that? I started to use DARTS for it because it is faster. Would you recommend switching to another NAS approach to use other mutation options?

2. If I understood correctly, these are layerwise approaches. Would it be possible to extend it to a blockwise approach as well?

Environment*: Remote Server

• NNI version: 2.6
• Training service (local|remote|pai|aml|etc):
• Client OS:Ubuntu 20.04.3 LTS
• Server OS (for remote mode only):Ubuntu 20.04.3 LTS
• Python version:Python 3.8.12
• PyTorch/TensorFlow version: Toch 1.10.0+cu111
• Is conda/virtualenv/venv used?: conda
• Is running in Docker?: No

Configuration:

• Experiment config (remember to remove secrets!): one-shot DARTS

  self.pointwise=LayerChoice([
               (nn.Conv1d(
                in_channels, out_channels, kernel_size=1, stride=1,
                dilation=dilation, bias=bias, padding=0
            ),
            nn.Conv1d(
               in_channels, out_channels, kernel_size=2, stride=1,
                dilation=dilation, bias=bias, padding=0
            ),
              nn.Identity()

            )],label='point')
        else:
            self.conv = Conv1d(
                in_channels, out_channels, kernel_size=kernel_size,
                stride=stride, padding=padding, dilation=dilation, bias=bias
            )

    def forward(self, x):
        if self.separable:
            x = self.depthwise(x)
            x = self.pointwise(x)
        else:
            x = self.conv(x)
        return x

ERROR:

File "mutatemodel.py", line 174, in __init__
    self.pointwise=LayerChoice([
  File "nni/nni/retiarii/nn/pytorch/api.py", line 95, in __init__
    self.add_module(str(i), module)
  File "python3.8/site-packages/torch/nn/modules/module.py", line 377, in add_module
    raise TypeError("{} is not a Module subclass".format(
TypeError: tuple is not a Module subclass

I look forward to your reply.

Thanks!

[singagan]

Hi, @JiahangXu and @ultmaster, update on this issue: I was finally able to parse my model to run it with ProxylessNAS. I did the following two things to make it work:

1. Moved all the logic from forward() to init()
2. Updated kernel_detector.py to support tensors with axis other than 4. (I guess you hard-coded it for imagenet).

Now I am debugging everything for functional correctness. Right now, I am using the cortexA76cpu_tflite21 device.

I realized that my search loss reduces abit and then saturates around 1.3. Could it be because I am using 1D modules (conv1d, BatchNorm1d, etc) while the cortexA76cpu_tflite21 is built using 2D modules? or I need to name the module in a certain way so nn-meter can predict? Or it’s something else?

I will try to plug my own device once you release the nn-meter builder tools

I guess the stationary_block in latency_table are non-layerchoice modules, right?

[singagan]

Thanks, @JiahangXu! Please keep me posted.

Regarding enabled devices: that would be great if you will allow users to build their own latency predictors! I would certainly try to contribute there.