[BUG] profiler crashes when alpha argument is set in torch.add()

Describe the bug profiler crashes when alpha argument is set in torch.add()

To Reproduce

import torch
from deepspeed.profiling.flops_profiler import FlopsProfiler

class MyModel(torch.nn.Module):
    def forward(self, input):
        return torch.add(input, input, alpha=5)


model = MyModel()
prof = FlopsProfiler(model)
prof.start_profile()
model(torch.ones([10]))
flops = prof.get_total_flops(as_string=True)

Expected behavior correct profile of flops

[sbrody18]

Up to you, of course. The big issue was the crash. A little inaccuracy in the estimation doesn’t make a big difference. but I would think that the calculation would be pretty simple: if alpha==1: element_wise else: element_wise + size(second tensor) since it would be multiplying the second tensor by alpha and then adding them.

[cli99]

@sbrody18, the flops computation is an estimation and we prefer to underestimate rather than overcount. As the default value for alpha is 1, I think we can keep the way of counting to be the same as other element-wise operators such as mul.