Calculating number of linear regions

Dear authors,

I am having a question for calculating number of linear regions. It seems that in TE-NAS, input images are augmented to be of size (1000,1,3,3): lrc_model = Linear_Region_Collector(input_size=(1000, 1, 3, 3), sample_batch=3, dataset=xargs.dataset, data_path=xargs.data_path, seed=xargs.rand_seed)

Could you explain what the reason is behind this?

[Arioll]

Hello, I’m trying to calculate number of linear regions but the linear region collector always returns number of dimensions (the same number for all networks).

        @torch.no_grad()
        def update2D(self, activations):
  
            n_batch = activations.size()[0]
  
            n_neuron = activations.size()[1]
  
            self.n_neuron = n_neuron
  
            if self.activations is None:
  
                self.activations = torch.zeros(self.n_samples, n_neuron).cuda()
  
            self.activations[self.ptr:self.ptr+n_batch] = torch.sign(activations)  # after ReLU
  
            self.ptr += n_batch
  
  
        @torch.no_grad()
        def calc_LR(self):
  
       (1) res = torch.matmul(self.activations.half(), (1-self.activations).T.half()) # each element in res: A * (1 - B)
  
       (2) res += res.T # make symmetric, each element in res: A * (1 - B) + (1 - A) * B, a non-zero element indicate a pair of two different linear regions
  
       (3) res = 1 - torch.sign(res) # a non-zero element now indicate two linear regions are identical
  
            res = res.sum(1) # for each sample's linear region: how many identical regions from other samples
  
            res = 1. / res.float() # contribution of each redudant (repeated) linear region
  
            self.n_LR = res.sum().item() # sum of unique regions (by aggregating contribution of all regions)
  
            del self.activations, res
  
            self.activations = None

Here are functions from lib/procedures/linear_region_counter.py which computes number of linear regions. However, as far as I understand, if we have ReLU network, then after each layer we have non negative outputs so torch.sign will return 0 or 1. So, according to the first function, self.activations matrix will contain only 0 or 1 as it’s elements. In this case result of the lines (1), (2) and (3) will always be an identity matrix. Could you explain the idea behind the algorithm? Thank you

[chenwydj]

Hi @taoyang1122 and @maryanpetruk,

We used V100 GPU to train ImageNet. It is true that training-from-scratch on ImageNet is slow: 4~5 days are very common.