Why is the HSIC not minimized but maximized?

Thank you for such a great work! When reading the paper and code, I have the following questions.

According to the definition of HSIC, it measures the level of independence and HSIC(U, V) = 0 indicates that U and V are independent. A larger HSIC value indicates that U and V are dependent to some extent.

So, to debias the representation of network f by using a biased network g, shouldn’t we minimize HSIC(f, g)?

Besides, the for loop in line 62 seems redundant because the g_dim will be over-written by the last loop, right?

Looking forward to your reply. Thanks!

[SanghyukChun]

@Cogito2012 In theory, you have to alternatively update to correctly solve a minimax problem, which aims to find a saddle point:

If you jointly optimize min and max problems at the same time, it will not guarantee a correct saddle point. I even cannot sure that such optimization will be converged to a certain point.

Note that our conceptual objective function is

\min_f [L(f) + \max_g HSIC(f, g)]

which have to solve alternative updates for f and g, respectively.

Such minimax problem is popular in many machine learning methods such as

• Generative adversarial networks (alternatively update generator – max and discriminator – min)
• Adversarial training (alternatively update the worst case input --max and the model parameter – min)

[Cogito2012]

Let me rephrase my concern. Can we simplify the alternative updating as minimizing the objective?

L(f) + L(g) + HSIC(f, \fixed{g}) - HSIC(\fixed{f}, g)

where \fixed{f} or \fixed{g} means the network parameters of f or g will not be updated by corresponding HSIC term.

If this is also working, then this kind of ReBias variant may be even easier to use.