Code vs paper

Hello,

thank you for publishing your code - outstanding work 😃 However I have a question regarding JSD/GAN based estimators and differences between implementation and formulation in your paper: Eq. 4:

Eq. 7: At the same time, in your code: (for the JSD estimator)

if mode == 'fd':
    loss = fenchel_dual_loss(l_enc, m_enc, measure=measure)
[...]
E_pos = get_positive_expectation(u, measure, average=False).mean(2).mean(2)
E_neg = get_negative_expectation(u, measure, average=False).mean(2).mean(2)
[...]
Ep = log_2 - F.softplus(-p_samples)  # Note JSD will be shifted
Eq = F.softplus(-q_samples) + q_samples - log_2  # Note JSD will be shifted

While I do know, where does thie log_2 come from [Nowozin et al., 2016], the addition of q_samples in Eq is a bit more mysterious 😄 And then for the prior matching:

    if not loss_type or loss_type == 'minimax':
        return get_negative_expectation(q_samples, measure)
    elif loss_type == 'non-saturating':
        return -get_positive_expectation(q_samples, measure)

Seems like you are using just half of the equation 7 to obtain loss value.

Could you clarify those differences (maybe I am missing something in the code)? I have been trying to merge DIM with my existing code (a bit different setup, yet should work together properly) and cannot get it to work well.

Thanks in advance!

[rdevon]

I think it’s an artifact of the olden days of Theano, as I’m guessing in PyTorch these are equivalent.

[SkyeLu]

Oh sorry, I think you have done this maximize procedure in class Discriminator. Sorry to disturb.