Question about policy_loss

Hi, thank you for your great work!!

I have a question related to #10. Can you explain the meaning of the code below in the GaussianPolicy??

# Enforcing Action Bound
log_prob -= torch.log(1 - action.pow(2) + epsilon)

Also, can you provide some information which you referenced to code this loss??

Anyway, thank you for sharing your great codes.

[toshikwa]

In gaussian policies, action is calculated like this.

1. x ~ Gaussian(mean, std)
2. y = tanh(x)
3. a = k * y

Here is how to calculate log likelihood of action (to get entropy).

1. likelihood of x is p(x) = 1 / (sqrt(2pi) * std) * exp(-1/(2std^2) * (x - mean)^2)

2. likelihood of y is p(y) = p(x) / |dy/dx| = p(x) / (1 - tanh(x)^2) = p(x) / (1 - y^2)

3. likelihood of a is p(a) = p(y) * |dy/da| = p(y) / k = p(x) / (k * (1 - y^2))

Finally, you get this. log(p(a)) = log(p(x)) - log(k*(1 - y^2))

[sunnyswag]

@ku2482 Ok, happy ending. 😃

Thank you !