Check finite difference gradient approximation in a random direction

scipy.optimize.check_grad is very convenient for checking analytical gradient calculations.

Unfortunately, it’s not really practical for even moderately large problems, because computing finite differences in every direction can be painfully slow.

One practical way to test such gradients, which I learned from the autograd project, is calculate finite differences in a random direction instead, which can be compared to a projection of the analytical gradient. This isn’t as thorough as check_grad but in practice seems to catch most of the same bugs, and runs way faster.

Would something like this be welcome in SciPy, maybe in the form of a random_projection=True keyword argument for check_grads?

[Mau-dvr75]

Hi, shoyer. One question. Why should we need to compute finite differences in every direction to compare analytical gradients with numerical ones??

[czgdp1807]

Thanks @shoyer for the ideas and detailed explanation.

but it makes sense to pick something with a non-zero component in most directions in order to catch bugs (rather than using a one-hot vector).

I was thinking the same. Using one of all the one-hot vector might be beneficial in the sense that we won’t be required to take the projection of analytical gradient, instead just indexing it will work. But yeah, bugs may escape. Using a fully random normalized vector would be much better.

I will raise a PR on Monday for this and we can implement and discuss different ways there. Thanks.