[Bug] cholesky singular U

🐛 Bug

To reproduce

I followed the directions here for a variational GP (https://gpytorch.readthedocs.io/en/latest/examples/05_Scalable_GP_Regression_Multidimensional/SVGP_Regression_CUDA.html#Overview) with the HEAD build of gpytorch and it trains for a while and then crashes with…

** Stack trace/error message **

  File "main.py", line 93, in <module>
    pred = model(x)███████████████████████████████████████████████████████████████████████▉       | 999/1088 [00:42<00:02, 31.22it/s]
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/models/approximate_gp.py", line 56, in __call__
    return self.variational_strategy(inputs, prior=prior)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/variational/whitened_variational_strategy.py", line 240, in __call__
    return Module.__call__(self, x)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/module.py", line 24, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/variational/whitened_variational_strategy.py", line 141, in forward
    induc_induc_covar = CholLazyTensor(induc_induc_covar.cholesky())                                                                
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/lazy/lazy_tensor.py", line 738, in cholesky
    res = self._cholesky()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/utils/memoize.py", line 34, in g
    add_to_cache(self, cache_name, method(self, *args, **kwargs))
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/lazy/lazy_tensor.py", line 413, in _cholesky
    cholesky = psd_safe_cholesky(evaluated_mat).contiguous()
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/utils/cholesky.py", line 39, in psd_safe_cholesky
    raise e
  File "/st2/jeff/anaconda3/envs/jeff/lib/python3.7/site-packages/gpytorch-0.3.6-py3.7.egg/gpytorch/utils/cholesky.py", line 22, in psd_safe_cholesky
    L = torch.cholesky(A, upper=upper, out=out)
RuntimeError: cholesky_cpu: U(1,1) is zero, singular U.

Expected Behavior

I’m not sure why it is running into a singular matrix there, as I would expect that case to be covered.

System information

Please complete the following information: 0.3.6 - cd1c8f8 1.2.0 Ubuntu

Additional context

Additionally, I get a cuda OOM error every time I try to call .to(device) using gpytorch, even with minimal data, which is strange.

[gpleiss]

@deltaskelta - this may or may not correct the issue, but can you replace WhitenedVariationalStrategy with VariationalStrategy? The examples for variational inference need some updating (to come soon).

Also @Balandat I’m putting up a quick PR to give better NaN warnings in psd_safe_cholesky.

[jeffwillette]

Thanks for the info!