CUDA 11 Test: FFT tests failing

Failing Tests

tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfft2_param_0_{axes=None, norm=None, s=None, shape=(3, 4)}::test_irfft2 FAILED [ 82%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfft2_param_2_{axes=(-2, -1), norm=None, s=None, shape=(3, 4)}::test_irfft2 FAILED [ 83%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfft2_param_5_{axes=None, norm='ortho', s=None, shape=(3, 4)}::test_irfft2 FAILED [ 84%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfft2_param_6_{axes=None, norm=None, s=None, shape=(2, 3, 4)}::test_irfft2 FAILED [ 84%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfft2_param_8_{axes=(-3, -2, -1), norm=None, s=None, shape=(2, 3, 4)}::test_irfft2 FAILED [ 85%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfft2_param_10_{axes=(0, 1), norm=None, s=None, shape=(2, 3, 4)}::test_irfft2 FAILED [ 86%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfft2_param_11_{axes=None, norm='ortho', s=None, shape=(2, 3, 4)}::test_irfft2 FAILED [ 86%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfftn_param_0_{axes=None, norm=None, s=None, shape=(3, 4)}::test_irfftn FAILED [ 87%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfftn_param_2_{axes=(-2, -1), norm=None, s=None, shape=(3, 4)}::test_irfftn FAILED [ 88%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfftn_param_5_{axes=None, norm='ortho', s=None, shape=(3, 4)}::test_irfftn FAILED [ 89%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfftn_param_6_{axes=None, norm=None, s=None, shape=(2, 3, 4)}::test_irfftn FAILED [ 89%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfftn_param_8_{axes=(-3, -2, -1), norm=None, s=None, shape=(2, 3, 4)}::test_irfftn FAILED [ 90%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfftn_param_10_{axes=(0, 1), norm=None, s=None, shape=(2, 3, 4)}::test_irfftn FAILED [ 91%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestRfftn_param_11_{axes=None, norm='ortho', s=None, shape=(2, 3, 4)}::test_irfftn FAILED [ 91%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestHfft_param_13_{axis=0, n=15, norm=None, shape=(10, 10)}::test_hfft FAILED [ 95%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestHfft_param_13_{axis=0, n=15, norm=None, shape=(10, 10)}::test_hfft_backend FAILED [ 95%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestHfft_param_13_{axis=0, n=15, norm=None, shape=(10, 10)}::test_hfft_overwrite FAILED [ 95%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestHfft_param_15_{axis=0, n=15, norm='ortho', shape=(10, 10)}::test_hfft FAILED [ 95%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestHfft_param_15_{axis=0, n=15, norm='ortho', shape=(10, 10)}::test_hfft_backend FAILED [ 95%]
tests/cupyx_tests/scipy_tests/fft_tests/test_fft.py::TestHfft_param_15_{axis=0, n=15, norm='ortho', shape=(10, 10)}::test_hfft_overwrite FAILED [ 95%]

It seems there is a precision decrease in CUDA 11.

E               AssertionError: 
E               Not equal to tolerance rtol=0.0001, atol=1e-07
E               
E               Mismatch: 0.667%
E               Max absolute difference: 4.2915344e-06
E               Max relative difference: 0.00027141
E                x: array([[ 8.750391e+01,  7.829688e+01,  1.021133e+02,  6.904395e+01,
E                        4.463257e+01,  5.814697e+01,  5.923633e+01,  6.878809e+01,
E                        9.283203e+01,  5.962695e+01],...
E                y: array([[ 8.750391e+01,  7.829688e+01,  1.021133e+02,  6.904395e+01,
E                        4.463257e+01,  5.814697e+01,  5.923633e+01,  6.878809e+01,
E                        9.283203e+01,  5.962695e+01],...

Enviornment

cu>>> cupy.show_config()
CuPy Version          : 8.0.0b5
CUDA Root             : /usr/local/cuda
CUDA Build Version    : 11000
CUDA Driver Version   : 11000
CUDA Runtime Version  : 11000
cuBLAS Version        : 11100
cuFFT Version         : 10200
cuRAND Version        : 10201
cuSOLVER Version      : (10, 5, 0)
cuSPARSE Version      : 11100
NVRTC Version         : (11, 0)
Thrust Version        : 100909
CUB Build Version     : 100909
cuDNN Build Version   : None
cuDNN Version         : None
NCCL Build Version    : None
NCCL Runtime Version  : None
cuTENSOR Version      : None

[pentschev]

This is the reply I got from the cuFFT team:

Change of behavior was to improve performance and made based on assumption of we are interchangeable with fftw and can destroy C2R input buffer as this is fftw default behavior. It seems that there is demand for this behavior so we’ll put it on roadmap to allow preservation of input buffer but without loss of performance.

Note that there’s no deadline for that at the moment. Therefore, I see two options:

1. We skip the tests checking that input data remains unchanged – different from SciPy’s behavior;
2. We make a copy of the input data to work on – increases memory footprint.

Any other ideas of what we could do? If not, what’s people’s preference to the two options above?

[pentschev]

I just confirmed it works for me with CUDA 10.0 on a V100. I’ve modified the 3_Imaging/convolutionFFT2D sample to compare the input data before and after executing cufftExecC2R and indeed in CUDA 11.0 the input gets modified, but CUDA 10.0 the input data stays the same. I’m asking internally whether this is a bug or expected behavior.