Failing tests

Hi, I have fresh installed speechbrain from 0.5.8 to the last release and I spotted 4 tests that are failing with weird assertions. I’m not 100% sure but I do not think these tests were failing on the older install.

Here is the tests output:

FAILED tests/unittests/test_features.py::test_features_multimic[cuda] - AssertionError: assert tensor(0.0008, device='cuda:0') < 5e-05
FAILED tests/unittests/test_profiling.py::test_profile_class[cuda] - assert 3 == 2
FAILED tests/unittests/test_profiling.py::test_scheduler[cuda] - assert 1 == 0
FAILED tests/unittests/test_profiling.py::test_aggregated_traces[cuda] - assert (4222 == 4209 or 4209 == 2993)

[GAN-pie]

On my side it behaves differently.

>>> print(brain_or_pretrained.profiler.events()) does not gives an empty line, but the following:

---------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                       Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
---------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
         cudaGetDeviceCount         0.03%     500.000us         0.03%     500.000us     500.000us             1  
         cudaGetDeviceCount         0.00%       0.000us         0.00%       0.000us       0.000us             1  
    cudaGetDeviceProperties         0.01%     116.000us         0.01%     116.000us     116.000us             1  
    cudaGetDeviceProperties         0.01%     105.000us         0.01%     105.000us     105.000us             1  
      cudaDeviceSynchronize        99.96%        1.921s        99.96%        1.921s        1.921s             1  
---------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 1.921s

and brain_or_pretrained.profiler.profiler returns:

[<FunctionEvent id=0 name=cudaGetDeviceCount device_type=DeviceType.CPU node_id=-1 cpu_time=500.000us start_us=32 end_us=532 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceCount thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaGetDeviceCount device_type=DeviceType.CPU node_id=-1 cpu_time=0.000us start_us=547 end_us=547 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceCount thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaGetDeviceProperties device_type=DeviceType.CPU node_id=-1 cpu_time=116.000us start_us=551 end_us=667 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceProperties thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaGetDeviceProperties device_type=DeviceType.CPU node_id=-1 cpu_time=105.000us start_us=726 end_us=831 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceProperties thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaDeviceSynchronize device_type=DeviceType.CPU node_id=-1 cpu_time=1.921s start_us=876 end_us=1921557 cpu_children=[] cuda_time=0.000us name=cudaDeviceSynchronize thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>]

However, when using:

brain_or_pretrained.evaluate(
    test_set=(
        [inputs, targets],  # +1x test_set -> step_num=2
        [inputs, targets],
        [inputs, targets],
        [inputs, targets],
    )
)

It returns a float: 0.682352602481842 Then print(brain_or_pretrained.profiler.events()) gives me:

-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                            aten::zeros         0.00%       8.000us         0.00%      10.000us      10.000us             1  
                                            aten::empty         0.00%       1.000us         0.00%       1.000us       1.000us             1  
                                            aten::zero_         0.00%       1.000us         0.00%       1.000us       1.000us             1  
                                          ProfilerStep*         0.01%     143.000us         0.01%     152.000us     152.000us             1  
                                            aten::empty         0.00%       1.000us         0.00%       1.000us       1.000us             1  
                                            aten::zeros         0.00%       1.000us         0.00%       3.000us       3.000us             1  
                                            aten::empty         0.00%       2.000us         0.00%       2.000us       2.000us             1  
                                            aten::zero_         0.00%       0.000us         0.00%       0.000us       0.000us             1  
enumerate(DataLoader)#_SingleProcessDataLoaderIter._...         0.00%       5.000us         0.00%       5.000us       5.000us             1  
                                            aten::empty         0.00%       0.000us         0.00%       0.000us       0.000us             1  
                                     cudaGetDeviceCount         0.03%     499.000us         0.03%     499.000us     499.000us             1  
                                     cudaGetDeviceCount         0.00%       0.000us         0.00%       0.000us       0.000us             1  
                                cudaGetDeviceProperties         0.01%     120.000us         0.01%     120.000us     120.000us             1  
                                cudaGetDeviceProperties         0.01%     103.000us         0.01%     103.000us     103.000us             1  
                                  cudaDeviceSynchronize        99.95%        1.905s        99.95%        1.905s        1.905s             1  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 1.905s

Finally brain_or_pretrained.profiler.profiler outputs the following list:

[<FunctionEvent id=1 name=aten::zeros device_type=DeviceType.CPU node_id=-1 cpu_time=10.000us start_us=25 end_us=35 cpu_children=[2, 3] cuda_time=0.000us name=aten::zeros thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=2 name=aten::empty device_type=DeviceType.CPU node_id=-1 cpu_time=1.000us start_us=26 end_us=27 cpu_children=[] cuda_time=0.000us name=aten::empty thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=3 name=aten::zero_ device_type=DeviceType.CPU node_id=-1 cpu_time=1.000us start_us=34 end_us=35 cpu_children=[] cuda_time=0.000us name=aten::zero_ thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=4 name=ProfilerStep* device_type=DeviceType.CPU node_id=-1 cpu_time=152.000us start_us=40 end_us=192 cpu_children=[5, 6, 8, 9] cuda_time=0.000us name=ProfilerStep* thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=5 name=aten::empty device_type=DeviceType.CPU node_id=-1 cpu_time=1.000us start_us=41 end_us=42 cpu_children=[] cuda_time=0.000us name=aten::empty thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=6 name=aten::zeros device_type=DeviceType.CPU node_id=-1 cpu_time=3.000us start_us=46 end_us=49 cpu_children=[7] cuda_time=0.000us name=aten::zeros thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=7 name=aten::empty device_type=DeviceType.CPU node_id=-1 cpu_time=2.000us start_us=46 end_us=48 cpu_children=[] cuda_time=0.000us name=aten::empty thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=8 name=aten::zero_ device_type=DeviceType.CPU node_id=-1 cpu_time=0.000us start_us=49 end_us=49 cpu_children=[] cuda_time=0.000us name=aten::zero_ thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=9 name=enumerate(DataLoader)#_SingleProcessDataLoaderIter.__next__ device_type=DeviceType.CPU node_id=-1 cpu_time=5.000us start_us=51 end_us=56 cpu_children=[10] cuda_time=0.000us name=enumerate(DataLoader)#_SingleProcessDataLoaderIter.__next__ thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=10 name=aten::empty device_type=DeviceType.CPU node_id=-1 cpu_time=0.000us start_us=52 end_us=52 cpu_children=[] cuda_time=0.000us name=aten::empty thread=1 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaGetDeviceCount device_type=DeviceType.CPU node_id=-1 cpu_time=499.000us start_us=226 end_us=725 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceCount thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaGetDeviceCount device_type=DeviceType.CPU node_id=-1 cpu_time=0.000us start_us=737 end_us=737 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceCount thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaGetDeviceProperties device_type=DeviceType.CPU node_id=-1 cpu_time=120.000us start_us=740 end_us=860 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceProperties thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaGetDeviceProperties device_type=DeviceType.CPU node_id=-1 cpu_time=103.000us start_us=931 end_us=1034 cpu_children=[] cuda_time=0.000us name=cudaGetDeviceProperties thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>, <FunctionEvent id=0 name=cudaDeviceSynchronize device_type=DeviceType.CPU node_id=-1 cpu_time=1.905s start_us=1081 end_us=1905643 cpu_children=[] cuda_time=0.000us name=cudaDeviceSynchronize thread=0 input_shapes=[] cpu_memory_usage=0 cuda_memory_usage=0 is_async=False is_remote=False seq_nr=-1 is_legacy=False>]

Hope it will help.

[anautsch]

Closing this issue as the PR fixing this has been merged. Please feel free to re-open.