Object of type 'LazyFilter' has no len()

I’m running into this weird issue, and I’m not sure why adding manifests should create a lazy iterator?

In [5]: m = prepare_ali_meeting('/export/c01/corpora6/AliMeeting')
Preparing Train: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 209/209 [02:56<00:00,  1.19it/s]
WARNING:root:Removed 241 supervisions starting after the end of the recording.
WARNING:root:Trimmed 5 supervisions exceeding the end of the recording.
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4026 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4028 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4030 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 4 supervisions that start at 0 for recording R1019_M4031 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4033 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4035 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4039 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4041 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4043 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1019_M4051 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 4 supervisions that start at 0 for recording R1019_M4052 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1021_M1935 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1021_M1942 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1021_M1944 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1021_M1959 (channel 0). Did you forget to set supervision start times?
WARNING:root:SupervisionSet contains 2 supervisions that start at 0 for recording R1021_M1961 (channel 0). Did you forget to set supervision start times?
Preparing Eval: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 10.20it/s]
Preparing Test: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 11.17it/s]

In [6]: s = m["eval"]["supervisions"]

In [7]: s1 = s.filter(lambda seg: seg.duration >= 0.5)

In [8]: len(s1)
Out[8]: 5193

In [9]: s = m["eval"]["supervisions"] + m["test"]["supervisions"]

In [10]: s1 = s.filter(lambda seg: seg.duration >= 0.5)
/export/c07/draj/lhotse/lhotse/lazy.py:327: UserWarning: A lambda was passed to LazyFilter: it may prevent you from forking this process. If you experience issues with num_workers > 0 in torch.utils.data.DataLoader, try passing a regular function instead.
  "A lambda was passed to LazyFilter: it may prevent you from forking this process. "

In [11]: len(s1)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-a700e8f68e2d> in <module>
----> 1 len(s1)

/export/c07/draj/lhotse/lhotse/supervision.py in __len__(self)
    815
    816     def __len__(self) -> int:
--> 817         return len(self.segments)

TypeError: object of type 'LazyFilter' has no len()

Any suggestions?

[pzelasko]

we can either

1. replace some integrity checks/fixes like remove_missing_recordings_and_supervisions with a two-pass strategy that first iterates to get the sets of compatible/incompatible IDs, and in the second pass filters out the incompatible ones.

or

1. read everything into memory with manifest.to_eager() (will break for people with not enough CPU RAM and large data)

[pzelasko]

You can call .to_eager() on the resulting manifest to evaluate the expression. I started moving multiple operations to lazy evaluation a while ago, the only downside I can see is missing len. It’s just an extra function call away and for that we get memory-friendly behavior by default.