Out of memory when running convert_dir_to_note_sequences

Hello everyone!

Today I observed something strange when using convert_dir_to_note_sequences.

When I run the tool on the Lakh dataset I observe this:

1. The script runs fine for a while. Converting one MIDI file after the other.
2. While the script is running the RAM usage is very low. I monitor with the tool free on Unix.
3. The script suddenly freezes. It stops at a file.
4. Looking at free shows that the memory usage quickly goes up until all memory is exhausted.
5. After a while the script terminates with the message Killed.

I have 60GB of RAM on the system. Magenta version is 2.1.0. Tensorflow version is 2.2.0.

I added a little logging to the script to get the file that is currently being processed. This is what I get:

INFO:tensorflow:Attempting to convert MIDI file lmd_full_split/train/a/a79fe18e50fb30cd1679e3a9e0960620.mid.
I0623 14:55:44.641093 140074287511360 convert_dir_to_note_sequences.py:141] 

Killed

I checked the file in Logic Pro X. It seems to work.

Any ideas?

[iljab]

@AmitMY @AI-Guru Same problem here, do you have any hints?

[AmitMY]

@AI-Guru I have the same problem. Did you manage to convert lakh dataset to tfrecord? If so, could you please share how?