Audio Feature: ValueError: could not broadcast input array from shape (360000,2,1) into shape (360000,1)

Trying to use the audio input feature and it fails with this message: ValueError: could not broadcast input array from shape (360000,2,1) into shape (360000,1)

I’ve tried this out with a wav and ogg/vorbis file as well and neither worked.

I’ve tested this on ludwig 0.4.1 with Python 3.8.10

Stack trace:

Traceback (most recent call last):
  File "model.py", line 30, in <module>
    model.train(
  File "env/lib/python3.8/site-packages/ludwig/api.py", line 415, in train
    preprocessed_data = self.preprocess(
  File "env/lib/python3.8/site-packages/ludwig/api.py", line 1337, in preprocess
    preprocessed_data = preprocess_for_training(
  File "env/lib/python3.8/site-packages/ludwig/data/preprocessing.py", line 1517, in preprocess_for_training
    processed = data_format_processor.preprocess_for_training(
  File "env/lib/python3.8/site-packages/ludwig/data/preprocessing.py", line 235, in preprocess_for_training
    return _preprocess_file_for_training(
  File "env/lib/python3.8/site-packages/ludwig/data/preprocessing.py", line 1649, in _preprocess_file_for_training
    data, training_set_metadata = build_dataset(
  File "env/lib/python3.8/site-packages/ludwig/data/preprocessing.py", line 1151, in build_dataset
    proc_cols = build_data(
  File "env/lib/python3.8/site-packages/ludwig/data/preprocessing.py", line 1306, in build_data
    proc_cols = add_feature_data(
  File "env/lib/python3.8/site-packages/ludwig/features/audio_feature.py", line 372, in add_feature_data
    audio_features = AudioFeatureMixin._process_in_memory(
  File "env/lib/python3.8/site-packages/ludwig/features/audio_feature.py", line 159, in _process_in_memory
    processed_audio = df_engine.map_objects(
  File "env/lib/python3.8/site-packages/ludwig/data/dataframe/pandas.py", line 50, in map_objects
    return series.map(map_fn)
  File "env/lib/python3.8/site-packages/pandas/core/series.py", line 4237, in map
    new_values = self._map_values(arg, na_action=na_action)
  File "env/lib/python3.8/site-packages/pandas/core/base.py", line 880, in _map_values
    new_values = map_f(values, mapper)
  File "pandas/_libs/lib.pyx", line 2870, in pandas._libs.lib.map_infer
  File "env/lib/python3.8/site-packages/ludwig/features/audio_feature.py", line 161, in <lambda>
    lambda row: AudioFeatureMixin._transform_to_feature(
  File "env/lib/python3.8/site-packages/ludwig/features/audio_feature.py", line 249, in _transform_to_feature
    audio_feature_padded[:broadcast_feature_length, :] = audio_feature[
ValueError: could not broadcast input array from shape (360000,2,1) into shape (360000,1)

The dataset has the path to the file and the model is being trained with the following code:

model = LudwigModel({
    'input_features': [{
        'name': 'audio_path',
        'type': 'audio',
    }],
    'output_features: [{
        'name': 'track_artist',
        'type': 'category',
    }]
})
model.train(
    training_set='./dataset/dataset.csv',
    test_set='./dataset/test.csv',
)

[connor-mccorm]

Hi @modernlearner, I’m going to take a look at this issue. I should have an answer for you in a few hours! Will keep you updated on my progress.

[modernlearner]

I think this is the fix.

# 0.5rc2
audio_feature_padded[:broadcast_feature_length, :] = audio_feature[:max_length, 1]

Then I ran into an issue with CUDA memory running out. However, it did get past the ValueError 🎉

For the other version of ludwig I was using the fix is this:

# 0.4.1
audio_feature_padded[:broadcast_feature_length, :] = audio_feature[
                                                             :max_length, 1]

And it did run correctly.