IPython.display.Audio not working for WAV audio

Bug: Notebook Editor, Interactive Window, Editor cells

Issue has previously been discussed, but I have new information that is relevant. I’m forced to create a new issue because all previous issues have been closed and locked. Please do not close it without considering this new information and responding to it. I’m only looking for support for uncompressed wav audio format. No need for mp3, mpeg, ogg, or anything else. New information:

1. Previous discussion with @rchiodo was limited to supporting ffmpeg codec. Support for wav audio files is not dependent in any way on ffmpeg codec. Wav audio can be played back without any codec. No dependency on ffmpeg exists for playing wav audio, which is the most common type used for machine learning because it can be sampled and reconstructed easily (ie wavenet/wavernn).
2. Microsofter and VSCode creater, @bpasero, claims that sound may be played back without DOM support via node.js https://github.com/Microsoft/vscode/issues/421 This would eliminate previous reasons for closing this issue in the past.
3. @mjbvz says that ffmpeg codec is unsupported by the core VSCode, but: a) does not state that ffmpeg cant be supported by an extension like vscode-python b) does not state that any codec is required in order to play wav audio files https://github.com/microsoft/vscode/issues/66050
4. @mjbvz again states here that only certain types of media are unsupported and mentions a dependency on ffmpeg https://github.com/microsoft/vscode/issues/32540 He does not state that all audio playback is unsupported.

Steps to cause the bug to occur

Type the following code in VSCode, and run the cell (e.g. using SHIFT+ENTER):

import numpy as np
import IPython
T = 2.0    # seconds
sr = 22050 # sample rate
t = np.linspace(0, T, int(T*sr), endpoint=False) # time variable
x = 0.5*np.sin(2*np.pi*440*t)                # pure sine wave at 440 Hz
IPython.display.Audio(x, rate=sr)

You will get a disabled player. The same code in the browser will render a usable player.

Actual behavior

IPython.display.Audio renders a disabled player:

Expected behavior

A valid, playable audio control should be rendered, same as in the browser:

Your Jupyter and/or Python environment

Please provide as much info as you readily know

• Jupyter server running: Local
• Extension version: 2020.5.80290
• VS Code version: 1.45.1
• Setting python.jediEnabled: true
• Setting python.languageServer: Microsoft
• Python and/or Anaconda version: 3.7.6
• OS: Linux (Ubuntu 18.04 LTS)
• Virtual environment: conda

Developer Tools Console Output

Far too big to fit here. No relevant information. Just deprecation warnings and info output.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[matangover]

Hi all, I’ve found a workaround using the Web Audio API. Save the following into a file called vscode_audio.py:

import IPython.display
import numpy as np
import json

def Audio(audio: np.ndarray, sr: int):
    """
    Use instead of IPython.display.Audio as a workaround for VS Code.
    `audio` is an array with shape (channels, samples) or just (samples,) for mono.
    """

    if np.ndim(audio) == 1:
        channels = [audio.tolist()]
    else:
        channels = audio.tolist()

    return IPython.display.HTML("""
        <script>
            if (!window.audioContext) {
                window.audioContext = new AudioContext();
                window.playAudio = function(audioChannels, sr) {
                    const buffer = audioContext.createBuffer(audioChannels.length, audioChannels[0].length, sr);
                    for (let [channel, data] of audioChannels.entries()) {
                        buffer.copyToChannel(Float32Array.from(data), channel);
                    }
            
                    const source = audioContext.createBufferSource();
                    source.buffer = buffer;
                    source.connect(audioContext.destination);
                    source.start();
                }
            }
        </script>
        <button onclick="playAudio(%s, %s)">Play</button>
    """ % (json.dumps(channels), sr))

In your notebook, use:

from vscode_audio import Audio
Audio(audio, sr)

where audio in a numpy array with shape (channels, samples) or just (samples,) for mono.

This is quite hacky and slows the whole notebook for longer signals, can be improved. Anyway I suggest to reopen this issue.

[Honghe]

It would be nice if it can play audio.