question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Extract PCM audio stream for waveform

See original GitHub issue

Heyya, I’m wanting to create a waveform based on a video selected by a user. We currently have this in iOS with the built in classes and it runs in about 0.33sec, however in our implementation on Android (non-LiTr) it is taking upwards of 11.41sec (it was 18.91sec before i added some threading into it 👀).

There appears to be a lack of info around how to do this efficiently on Android. I came across LiTr and was curious if its internal code is much better than mine and could be of more use.

Some background into my current Android code. Current code is in Xamarin.Android. This means I am using C# for my code so some class/method names may appear slightly different. For this usecase though the performance issues I am seeing should not be related to not using vanilla Java+Android. In my current code I am using a MediaExtractor + MediaCodec Decoder, selecting only the audio track and then in the OnInputBufferAvailable queuing that input buffer on the MediaCodec which then fires the OnOutputBufferAvailable. From here I can get the output buffer and loop over it to extract the shorts for me to parse and create a waveform.

At this point I am not even 100% that what I have is the raw audio data or if its just AAC data (eitherway it generates a the same waveform, so… yay?). The iOS code works very differently. You tell it what output format you want, LinearPCM, 16bit, not as a float, one channel. And then it fires and I parse the result data in the exact same way.

My thought here is if I can use LiTr almost as the example in the readme is.

mediaTransformer.transform(requestId,
                           sourceVideoUri,
                           targetVideoFilePath,
                           targetVideoFormat,
                           targetAudioFormat,
                           videoTransformationListener,
                           transformationOptions);

Where targetVideoFormat is null because I just want to pass it through and targetAudioFormat is mono 16bit linear PCM. I didn’t get very far with that because it’s essentially transcoding the file. I’d still need to open it and parse the audio a then do my waveform. Neat if it’s still faster than 11.41sec.

Looking around for PCM I only found this comment that mentions decode audio frame into PCM. It’s for a different purpose but same result. So I think what I need to do is to use the TrackTransform class.

If I were to go this route would I extract the data with a Renderer, or would I some how capture this data after the Encoder has encoded it as PCM linear 16bit audio? Any heap or guidance with extracting PCM data would be greatly appreciated.

As a bonus I actually have setup a binding library over here. After I’ve tidied it up and added some samples I’ll release it as a nuget package so any other Xamarin.Android developer can use LiTr just like they would if they included it with gradle 👍

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
beeradmoorecommented, Feb 17, 2022

@izzytwosheds an update on all of this.

The repo with full Xamarin.Android binding is up and available here, and the nuget package for any Xamarin.Android developer to use is available here. The nuget package for filters is its own seperate thing and it is here.

When an update comes through to this repo all I need to do is find and replace 1.4.18 with 1.4.19 in my repo, re-run the build nuget packages script and it will fetch updated aars and build both nugets for me to go upload. At some point I’d like to go through the entire demo app and convert it to Xamarin.Android and then try keep it as updated as possible as its a very useful library.

In terms of how this helped us out. Our app generates videoLength / 4 thumbnails and then choses closest to display. We could improve this by only generating what we need. The setup of FrameExtractParameters and VideoFrameExtractor actually lets us do that a lot more easier now.

For my demo video of 92 seconds (23 small thumbnails) our generation time went from 5.04sec down to 4.58sec. ~9% improvement. Not huge, but I’ll take it. This could be improved A LOT as in this current view we only actually display 3 images 😂

In terms of waveform generation we had HUGE improvements. We got it from 17.54sec down to 5.67sec. ~67% improvement. There may be more improvements to have here such as making the audio mono before we count samples, reducing bitrate/sample rate as that level of accuracy isn’t needed for our waveforms. I am sure there are other parts that can be optimised but that is what I have off the top of my head.

Thanks for the effort on your end to help this all happen 👍

1reaction
izzytwoshedscommented, Jan 26, 2022

Update: writing PCM audio into WAV file is coming soon.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Extracting raw audio/waveform from an MP3 - Stack Overflow
What I would like to do is extract the actual waveform/PCM of an MP3 file, so that I can play it using the...
Read more >
PCM Audio - COMPLETE GUIDE AND REFERENCE
PCM audio (Pulse Code Modulation) is a process that is utilized to convert analog audio signals that are represented by waveforms to digital ......
Read more >
send raw PCM audio or a waveform pattern to an output device
The audio stream is read, but no output occurs. -v, --verbose Increase the verbosity level. -q, --quiet Quiet. Suppress diagnostic messages. --aggressive Tries ......
Read more >
9 Best Audio Extractor to Extract Sound from Video - Geekflare
Here is a list of audio extractors that can easily extract audio from the video. Pazera Software. Pazera software is an easy-to-use and...
Read more >
Is There Any Ways To Create Pcm Waveform From A Large ...
createdAt 7 hours izzytwosheds issue comment linkedin/LiTr Extract PCM audio stream for waveform TransformationJob.transformTransformationJob.java:119 at.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found