Librosa in 128MB memory or less

I want to package librosa onto AWS Lambda (a serverless compute service) to do audio analysis (similar to EchoNest API) cheaply and scalably. To minimize cost and maximize performance, I want to use as little memory as possible. 128MB is ideal. But I also want to support the analysis of DJ mixes that are an hour+ long.

If I partially transcode->STFT->etc n samples at a time, I can still do mostly what I need: HPSS, cqt, transient-segmentation, timbre, pitch, or time stretch.

Let’s say the input is a URL to an audio file (could be MP3, M4A/AAC, WAV, FLAC)

1. Partially download n bytes
2. Send input through the transcode->STFT->etc pipeline
3. del bytes for garage collection when no longer needed
4. Print output as I go (instead of gathering into a dictionary)
5. Hop to next bytes

Byte range needs to take into consideration the HPSS values (and segmentation boundaries) at frame f depend on a few frames before and after it.

Aware of anything stubborn/tricky about transcoding partially downloaded files?

Recommendation on how I might best structure/integrate this into librosa? Perhaps something in examples/

Recommendations for memory profilers?

[misrasaurabh1]

How did you even make librosa even work on AWS lambda? How did you package the linux libav-tools dependency on Lambda? Also the limit of code one can package is 250MB on AWS lambda. librosa along with all its dependencies take up close to 300MBs in python packages.

[Jasparr77]

@yousefjnm Nothing yet, I’m working through a few issues specific to my use-case. I would recommend trying Zappa out, though - it’s a pretty straightforward solution and the slim_handler feature leverages s3 to “support up to 500M of zipped up Python projects.”