Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Set `MALLOC_TRIM_THRESHOLD_` before interpreter start

See original GitHub issue

In https://github.com/dask/distributed/pull/6777 we reverted https://github.com/dask/distributed/pull/6681 in order to avoid https://github.com/dask/distributed/issues/6749 (which broke dask-cuda). This revert is meant to be temporary with the intent of restoring the functionality of https://github.com/dask/distributed/pull/6681, possibly addressing some of the implementation comments in https://github.com/dask/distributed/issues/6749. Opening up an issue so we don’t loose track of getting https://github.com/dask/distributed/pull/6681’s functionality back into main.

cc @crusaderky @gjoseph92 @pentschev @jakirkham who all engaged on the previously linked to issue / PRs

Issue Analytics

State:
Created a year ago
Reactions:2
Comments:7 (4 by maintainers)

Top GitHub Comments

1reaction

crusaderkycommented, Aug 4, 2022

if we’re willing to poke somewhat deep into internals.

I am extremely unwilling to poke in the Python internals. An upstream PR to CPython would be a good contribution, however I don’t think that the effort is justified just for our use case. Also we need this feature now and it would not be able to rely on it, at the earliest, before we require python >= 3.12, several years from now.

0reactions

gjoseph92commented, Aug 5, 2022

Yeah, though I’d love to see a proper fork/exec solution, I don’t expect to right now. We can get something pragmatic in much quicker.

Top Results From Across the Web

Run Python Applications Efficiently With malloc_trim

One approach to run Python web servers more efficiently is to pack as many processes of the application onto a single host (or...

Run Python Applications Efficiently with malloc_trim

The idea is to tweak when `mmap` or `malloc` are used by the Python interpreter. One allows memory to be released to the...

Memory Management — Python 3.11.1 documentation

Overview: Memory management in Python involves a private heap containing all Python objects and data structures. The management of this private heap is ......

malloc_trim(3) - Linux manual page - man7.org

The malloc_trim() function attempts to release free memory from the heap (by calling sbrk(2) or madvise(2) with suitable arguments).

main_thread_malloc.cpp - Apple Open Source

Calling malloc_stats or mallinfo with DEBUG set will attempt to check every ... unused top-most memory to keep before releasing via malloc_trim in...