Improve HGBT leaf values updates

Describe the workflow you want to enable

This issue references this TODO comment:

https://github.com/scikit-learn/scikit-learn/blob/be89eb75f250dc5a769281939ba01e570fb12ae1/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L65-L67

Describe your proposed solution

Some APIs (like std::nth_element) allows getting the median or a given quantile of a contiguous buffer in O(n) but it does need to mutate a data-structure to sort (it can be the buffer or another data-structure if using a Comparator).

Could it be used there?

Moreover, why can’t we compute the median in parallel here?

Additional context

Follows-up with discussions: https://github.com/scikit-learn/scikit-learn/pull/20811/files/4df17828e439ad09a7784e99cc3d8d956eb50fe0#r781036357

---

/cc @NicolasHug who might be interested in following this issue as he initially worked on the update and authored this comment.

[NicolasHug]

We can probably work around this “list of arrays of different sizes” issue by doing something similar to https://github.com/scikit-learn/scikit-learn/blob/5d7dc4ba327c138cf63be5cd9238200037c1eb13/sklearn/ensemble/_hist_gradient_boosting/_gradient_boosting.pyx (in particular the start/stop logic)

will force the whole HGBT Cython base to be C++.

I’m not sure what you mean by that, could you clarify? Do you mean that the Cython code will be compiled to C++ instead of C?

[jjerphan]

@jjerphan Does your profiling indicate other room for improvements?

We could at least remove the TODO comment hinting at this issue.

I think the best way to tell is to try optimizing it.

For what I recall of my exploration of the OpenMP parallel sections (namely GOMP_parallel, as shown above which are large portions of the profiling report) a few months, it seems that they can’t really be improved much (those are nogil-context and the instructions are simple and optimized there).