Add permutation based feature importance?

I think adding permutation based feature importances would be cool: https://link.springer.com/article/10.1186%2F1471-2105-8-25

There is a python package that does it for our random forests with bagging: https://github.com/parrt/random-forest-importances

But I’d rather like to see a generic permutation based importance score with cross-validation or hold-out. I think this would be great analysis tool.

One easy way to implement it would be to provide a function for plotting etc. But Ideally we’d be able to use it in feature selection, I think, so we’d need to create a meta-estimator PermutationImportanceCV that only provides feature_importances_, I guess, so it can be wrapped with SelectFromModel?

[amueller]

Also see this blog post: http://parrt.cs.usfca.edu/doc/rf-importance/index.html

I’m not suggesting we change our RF feature importances, but having a more expensive but possibly higher quality alternative would be great.

[VineethKanaparthi]

I am waiting for this method to be included. We need reliable results more than anything even if it’s not fast. We prefer not be wrong anytime. I implemented it myself for my model.(for a competition)