DEPR: Dropping nuisance columns in core.apply

Ref: https://github.com/pandas-dev/pandas/pull/41475, https://github.com/pandas-dev/pandas/issues/39993#issuecomment-926281200

To keep the API consistent between groupby / non-groupby UDF methods (apply, agg, transform), we should not allow silent failure in core.apply.

• transform_dict_like #43743
• agg_list_like - #43741

[rhshadrach]

@jbrockmendel - I believe I left this open because there was some opposition to the changes. I’m still behind the deprecation, and it seems like conversation has ceased. I’m going to close for now, and can reopen if anyone would like to continue the discussion.

[emirkmo]

I hope you can provide some guidance to us @rhshadrach on whether it is appropriate to open up an issue to track dropping of nuisance columns not just in core.apply but elsewhere as well. At least, this sort of silent dropping is extremely useful in many contexts and to my recollection was widely relied upon when Pandas first became popular back in 2012(?)… In the past, one would have often have to manually clean data each time, arduously, before loading it into say a numpy recarray. Pandas opened up the possibility to just start working with mixed data without necessarily investing a huge amount of time pre-cleaning, as was necessary using other existing python solutions.

It’s sad to see what was a selling point or at least widely relied upon functionality dropped, even though I understand that inconsistent blackbox behavior like this is heavily frowned upon. It’s still extremely helpful for data analysis contexts.