DOC: Breaking change for join, int to float type coercion

Pandas version checks

• I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.join.html

Documentation problem

The behaviour of join appears to have changed somewhere between pandas version 1.2.4 and 1.4.2.

If if perform an inner join on a float col with a int col then in 1.2.4 the result is ant int64 column. In 1.42 it’s a float64. The below test passes in 1.2.4 but fails in 1.4.2.

I couldn’t find any documentation of this behaviour although it seems reasonable it caused me a bit of pain to get to the bottom of and is a breaking change. Is this behaviour documented anywhere and if not could it be?

import pandas as pd
from pandas.testing import assert_frame_equal


def test_join_type_coercion():
    left = [
        {"id": 1, "name": "Chase"},
        {"id": 2, "name": "Sky"},
        {"id": 3, "name": "Marhsall"},
        {"id": None, "name": "Daring Danny X"},
    ]
    right = [
        {"id": 1, "color": "Blue"},
        {"id": 2, "color": "Pink"},
        {"id": 3, "color": "Red"},
    ]

    left_df = pd.DataFrame(left)
    right_df = pd.DataFrame(right)

    joined = (
        left_df.set_index("id")
            .join(right_df.set_index("id"), how="inner")
            .reset_index()
    )
    assert_frame_equal(pd.DataFrame([
        {"id": 1, "name": "Chase", "color": "Blue"},
        {"id": 2, "name": "Sky", "color": "Pink"},
        {"id": 3, "name": "Marhsall", "color": "Red"},
    ]), joined)

Suggested fix for documentation

I’m not too sure what the pattern is here but maybe a note to describe the change in behaviour and at what version it was introduced.

[guyrt]

Sorry for misreading that. This change happened between 1.2.5 and 1.3.0 and nothing in 1.3.0 change log would have obviously caused it. So, that’s the right place to include a note on the breaking change. I’m happy to put together right message and submit PR (doing a little investigation first to understand when the change happens - e.g. I know it only occurs when you use indices in the underlying merge).

[oxlade39]

The core problem is that numpy’s int datatypes do not allow NaN, but float does. The dtype is coalesced into a numpy float64. This is covered explicitly in https://pandas.pydata.org/docs/user_guide/gotchas.html#support-for-integer-na. See in particular the “NA Type promotions” section in second link.

I’ll defer to those with more experience in Pandas, docs, but this seems inappropriate to call out explicitly in the join documentation given it’s well covered elsewhere. I can see the argument for including in changelogs given it’s changed behavior. Are those ever edited post hoc?

How has this changed in pandas 1.2.4 vs 1.4.2? My point is really that the behaviour of join has changed in a breaking way. This (the change in behaviour) should surely be documented somewhere?

To hopefully add clarity. Although the None does cause the first DataFrame to coalesce the id to a float that’s unrelated to what I’m trying to demonstrate here. I could (and perhaps should) have typed out the ids as floats.

Either way, a join on a float with int used to result in an int, now it doesn’t.