Joining on a nullable column

import pandas as pd

left = pd.DataFrame({"a": [1, 2, pd.NA]}, dtype="Int64")
right = pd.DataFrame({"b": [1, 2, pd.NA]}, dtype="Int64")

pd.merge(left, right, how="inner", left_on="a", right_on="b")
#       a     b
# 0     1     1
# 1     2     2
# 2  <NA>  <NA>

(Above is from 1.0.1 and master)

I think when joining on a nullable column we should not be matching NA with NA and should only be joining where we have unambiguous equality (as in SQL). Also worth noting that this is the same as what happens when we have NaN which also seems incorrect, so could be an opportunity to fix this behavior?

pd.merge(left.astype(float), right.astype(float), how="inner", left_on="a", right_on="b")
#      a    b
# 0  1.0  1.0
# 1  2.0  2.0
# 2  NaN  NaN

Expected Output

      a     b
0     1     1
1     2     2

cc @jorisvandenbossche

[TheGustafson]

Has there been any additional discussion on this? I came across this behavior in my work, and I was genuinely surprised that this was the default behavior of Pandas.

The merge: match_na solution with a deprecation warning that @kasuteru proposed is a great idea.

In the interim, a warning should be put into the merge documentation. This should be a high priority…

[jorisvandenbossche]

The issue is https://github.com/pandas-dev/pandas/issues/22491, but we apparently have many other related, possibly duplicate, issues (from a quick search): https://github.com/pandas-dev/pandas/issues/28522, https://github.com/pandas-dev/pandas/issues/22618, https://github.com/pandas-dev/pandas/issues/25138, https://github.com/pandas-dev/pandas/issues/7473 (we should probably clean that up a bit)

But in any case, it would be good to directly have the “correct” behaviour for the new nullable dtypes (where we have less concerns about backwards compatibility)