ENH: Add set of tests to type check the public APIs

From discussion on pandas typing meeting on January 7, 2022 (notes at https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-licoBmBU/)

Follow up to discussion here: https://github.com/pandas-dev/pandas/issues/28142#issuecomment-991946409

Goal is to have a set of tests that make sure that users of the pandas API writing correct code work with whatever type stubs we publish.

Example of an approach that is suggested was developed within the pandas-stubs project based on various code snippets which are then tested with mypy : https://github.com/VirtusLab/pandas-stubs/tree/master/tests/snippets @zkrolikowski-vl and @joannasendorek from that project have offered to help (nudge, nudge)

Ideally we would add this testing framework to the CI before starting to bring over the stubs.

[Dr-Irv]

I really like the idea of testing pandas with its own annotations and pandas-stubs, but I’m honestly not sure whether pandas-stubs can be used to check pandas without having pandas-stubs-specific ignore comments in pandas (or some patch/sed magic).

I think we are not talking about using the pandas-stubs to test pandas source. I think the idea is that we make sure that changes to pandas-stubs work with both released versions of pandas, and in-development versions of pandas, and, conversely, that changes to pandas in development don’t break the stubs. The stubs are tested with mypy, pyright, but also pytest, so you could imagine a change in pandas breaking a pytest test.

Instead of checking pandas with pandas-stubs, it might be an option to somehow (don’t ask me how) enforce that pandas-stubs only contains annotations where different annotations are intended (for example __getitem__) and functions that are not annotated in pandas. Functions that have suitable annotations in pandas could simply(?) be imported from pandas (if pandas is a dependency of pandas-stubs).

I don’t think that you can have some stubs in a pandas-stubs package, and the rest in the source code. Not sure the type checkers will manage that.

[Dr-Irv]

@simonjayhawkins Yes. AFAIK pytest will pick them up as well. That’s the way we’ve been testing if the snippets execute correctly.

So the only issue here is that we will probably add tests for typing to test if a type is too wide, and put a #type: ignore comment on them, but the code will then execute with pytest, and raise an exception, which means if pytest is run on it, we have to expect that exception. For example:

>>> s = pd.Series([1,2,3])
>>> s.corr(pd.Series([4,5,6]), method="foobar")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\envs\b38np122try2\lib\site-packages\pandas\core\series.py", line 2512, in corr
    raise ValueError(
ValueError: method must be either 'pearson', 'spearman', 'kendall', or a callable, 'foobar' was supplied

So when we type Series.corr() to have method be a Union[Literal["pearson", "spearman", "kendall"], Callable], then you want the typing test to test that foobar is invalid for method. To do that you’d do something like

 s.corr(pd.Series([4,5,6]), method="foobar") # type: ignore

so that the type checker was happy, but then pytest will fail until you surround the code with pytest.raises types of things.

Maybe that means we should have a pandas/tests/typing/valid section - which contains valid code that is tested with pytest and the type checker, and a pandas/tests/typing/invalid section where we don’t run pytest. Then in the files in the invalid directory, have something in there that just skips the tests.