Can't open paths beginning with ~

Plan

https://github.com/astropy/astropy/pull/13107#issuecomment-1104552457

Description

astropy.io.fits can’t open files when the path is given with a ~ representing the home directory, e.g. fits.getdata("~/data/file.fits"). Instead, the home directory must be spelled out, e.g. fits.getdata("/home/sam/data/file.fits"). This can be fixed by passing paths to os.path.expanduser, which should do the right thing across operating systems.

There was some discussion about expanduser in #10705, and it looks like the function is now in use for finding the Astropy configuration directory.

There is also precedent in Sunpy, where expanduser (as wrapped by pathlib) is used for paths passed to Sunpy’s Map object, so that sunpy.map.Map("~/data/file.fits") works. (Map is their general object for loading, manipulating and plotting data).

If there’s not a reason to avoid adding calls to os.path.expanduser, I’m happy to dig into the code and prepare a PR. I’m most interested in the FITS loader, but I might take a look at the rest of astropy.io. (The Github search tool doesn’t show any uses of expanduser there.)

Expected behavior

astropy.io.fits.getdata("~/data/file.fits") loads the file file.fits in the data directory within the user’s home directory.

Actual behavior

astropy.io.fits.getdata("~/data/file.fits") raises FileNotFoundError: [Errno 2] No such file or directory: '~/data/file.fits'

Steps to Reproduce

Attempt to load a FITs file in the user’s home directory, giving a path beginning with a ~ representing the path to the user’s home directory.

System Details

Linux-5.13.0-27-generic-x86_64-with-glibc2.31 Python 3.10.1 | packaged by conda-forge | (main, Dec 22 2021, 01:39:05) [GCC 9.4.0] Numpy 1.22.1 pyerfa 2.0.0.1 astropy 5.1.dev516+g20513ae79 Scipy 1.7.3 Matplotlib 3.5.1

[pllim]

Thanks again. Un-tilde next time! 😏

[svank]

I just took a shot at adding a call to expanduser to that bottom-level fileobj_open method you pointed out:

def fileobj_open(filename, mode):
    filename = os.path.expanduser(filename)
    return open(filename, mode, buffering=0)

This enables the ~ shorthand for fits.open, fits.getdata, fits.getheader, fits.info, fits.writeto, and HDUList.writeto (and surely others), without breaking any tests.

To test the behavior more fully, I hacked up the FitsTestCase class in io/fits/tests/__init__.py. Its data() method takes a filename and returns that filename joined with the absolute path to io/fits/tests/data, and it appears to be used throughout the FITS test cases. I had it instead return ~/{filename}, and I set my $HOME environment variable to that data directory. When I do this:

• 125 tests fail without the added expanduser call and pass with the added call. (115 raise a FileNotFoundError, the remaining 10 are from the tests for the fitscheck, fitsheader, and fitsinfo, where it looks like the underlying script is experiencing the same error)
• 51 tests fail for the expected reason that my crude hack broke them. I checked each, and mostly these failures get a path from that data() method and then open it directly with the builtin open(), rather than going through an io.fits function. These tests shouldn’t be expected to pass in this case.

If the io.fits tests exercise all the functions that read or write FITS files, I think this suggests that they all go through fileobj_open, and this one change would enable ~-abbreviated paths without breaking anything.

I’d advocate for supporting these ~-style paths as a nice convenience feature, especially since it appears to be that one-line addition (at least for the FITS module). If it sounds good, I’ll think about how to properly test it and prepare a PR.