Support for `memmap` in `Quantity`
See original GitHub issueDescription
I if possible, would like the Quantity
object to support memmap
arrays, meaning that given a memmap
array, a quantity constructed from that array remains memory mapped.
Additional context
It is possible that Quantity
supports this, but it remains unclear. Right now if I run:
import numpy as np
import astropy.units as u
a = np.memmap('test', mode="w+", shape=(3,4))
q1 = u.Quantity(a, u.kg)
q2 = u.Quantity(a, u.kg, copy=False)
print(f'{isinstance(a, np.memmap)=}')
print(f'{isinstance(q1, np.memmap)=}')
print(f'{isinstance(q1.value, np.memmap)=}')
print(f'{isinstance(q2, np.memmap)=}')
print(f'{isinstance(q2.value, np.memmap)=}')
It returns
isinstance(a, np.memmap)=True
isinstance(q1, np.memmap)=False
isinstance(q1.value, np.memmap)=False
isinstance(q2, np.memmap)=False
isinstance(q2.value, np.memmap)=False
Which from the memmap
documentation indicates to me that the the data is no longer memmap
.
Basically, I would like a way to pass a memmap
array to a Quantity
and that Quantity
to use the array data through the memmap
(ie the data on disk mapped via the memmap
to the array gets updated when I update the Quantity
). This includes a way to interrogate the Quantity
to confirm it is or is not using a memmap
.
Issue Analytics
- State:
- Created a year ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Support operations necessary for memmap #832 - GitHub
Hello, I've started using fsspec for https://github.com/criteo/autofaiss and it works well. The use case is to read embeddings stored as ...
Read more >numpy.memmap — NumPy v1.24 Manual
Create a memory-map to an array stored in a binary file on disk. ... If mode == 'r' and the number of remaining...
Read more >mmap(2) - Linux manual page - man7.org
The mmap() call doesn't fail if the mapping cannot be populated (for example, due to limitations on the number of mapped huge pages...
Read more >mmap()--Memory Map a File - IBM
The mmap() function is only supported for *TYPE2 stream files (*STMF) existing in the "root" (/), QOpenSys, and user-defined file systems.
Read more >numpy memmap memory usage - want to iterate once
Fortunately, I recently found a solution that will allow you to iterate through the entire memmap array while capping the RAM usage. The...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Indeed, not immediately. I briefly tried whether it would work to define a new class
QuantityMemmap(u.Quantity, np.memmap)
but that in itself did not seem good enough. I have not investigated why, though think in principle it should be possible to get it to work; it might need the kind of automatic class construction that was used forDistribution
andMasked
and which @nstarman has been wanting to generalize. With that,Quantity
might also be able to wrap dask arrays, etc.In principle, it should be possible to make this work, since
np.memmap
is anndarray
subclass, but right now our use ofndarray
is perhaps a bit too much built in. The suggestion you got from the documentation does work: