Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to explain these memory behaviors when using numpy?

See original GitHub issue

When running a very simple script based on numpy functions, we can get the following results:

test2.py: % of CPU time = 100.00% out of   3.59s.
  	 |     CPU % |     CPU % | Avg memory  | Memory      |
  Line	 |  (Python) |  (native) | growth (MB) | usage (%)   | [test2.py]
--------------------------------------------------------------------------------
     1	 |           |           |             |             | import numpy as np
     2	 |           |           |             |             |
     3	 |     0.30% |    48.40% |         -80 |       1.03% | x = np.array(range(10**7))
     4	 |     0.59% |    50.72% |           0 |      98.97% | np.array(np.random.uniform(0, 100, size=10**8))
     5	 |           |           |             |             |

How can we get:

A negative memory growth for the first line?
A null memory growth on the second line?

System info

Platform : Mac OS X
Python: 3.7 (brew)
Numpy: 1.16.4

Issue Analytics

State:
Created 4 years ago
Comments:18 (12 by maintainers)

Top GitHub Comments

1reaction

emerybergercommented, Mar 3, 2020

Memory usage: ▁▁▁▁▁▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ (max: 1542.64MB)
phylliade/test2-3.py: % of CPU time =  89.31% out of   2.42s.
  	     |     CPU % |     CPU % | Avg memory  | Memory      | 
  Line	 |  (Python) |  (native) | growth (MB) | usage       | [phylliade/test2-3.py]
--------------------------------------------------------------------------------
     1	 |           |           |             |             | import numpy as np
     2	 |           |           |             |             | 
     3	 |           |           |             |             | @profile
     4	 |           |           |             |             | def main():
     5	 |           |           |             |             |     #x = np.array(range(10**7))
     6	 |     0.83% |    88.48% |         777 | ▅▁▅         |     y = np.array(np.random.uniform(0, 100, size=(10**8)))
     7	 |           |           |             |             | 
     8	 |           |           |             |             | main()
     9	 |           |           |             |             |

In this example, you can see the increase and decrease over time, which to me is pretty suggestive that there is copying going on (from an old, reclaimed object – corresponding to the dip – into the newly allocated one).

1reaction

emerybergercommented, Mar 3, 2020

Just committed a version with memory usage as per-line sparklines (I wonder if printing a % beside it would be of value). Take a look! https://github.com/emeryberger/scalene/commit/037dbd236eb37b3564052aacd862fd05c8a40278

Top Results From Across the Web

Python memory usage of numpy arrays - Stack Overflow

I use this code to get a listing of all of them and their size. Not sure if locals() or globals() is better...

Internal organization of NumPy arrays

NumPy arrays consist of two major components: the raw array data (from now on, referred to as the data buffer), and the information...

The Basics of NumPy Arrays | Python Data Science Handbook

Attributes of arrays: Determining the size, shape, memory consumption, and data types of arrays; Indexing of arrays: Getting and setting the value of...

Numpy Load, Explained - Sharp Sight

This tutorial shows how to use Numpy load to load Numpy arrays from stored npy or npz files. It explains the syntax and...

Introduction to NumPy - W3Schools

NumPy arrays are stored at one continuous place in memory unlike lists, so processes can access and manipulate them very efficiently. This behavior...