Unable to read in big csv file

This is most probably related to #7628

Given a large, 19G csv file, and I’m unable to read it using astropy only.

When using the defaults naively, I run into a Segfault pretty quickly (within a minute):

from astropy.table import Table
testd=Table.read('test_set.csv')

Opting out of the fast reader, and turning of guessing etc, it’s still running now after 50 mins, already eating up substantial amount of memory, so I’m killing it:

119752 bsipocz   20   0  204.2g 197.6g  14628 R  99.3 39.2  49:00.38 ipython   
117282 bsipocz   20   0   33.9g  29.7g  19440 S   0.0  5.9   9:48.62 ipython     

The second session above is the one where the file was read in using pandas. Converting that DataFrame to Table then of course works nicely.

[saimn]

So with a small fix I’m able to load the file (PR to come):

❯ python test.py
324.1676182746887 sec.
453653104 lines

Quite fast actually! The issue is that the array storing the column length uses an integer. This file has a lot of lines, and the array size gets too big for a 32 bits integer.

[ajtanskanen]

This is what trying to open test_set.csv on a Macbook Pro (2015, 16 GB, OSX 10.14) looks like:

Python(3685,0x11259b5c0) malloc: can’t allocate region *** mach_vm_map(size=18446744073608888320) failed (error code=3) Python(3685,0x11259b5c0) malloc: *** set a breakpoint in malloc_error_break to debug Segmentation fault: 11

That is a quite large malloc size.