CSVParser - "OutOfMemoryError: GC overhead limit exceeded" with large file
See original GitHub issuemy file has about 20,000,000 rows X 6 cols.
main code:
Table user = Table.read().csv("filename.csv");
out("Table name: " + user.name());
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.String.substring(String.java:1969)
at com.opencsv.CSVParser$StringFragmentCopier.peekOutput(CSVParser.java:609)
at com.opencsv.CSVParser$StringFragmentCopier.takeOutput(CSVParser.java:616)
at com.opencsv.CSVParser.parseLine(CSVParser.java:421)
at com.opencsv.CSVParser.parseLineMulti(CSVParser.java:309)
at com.opencsv.CSVReader.readNext(CSVReader.java:338)
at tech.tablesaw.io.csv.CsvReader.detectColumnTypes(CsvReader.java:430)
at tech.tablesaw.io.csv.CsvReader.read(CsvReader.java:161)
at tech.tablesaw.io.DataFrameReader.csv(DataFrameReader.java:59)
at tech.tablesaw.io.DataFrameReader.csv(DataFrameReader.java:55)
at tech.tablesaw.io.DataFrameReader.csv(DataFrameReader.java:35)
at ReadCSV.main(ReadCSV.java:7)
when I set JVM: -Xms2g -Xmx2g -Xmn512M .
problem exits still.
ps. my laptop has 12g memory. my Java jdk version is 8.
Issue Analytics
- State:
- Created 6 years ago
- Comments:10
Top Results From Across the Web
GC overhead limit exceeded while read from CSV file
I am getting error on the method : csvReader.readAll() as mentioned above. I am not sure what is the problem which the code,...
Read more >java.lang.OutOfMemoryError: GC Overhead Limit Exceeded
Simply put, the JVM takes care of freeing up memory when objects are no longer being used. This process is called Garbage Collection...
Read more >"[java.lang.OutOfMemoryError: GC overhead limit ... - ERROR
OutOfMemoryError : GC overhead limit exceeded]" when loading the model files for probabilistic labeling and parsing operations fails ...
Read more >113. ModelSegment error Exception in thread main ...
OutOfMemoryError : GC overhead limit exceeded at java.lang.AbstractStringBuilder.(AbstractStringBuilder.java:68) at java.lang.StringBuilder.
Read more >GC overhead limit exceeded while processing CSV file and ...
OutOfMemoryError : GC overhead limit exceeded while processing CSV file and upsert to Sal. Hi,. I'm trying to read csv file which is...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
it might be easier to get OpenCSV to fix the bug, if it is theirs, than to replace the CSV parser.
On Thu, Dec 7, 2017 at 4:42 PM, SALES notifications@github.com wrote:
fixed.