question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

It's impossible to parse CSV file with more than 512 columns

See original GitHub issue

I try to parse feature index file into Table (it’s about 2000x36000 matrix). I try to use: Table t = Table.read().csv(csv); but got error from com.univocity.parsers.csv.CsvParser (log is attached 512-ERROR.txt) that there is more than 512 columns. I’ve checked sources and found out that you create CsvParser with default settings, and there is no option to change some how this settings from your API.

I think there should be some way to change com.univocity.parsers.csv.CsvParserSettings from Table.read() place, or at least change columns size limit.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:7 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
benmccanncommented, Sep 26, 2018

I wonder if we should just hardcode parserSettings.setMaxColumns(Integer.MAX_VALUE). I don’t really understand why there would be a column limit

0reactions
Ebalaitungcommented, Nov 8, 2018

I’ve create PR for default value of 10_000

Read more comments on GitHub >

github_iconTop Results From Across the Web

Unable to import csv file · Issue #3404 · sequelpro ... - GitHub
The CSV was read as containing more than 512 columns, more than the maximum columns permitted for speed reasons by Sequel Pro.
Read more >
How to import csv files with massive column count into Apache ...
I'm running into a problem importing multiple small csv files with over 250000 columns of float64 into Apache Spark 2.0 running as a...
Read more >
Easy way to import messy CSV files into Excel The ... - YouTube
If you constantly need to deal with ugly CSV files that do not split the data into columns when you open them in...
Read more >
5 most common parsing errors in CSV files (and ... - Medium
#2 — Text field with an unescaped delimiter ... If the column separator appears unescaped in a text field, this will cause the...
Read more >
CSV not parsed into columns despite Get Data delimiter set
Yes, I see. On preview stage Excel make some guesses how to transform your data, not always correct ones. It's always better to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found