question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to set data type to whole Table.

See original GitHub issue

Hello tablesaw team. I’ve question regarding setup of data type for the Table if I don’t know the amount of colums. I have 30_000x2000 feature csv file with 0.0 and some other amount of Double numbers. If I call csv parsing via:

CsvReadOptions options = CsvReadOptions.builder(csv)
                    .header(false)
                    .maxNumberOfColumns(50_000).build();

Table t = Table.read().csv(options);

I got Number format exception, as all 0.0 number are treated as Short 0. So when reader gets to real numbers like 13.5 if throws NFE.

But if I add sample(false) to reader options if takes about 2:40 to parse such file.

How can I setup data type for whole Table, as far as I can see only by setting columnType in parser option, but it’s won’t work as I don’t know a number of columns on csv file?

P.S. I used com.univocity.parsers.csv.CsvParser separately to read the same file so it takes 2:20 for parser.parsAll and 1:20 for parsing file by row.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:1
  • Comments:17 (8 by maintainers)

github_iconTop GitHub Comments

2reactions
Ebalaitungcommented, Nov 21, 2018

Yes, exactly!

0reactions
lwhite1commented, Jan 25, 2022

if you’re reading a CSV file, there is an option in CsvReadOptions that lets you specify the types to be used when you read the file. If you just provide one type (e.g. STRING), they would all be the same. IDK if this is implemented for Excel, however.

On Tue, Jan 25, 2022 at 9:28 AM Wesley @.***> wrote:

I know this is closed, but I’m going to plug for this functionality anyway. It doesn’t seem “too specialized” at all.

The suggestion to “read a table then copy and paste its types” is far from programmatic and becomes impossible to automate. If we were all just working in Jupyter notebooks, I’d not be complaining but I’m trying to develop software.

— Reply to this email directly, view it on GitHub https://github.com/jtablesaw/tablesaw/issues/415#issuecomment-1021237092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2FPAVCJDTZD2W7ORSQVMLUX2XPBANCNFSM4GFDZ2NA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

Read more comments on GitHub >

github_iconTop Results From Across the Web

Modify or change the data type setting for a field
Convert to this type From this type Changes or restrictions Text Memo Access deletes all but the first 255 characters. Number No restrictions. Currency No restrictions....
Read more >
SQL queries to change the column type
This article will show the way to change the data type of the columns ... We can use ALTER TABLE ALTER COLUMN statement...
Read more >
How to Select the Right Data Types | Tutorial by Chartio
The basic strategy for selecting the best data type is to select the smallest data type that matches the kind of data you...
Read more >
ArcGIS field data types
When you create feature classes and tables, you select a data type for each field. The available types include a variety of number...
Read more >
Data Types - Tableau Help
Change the data type for a field in the Data Source page · Click the data type icon for the field (as shown...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found