question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to define missing values in a single field?

See original GitHub issue

There are cases where a missing value in a column is a valid value in another column. For example, https://data.gov.au/dataset/colac-otway-shire-trees/resource/bcf1d62b-9e72-4eca-b183-418f83dedcea has a missing value for Year_Planted as 0, but that might be valid in another columns.

From https://frictionlessdata.io/specs/table-schema/, it appears missing values can only be defined on the table level, so this case can’t be defined in table schema.

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:1
  • Comments:11 (10 by maintainers)

github_iconTop GitHub Comments

3reactions
ashepherdcommented, May 4, 2018

Just wanted to add our use case for the missingValues property on a field. @BCODMO provides data management support for individual researchers on short-term funding (1-3 yrs) from NSF. Our researcher’s data can sometimes contain short-hand values that can be joined with lookup tables after they come back from sea. These fields are usually things like species names or quality flags that they abbreviate to save time during collection. There could be a scenario where a missing data value of ‘nd’ in one column might be a valid data value in another column. Having missingValues with a string of nd at the table-level can lead to NULL values in a column where nd might be a valid value for an abbreviated species name or quality flag.

2reactions
rufuspollockcommented, Dec 12, 2017

@vitorbaptista originally they were per field but we switched to per resource. I think it could make sense to support both and not complex to do so but i’d appreciate thoughts form @pwalsh

Read more comments on GitHub >

github_iconTop Results From Across the Web

Defining missing values - IBM
To define null or blank values as missing for a string variable, enter a single space in the Discrete value field. Parent topic:...
Read more >
Pandas: Number of missing values in the single column of a ...
Write a Pandas program to count the number of missing values of a specified column in a given DataFrame. Test Data:
Read more >
Missing Values in R
A missing value is one whose value is unknown. Missing values are represented in R by the NA symbol. NA is a special...
Read more >
Missing Field Values
If the original field has a missing value, that missing value can be propagated to the internal field by setting the COMPMISS parameter...
Read more >
Dealing with Missing Values
If we want to recode missing values in a single data frame variable we can subset for the missing value in that specific...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found