Proposal: Table Dialect Spec
See original GitHub issueOverview
For now, we have:
- a resource describes a data entity
- a resource can be tabular
- a tabular resource has
schema
property that must beTabular Schema
- a tabular resource has
dialect
property that must beCSV Dialect
- a tabular resource has
It means that we have only two mechanisms to add tabular information to the resource: schema
and dialect
properties:
schema
: what is the datadialect
: how to extract the data
Maybe at some point this list can be extended e.g. providing table filtering ability etc but, as for now, I think we definitely can generalize the dialect
property. Instead of having it csv-only
we can have a general Table Dialect
spec helping describe any tabular format details.
The proposed Table Dialect
spec will create a nice symmetry with already existent Table Schema
spec. Here is a quick overview of the proposal. The spec is hierarchical so e.g. Csv Table Dialect
inherits all the props from Table Dialect
.
Table Dialect
Core Table Dialect
spec will handle header
management.
header (bool)
default: true
Whether the table has a header row(s)
headerRows (int[])
default: [1]
An array of header row numbers. Can describe a multiline header.
headerJoin (str)
default: ’ ’ (one space)
A string to concatenate a multiline header. Has no effect for a single row header.
Csv Table Dialect
@amercader hints that we also need to re-review the CSVW spec in case we miss something - https://www.w3.org/TR/2015/REC-tabular-metadata-20151217/#dialect-descriptions
It will support all the header
options and the options below which is standard for csv
.
delimiter (str)
default: ,
lineTerminator (str)
default: \r\n
quoteChar (str)
default: “”
doubleQuote (bool)
default: true
escapeChar (str)
default: not set
nullSequence (str)
default: not set
skipInitialSpace (bool)
default: false
I propose the following changes to the current Csv Dialect spec:
- make
skipInitialSpace=False
by default to sync with Python/Pandas/JS/etc behaviour - remove
caseSensitiveHeader
as I guess it should be an option for someinfer
function but for general data description I’m not sure what it does - review
commentChar
option as partially its role will be handled byheaderRows
and, at the same time, there is more functionalskipRows
supported by the software. In software, I’ve moved all theskip/pick/limit/offset_fields/rows
functionality to a separate group calledTable Query
(orTable Discovery
previously) which should probably exist only in software because we don’t want to make ETL from the specs, although I think there are options to consider.
Excel Table Dialect
It will support all the header
options and:
sheet (str|int)
default: 1
String or integer to address an excel sheet e.g. 2
or Sheet 2
.
Options to consider:
- fillMergedCells
- preserveFormatting
- adjustFloatingPointError
Json Table Dialect
It will support all the header
options and:
keyed (bool)
default: false
Whether a source is keyed i.e. an array of dictionaries instead of an array of arrays.
keys (str[])
default: not set
For a keyed source, an array of keys to use as a header row.
Options to consider:
- property (path to the data within json e.g.
dogs/data
)
In conclusion, the idea is:
- csv is not the only tabular format; let’s describe others, the most importantly Excel
- to have one hierarchical spec which will help standardize different formats’ dialects
- new formats and properties addition should be considered based on users’ demand and should happen gradually
Issue Analytics
- State:
- Created 3 years ago
- Reactions:3
- Comments:7 (7 by maintainers)
Top GitHub Comments
Json Table Dialect requires further discussion as many ways exist to encode tabular data in JSON.
See https://www.w3.org/TR/csv2json/ (CSVW) for an example of a specification that supports 1 (simple) and 3 (slightly complicated). A simplified form of this uses an object with property
rows
for rows and propertycolumns
with an array of objects, each having propertylabel
at least.So
keyed
is probably fine butkeys
is more complex.Moreover cells in JSON Tables and Excel Tables can have data types other than plain strings.
Datatypes can be defined with columns as done in CSVW but less complex (e.g. only string, number, logical).
Thanks, @nichtich!
I think it should not be a blocker as in specs like this we have a privilege to start from a small core and extend once other properties are discussed and justified