[epic] Reintroduce readable style for specs

In v1-alpha upgrade (dec-jan) we did a big-bang change to layout and generation of specs (and also moved from jekyll to lektor).

Though bringing various improvements this change also made most specs much less readable in a variety of ways e.g. much less readable intros, repetitive property definitions (e.g. JTS). It also made specs very hard to edit as the bulk of content is now spread over multiple partial files and the core content is auto generated from a definition dictionary.

Furthermore, this change was coincidental with a substantial set of improvements and refactoring of the content of the specs e.g. creation of Data Resource spec, introduction of generic URI concepts.

Disentangling these two changes is very hard. Thus the approach we are taking here is:

• Revert all specs to pre-v1-alpha - Dec 15 4474b04969bca255a87009b401bb6a49313e4894
  • Though using new file names and adapting to new structure where possible
  • Rework Data Resource in line with those specs
• Reintroduce substantive changes to all specs
  • TODO: identify these
• Remove those we don’t want e.g. use of JSON pointers (this could be part of previous item)
• Introduce new changes e.g. the dedicated property for inline data on data resources #414

Tasks

• Fork off old template and create new one so we can incrementally switch specs over
• Revert Table Schema spec
• Refactor Data Resource spec
  • Refactor Tabular Data Resource spec
• Refactor Data Package spec
• Refactor Tabular Data Package spec
• Refactor csv dialect (?)

Things to reintroduce

• Data Package
  • Reintroduce id field for DP: c080f69470d5166cc0e3263a2c555762752f1d11
• Reintroduce profile property (do we want to do this - or should this be a separate mini-spec)
• Table Schema
  • missingValue => missingValues
  • constraints below types (??)
  • gyear => year, and gyearmonth => yearmonth
  • explanation about duration reads really weirdly
• Data resources
  • Schema references for data resources need to be clarified / improved

At end

• Re-add appendix style list of all properties (like current specs)
  • QU: to do that we’d need to get convergence again between dictionary list of properties and list in the specs.
• format blocks throughout with ```javascript

Can ignore as not using the new spec structure:

• Data Package Identifier
• Fiscal Data Package

Outstanding Questions

• Should we move profiles out to their own spec (??)

[rufuspollock]

Here’s the summary of changes done for v1 alpha in #337 and their status now:

Substantive changes for v1 alpha

• Integrate our JSON Schemas as a single point of truth for everything common across all specs.
  • Much, much less ambiguity!
  • Direct alignment between spec and implementation!
• Standardise all language across all specs (prime example: field, attribute, property are used interchangeably)
• Remove author in preference for contributors with roles. Currently two roles: author and maintainer
• Change license type to license name
• Remove optional fields wording in favour of simpler required and recommended (NO)
• Rename JSON Table Schema to Table Schema #334
• (?) Stabilize for v1 - #330
• Make Data Resource name required (and unique) - #309 AGREED
• Readability of code blocks #318 AGREED
• Remove data dependencies - #341 AGREED
• Template for writing a data package spec #196 - REDO or INVALID
  • Create spec as a structure for producing RFCs #328
• Data package profiles #87 - AGREED to restore (somehow)
• Split out resource spec - #327 (duplicate ??)
  • PR for Data Resource spec - #335
• Miscellaneous revisions e.g. license -> licenses, created #298 CHECK
• Referencing schemas #325 SORT OF (no json pointer now, but we have removed schemas property on dp.json)
• JSON references / JSON Pointer #295 AGREED (Going)
• language around csv delimiter #340 AGREED
• no license, only licenses array #236 AGREED
• licenses is array of objects, and uses URL and not license identifiers #146 AGREED TODO
  • switch to URL and
  • Question: do we allow relative paths (like resource paths??)
• remove data package references from foreign keys #314 AGREED
• missingValues default value #359 AGREED
• missingValue -> missingValues #353 AGREED
• Copy editing and examples #356 AGREED
• Clarify constraints for each type #296
  • Explicitly describe, for each constraint, the types it applies to, and, that all constraints apply to cast values. esp. clarify wording around date types
• Merge path into data #336 - WONTFIX
• Removes the need for a seperate schemas repository - everything is generated here in a single place, including the registry
• Makes the registry itself a JSON file. Users still contribute to the CSV, but consuming code uses JSON

[rufuspollock]

@roll profile is back 😄