Merging path into data

We’ve already merged url into path: path is now a URI, being a file on a filesystem or a resource over http.

In data packages, we have a general concept where a thing is either inlined or referenced. This pattern exists for schema, licenses and other properties: the payload for the property can be directly inlined, or, pointed to with a file path or an http address.

The only property that deviates from this pattern across all the specifications is that which handles the very thing we are packaging: path is reserved for references to data, and the additional data property is reserved for the inlining of data.

Implementations need to handle this reference to/inlining of data anyway, so what valid reasons do we have to maintain the path/data distinction?

I propose we merge path into data, or, we do a better job of being explicit about why we need two properties, and what the specific behaviour is with them, especially if they both appear in a package.

Pros

• Consistent with the inlining/referencing pattern of other properties
• Don’t have 2 properties for the same concept (and therefore, don’t require awkward wording in the spec, and handling in code, of path or data, but not both, which one takes precedence, etc.)

Cons

• Some types of data will be strings. So, because we don’t have a scheme for our file paths when path is a file path, it could be hard to implement code that handles both file paths and inlined data (but honestly, not that hard).
• Gets complicated with path as array - how does an implementation distinguish between an array of URIs, and an array which is inlined data?

Solution

• use package:// as the scheme for file paths. We don’t want to use file:// as we are explicitly declaring that data sources must be nested below the location of the descriptor, so the semantics here are that package:// refers to that location as the root. This enables implementations an explicit hint on the type of URI, and, if the value of data is not a URI.
  • Has the added benefit of more clarity on our forbidding of absolute and relative parent paths
  • this works fine for simple scenarios. it does not work well for the case of path as array. An implementor can’t inspect the string to check the type of data. A workaround would be to check the first string in the array, but it feels like a great big hack by then.
    • still, if we are serious about this wording: “all files in the array MUST be similar in terms of structure, format etc. Implementors SHOULD be able simply concatenate the files together and treat the result as one large file.” then implementors will have to do some inspection on each string in the array anyway.

Closing

This all may be just too complex for implementations, but I think it is clearly better for publishers.

I still want to open this issue to record the info for future reference if needed, even if we probably will not try to “solve” this.

[roll]

Goodtables, jsontableschema.Table, tabulator etc use source (aka data) and various schemes to distinct between data types. Spliting on path and data on spec level complicates things for implementations not otherwise.

[pwalsh]

FIXED In #337