Support separate but local data files via data.url property for Vega-Lite

Thank you very much for supporting Vega and Vega-Lite. I works nicely.

I have a specific request regarding the location of the visualized data:

I prefer keeping the visualized data in a separate and local local file (instead of inlined in the diagram specification) via the “data.url” property (https://vega.github.io/vega-lite/docs/data.html#url), because it can be smaller (e.g. CSV instead JSON), can be edited easier, there is a better separation of concerns…

Example: I prefer

{
	"data": {
		"url": "data.csv"
	}
}

over

{
	"data": {
		"values": [
			{
				"a": "2020-01-05",
				"b": 0.3,
				"c": "C1"
			},
			{
				"a": "2020-01-15",
				"b": 0.7,
				"c": "C1"
			}
		]
	}
}

A separate data file is currently not possible with “asciidoctor-kroki” (It works when using Vega-Lite directly.), because an error is created:

Skipping vegalite block macro. No such file: https://kroki.io/vegalite/svg/....

I suppose, that the reason for this error is, that the referenced file containing the data (i.e. “data.csv”) is not uploaded to kroki. One “workaround” could therefore be to make “data.csv” publicly available:

{
	"data": {
		"url": "http://.../data.csv"
	}
}

But this does not work either (It works when using Vega-Lite directly, i.e. without “asciidoctor-kroki”). Even if this remote URL would work, it would have the disadvantage the file must be available before the asciidoctor generation with :kroki-fetch-diagram: true.

I created a public repo where you can find all three variants: https://gitlab.com/winni/asciidoctor-kroki-vegalite

The generated result can be found here: https://winni.gitlab.io/asciidoctor-kroki-vegalite/test.html

I don’t know if there is a feasible solution to support local data files and how much effort it would be to implement.

Perhaps the “workaround” with remote URLs can be implemented easily (or could it cause a security problem)?

I don’t know if this is a very special use case in combination with Vega-Lite, or if this or similar requirements would make sense for other types of diagrams.

What do you think?

[stenzengel]

I think that’s not an issue because you can inline the data, see: https://vega.github.io/vega-lite/docs/data.html#inline

I didn’t know that. That’s cool!

I don’t want to implement a multiparts form request because:

* It won't work with GET requests

* The API will be more complicated

* I don't want to handle files server-side (the less we rely on the disk the better)

I’m convinced. Your proposition is the right way to go.

It’s not really a security concern even though I want to disallow the containers to write on disk (because they don’t have to). I just don’t want to do additional works on the server-side.

Yes, KISS.

I strongly believe that a diagram library should not care about reading/including files. It’s a nice feature but they should take an input and produce an image, that should be their main focus. Fortunately, Vega-Lite is well-designed and it’s possible to inline data so all is good 👍

Yes and I agree that Vega-Lite is very nice.

Indeed! @stenzengel If you want to help, you can try to implement what I described above:

* parse Vega/Vega-Lite input (I think they are using json5)

* extract `data.url`

* read the file content

* replace `data.url` with the inline data `data.values`

For PlantUML the approach is similar, expect we are looking for include directives. We are using an interface (called Virtual File System) to read files from different context (Antora, Browser, Node) so you will probably need to use the vfs.read function to get the file content.

Let me know if you have any questions.

I’ll try and I will certainly have questions. Thanks for your offer to help.

[Mogztter]

I suppose, that the reason for this error is, that the referenced file containing the data (i.e. “data.csv”) is not uploaded to kroki. One “workaround” could therefore be to make “data.csv” publicly available

Kroki does not fetch data from local or remote location for security reasons. I’m not a security expert but I believe that it’s a major security concern to load arbitrary data from untrusted origin.

I think it’s similar to my request for PlantUML and @Mogztter also mention Vega’s data: #49 (comment) So it’s disabled for security reasons.

Yes you are absolutely right.

I like the idea of a preprocessing task that can fetch the data or include from remote location (http) to local “cache” or just use the local file and then embed (replace) this in the original file.

Indeed, I think that’s the right way to solve this issue 👍