CSV to JSON is too fragile

The CSV to JSON script is too fragile and breaks on real-world data such as double-quoted columns to contain commas and double-double-quotes for literal double-quotes.

I had a \r in some of my data which means the script isn’t standardizing to \n line endings.

Test:

Result:

Expected:

The JSON to CSV gets similarly confused in that it uses \" for literal double-quotes instead of the CSV convention of double-double-quotes within a quoted column.

[Flare576]

For what it’s worth: I grabbed the minified version of Papa Parse and pasted it straight into my csv_to_json.js, and it did work. I did have to take out the require and fix the code to be

function main(state) {
  try {
    const { data } = Papa.parse(state.text);
    state.text = JSON.stringify(data);
  }
  catch(error) {
    state.postError("Something strange happened here...");
  }
}

If the pattern Boop wants to follow is to have all dependencies in the lib folder, then creating a new file in there for Papa (with its LICENSE, of course) is probably the cleanest way, but before I throw up a PR I wanted to double-check that this is the approach you want.

[Flare576]

Sorry for the double-post, but I just realized that there’s a combination of an assumption and a “Limitation” at play around the “header” stuff in this script.

First, we’re assuming that this data has a header (both in the original and in my fix). There’s a “Limitation” (and I use that term VERY loosely because I actually see this as a feature/simplicity) in Boop that you can’t pass parameters to the scripts.

So… the solutions I see are:

1. Add another script and define one as CSV to JSON (headers), the other CSV to JSON (headerless)
2. 🤮 Add some sort of design where the first line of state.text could be settings for a script 🤮
3. ❓ Auto-detect header rows ❓ (I have NO idea how to do this and am pretty sure it’s a fool’s errand)

If we went with # 1, I’d argue that the 2nd script would just be an optional script, and the default would be headers