Add json validation when ingesting jsonl files for training

Can be some simple bash like:

ls *.jsonl > file_names.txt
while read f; do
  if cat $f | jq -e . >/dev/null 2>&1; then
    echo "Parsed JSON successfully: $f"
  else
    echo "Failed to parse JSON: $f"
  fi
done <file_names.txt

in the data dirs.

Goal is just to check that json files aren’t truncated somehow. Can also add as a step when building index file for JsonlDatasets.

[punitkoura]

Fixed by @KUNAL1612 , closing

[punitkoura]

@suchenzang So to confirm, the json validation here is just making sure we can read a jsonl file and has nothing to do with the actual content of the files?

I think we also want to ensure that each json line is properly formatted.