`dask.bag` JSONDecodeError when reading multiline json arrays

When using dask.bag to read json files I get a JSONDecodeError when the json in the file is multiline.

import json
import dask.bag as db

db.read_text('single-line.json').map(json.loads).compute()
[[{'a': 'b'}, {'c': 'd'}]]    

db.read_text('multi-line.json').map(json.loads).compute()
JSONDecodeError: Expecting value: line 2 column 1 (char 2)

Here are the example files look like:

• single-line.json

[{"a": "b"}, {"c": "d"}]

• multi-line.json

[
    {"a": "b"},
    {"c": "d"}
]

Is this a bug or is there something I’m missing?

Also worth noting that I can read the multi-line file using just the standard lib

with open('multi-line.json') as f:
    data = f.read()
    print(json.loads(data))

[{'a': 'b'}, {'c': 'd'}]

This is using Python 3.6.0 and dask 0.15.0

[TomAugspurger]

http://docs.dask.org/en/latest/bag-creation.html

So each partition of your bag is a stringified JSON document. I suspect that .map_partitions(json.loads) should work. If not, I’d recommend a stackoverflow question that has a minimal example of what you have and what you’re trying to achieve. Those questions are generally easier to find later than the bottom of a GitHub issue 😃

For dask.delayed: http://docs.dask.org/en/latest/delayed.html

[AlJohri]

wow thanks for the speedy responses! it is not line-delimited, unfortunately- it is a json blob per file. I’ll have to look into map_partitions and dask.delayed, I’m not familiar- transitioning from spark right now. if you could link to an example usage that would be helpful

a small example file would be:

{
  "url": "http://example.com",
  "headline": "",
  "byline": "",
  "timestamp": "",
  "html": "",
  "description": "",
  "keywords": "",
  "published_time": "",
  "modified_time": "",
  "topics": []
}