JSONDecodingError while archiving a specific website

Describe the bug

I’m getting this JSONDecodingError on a specific website.

Steps to reproduce

1. Ran ArchiveBox with the “default” config (didn’t change the docker-compose.yaml file much, apart from naming networks differently)
2. Saw this output during archiving

u@h:~/docker/ArchiveBox$ echo "https://www.zdnet.com/article/new-simjacker-attack-exploited-in-the-wild-to-track-users-for-at-least-two-years/" | /usr/local/bin/docker-compose exec -T archivebox /bin/archive

Traceback (most recent call last):
  File "/bin/archive", line 136, in <module>
    main(*sys.argv)
  File "/bin/archive", line 98, in main
    update_archive_data(import_path=import_path, resume=resume)
  File "/bin/archive", line 106, in update_archive_data
    all_links, new_links = load_links_index(out_dir=OUTPUT_DIR, import_path=import_path)
  File "/home/pptruser/app/archivebox/index.py", line 61, in load_links_index
    existing_links = parse_json_links_index(out_dir)
  File "/home/pptruser/app/archivebox/index.py", line 108, in parse_json_links_index
    links = json.load(f)['links']
  File "/usr/lib/python3.5/json/__init__.py", line 268, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 5283 column 44 (char 271619)

[gjedeer]

Cool, and I’ve just extracted individual archived page URLs from $subdir/index.js, removed the currupted main index.js file and archived the URLs again - fortunately, they were still available.

BTW, there was no power loss or anything like that. I’ve seen in the sources that you’ve had in-place JSON changing code, yeah, agree that SQLite is a better solution.

[pirate]

Closing this in favor of #234