cdx format, includes json

Dear helgeho,

The common crawl its cdx files have a different structure then you library expects. au,com,canberratimes)/business/mining-and-resources/bhp-on-the-hunt-for-new-chief-ft-20121106-28x2j 20170820020911 {"url": "http://www.canberratimes.com.au/business/mining-and-resources/bhp-on-the-hunt-for-new-chief-ft-20121106-28x2j", "mime": "text/html", "mime-detected": "text/html", "status": "200", "digest": "KZDZMH5BCMZJOCZPQCRUOTAPYH2CYQVJ", "length": "35157", "offset": "509044848", "filename": "crawl-data/CC-MAIN-2017-34/segments/1502886105961.34/warc/CC-MAIN-20170820015021-20170820035021-00503.warc.gz"}

I will implement the support for this, since your library seems really promising. When it works i will send you a pull request.

Another question would it be a good idea to publish your library to mvn central? This would enable more people to use it.

[ikreymer]

FWIW, the format is simply called CDXJ, and there’s not yet an official standard for it… but essentially just keeping the url key and timestamp as header (for sorting) and using a JSON block for the rest of the fields… It should be easy to support in general and allows for flexibility as far as what fields are included. There should be a spec for the exact format used by CommonCrawl. Will let you know when I have a chance to do that, if that helps at all. Or, happy to answer any questions about the format.

[helgeho]

This should not be an issue as the DataSpec API has not changed since 2.5, and as you have defined ArchiveSpark as provided in your build.sbt, using it with a newer version of ArchiveSpark will just work I guess (not tested though).

Best, Helge