Extend WARC file with all requests made via all archive methods

Right now the FETCH_WARC option only creates a simple html file WARC with wget, it doesn’t save all the requests made dynamically after JS executes by chrome headless.

We should set up https://github.com/internetarchive/warcprox so that all requests made during the archiving process are saved to a unified WARC file.

In the ideal scenario, the WARC should include:

• √ base html for the page
• √ all assets like images, styles, fonts, js
• all dynamically requested assets after JS executes in chrome (e.g. images, ajax requests, etc)
• any media files requested

I think we can record the wget warc first, then use warcat to merge it with a warcproxy-created warc containing all the chrome headless requests.

[pirate]

There is a way to do this already right now:

1. Uncomment the example pywb proxy server in the docker-compose file
2. Enable using that proxy via CLI flag on chrome/other dependencies you want to use it with archivebox config CHROME_ARGS

[goelayu]

Correct me if I am wrong, but I don’t think there is a way to pass Chrome arguments using the CLI as of now. The following are the only options it reads from the config file. @pirate https://github.com/ArchiveBox/ArchiveBox/blob/49faec8f6dfc15075203ad332abfea0940f4e7b7/archivebox/util.py#L219-L263