Support Network requests in Workers

WebWorkers can fetch data; we should surface traffic from web workers in puppeteer.

This is blocked on #2548 since nested targets are broken with request interception. See https://github.com/GoogleChrome/puppeteer/pull/2717#issuecomment-398899616 for details.

[mccolljr]

It’s now October 2019, I take it this is still not enabled? Is there an ETA?

[JBoudMS]

@aslushnikov

Regarding the workaround, you can attach raw devtools protocol session with target.createCDPSession() and enable network domain manually there. It should give you some events.

Could you elaborate on this: enable network domain manually there.?

Not sure if it would still be useful 3 months later, but in case anybody else finds it useful…

This is based on https://jarrodoverson.com/post/using-chrome-devtools-protocol-with-puppeteer-737a1300bac0/ but updated to use Fetch as Network has been deprecated. It is working with "puppeteer": "^1.18.1"

/**
 * Captures all traffic including from Web Workers, does something with it, and continues the request
 * @param page The page/tab to capture from.
 */
const interceptAllTrafficForPageUsingFetch = async (page) => {
    if (page) {
        const client = await page.target().createCDPSession();
        //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-enable
        await client.send('Fetch.enable', {
            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#type-RequestPattern
            patterns: [{
                urlPattern: '*',
                requestStage: 'Response'
            }]
        });
        //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#event-requestPaused
        await client.on('Fetch.requestPaused', async ({ requestId, request, responseHeaders }) => {
            const { url } = request;
            console.log(`Intercepted ${request.url}`)
            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-getResponseBody
            const responseBody = await client.send('Fetch.getResponseBody', {
                requestId
            });

            //Do something like saving the response to disk

            //see: https://chromedevtools.github.io/devtools-protocol/tot/Fetch#method-continueRequest
            await client.send('Fetch.continueRequest', {
                requestId
            })
        })
    }
}
/**
 * Launches a puppeteer controlled Chrome and intercepts all traffic.
 */
(async () => {
     const browser = await launch({
        headless: false,
        defaultViewport: VIEWPORT,
    });
    const page = (await browser.pages())[0];

    await interceptAllTrafficForPageUsingFetch(page);
    browser.on('targetcreated', async (target) => {
        const page = await target.page();
        await interceptAllTrafficForPageUsingFetch(page);
    });

   //Normal code like navigation, closing browser session, etc.
})