Different audit results between DevTools and Node module

Provide the steps to reproduce

1. I’m running LH on Chrome Canary (Version 78.0.3866.0 (Official Build) canary (64-bit)) using the DevTools with the following settings:

1. I’m running LH programmatically using chrome-launcher, here my dependencies:

• lighthouse 5.2.0

The code:

import * as ChromeLauncher from "chrome-launcher";
import * as lighthouse from 'lighthouse';
import * as LighthouseLogger from 'lighthouse-logger';
import * as ReportGenerator from 'lighthouse/lighthouse-core/report/report-generator';
import {DeviceType} from "../models/device-type.enum";
import TrackingUrlService from "./api/tracking-url.service";
import Report from "../models/api/report.model";
import EventEmitterHandler from "../helpers/eventEmitterHandler";

export default class LighthouseService {

    constructor() {
    }

    public static async launchHeadlessChromeAndRunLighthouse(config = null) {

        /**
         * Adjustments needed for DevTools network throttling to simulate
         * more realistic network conditions.
         * @see https://crbug.com/721112
         * @see https://docs.google.com/document/d/10lfVdS1iDWCRKQXPfbxEn4Or99D64mvNlugP1AQuFlE/edit
         * @see https://github.com/WPO-Foundation/webpagetest/blob/master/www/settings/connectivity.ini.sample
         */
        const DEVTOOLS_RTT_ADJUSTMENT_FACTOR = 3.75;
        const DEVTOOLS_THROUGHPUT_ADJUSTMENT_FACTOR = 0.9;
        const TARGET_LATENCY = 70;
        const TARGET_DOWNLOAD_THROUGHPUT = Math.floor(12 * 1024);
        const TARGET_UPLOAD_THROUGHPUT = Math.floor(12 * 1024);

        let flagsDesktop: any = {
            logLevel: 'info',
            throttlingMethod: 'provided',
            // disableDeviceEmulation: true,
            // disableCpuThrottling: true,
            disableStorageReset: true,
            emulatedFormFactor: 'desktop',
            port: null,
            throttling: {
                rttMs: 0,
                throughputKbps: 0,
                requestLatencyMs: 0,
                downloadThroughputKbps: 0,
                uploadThroughputKbps: 0,
                cpuSlowdownMultiplier: 0,
            }
        };

        let flagsMobile: any = {
            logLevel: 'info',
            throttlingMethod: 'devtools',
            // disableDeviceEmulation: false,
            // disableCpuThrottling: false,
            disableStorageReset: true,
            emulatedFormFactor: 'mobile',
            port: null,
            throttling: {
                rttMs: TARGET_LATENCY,
                throughputKbps: TARGET_DOWNLOAD_THROUGHPUT,
                requestLatencyMs: TARGET_LATENCY * DEVTOOLS_RTT_ADJUSTMENT_FACTOR,
                downloadThroughputKbps: TARGET_DOWNLOAD_THROUGHPUT * DEVTOOLS_THROUGHPUT_ADJUSTMENT_FACTOR,
                uploadThroughputKbps: TARGET_UPLOAD_THROUGHPUT * DEVTOOLS_THROUGHPUT_ADJUSTMENT_FACTOR,
                cpuSlowdownMultiplier: 4,
            }
        };

        LighthouseLogger.setLevel('info');

        let trackingUrlsList = await TrackingUrlService.getTrackingUrls();

        if (trackingUrlsList && trackingUrlsList.length > 0) {

            let chrome: any = await ChromeLauncher.launch({chromeFlags: ['--headless', '--no-sandbox', '--disable-setuid-sandbox', '--window-size=1366,768']});

            try {

                flagsDesktop.port = chrome.port;
                flagsMobile.port = chrome.port;

                for (let i = 0; i < trackingUrlsList.length; i++) {

                    let results = [];
                    let parsedData = [];
                    const url = trackingUrlsList[i].requestedUrl;
                    const trackingId = trackingUrlsList[i].trackingId;

                    console.log('--- Auditing desktop on ', url);
                    results.push({result: await lighthouse(url, flagsDesktop, config), device: DeviceType.Desktop, trackingId});
                    console.log('--- Auditing mobile on ', url);
                    results.push({result: await lighthouse(url, flagsMobile, config), device: DeviceType.Mobile, trackingId});

                    results.forEach(async (data: any) => {
                        console.log('Generating html report...');
                        let html = ReportGenerator.generateReport(data.result.lhr, 'html');
                        console.log('Parsing data...');
                        parsedData.push({data: LighthouseService.parseData(data), html});

                    });

                    EventEmitterHandler.getInstance().emit('reportsReadyForProcessing', parsedData);
                }

                await chrome.kill();
            } catch (e) {
                chrome.kill().then(
                    () => {
                        throw e;
                    },
                    console.error
                );
            }
        }
    }

    private static parseData(data): any {
       ...
    }
}

What is the current behavior?

There is a big difference between both TTI:

The runtime settings are the same fro both:

What is the expected behavior?

I expected to get similar results in both cases and that’s not the case;

Environment Information

• Affected Channels:
• Lighthouse version: 5.2.0
• Node.js version: 11.10.0
• Operating System: macOS Mojave 10.14.6

Related issues

[rainergp]

Thank you so much @patrickhulce . Yes, for what I need I must use puppeteer, so far this is handling the issue:

let chrome: any = await ChromeLauncher.launch({
  chromeFlags: ['--no-sandbox', '--disable-setuid-sandbox', '--window-size=1366,768']
});

const resp = await util.promisify(request)(`http://localhost:${chrome.port}/json/version`);
const {webSocketDebuggerUrl} = JSON.parse(resp.body);
const browser = await puppeteer.connect({browserWSEndpoint: webSocketDebuggerUrl});

const page = (await browser.pages())[0];
await page.goto('https://www.test.com', {waitUntil: 'networkidle2'});

await lighthouse(page.url(), flags, config)

I’m closing the issue, thank you for your help.

[rainergp]

I’m sorry @patrickhulce I pushed the unfinished description by mistake. Now you have the full description.