Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

smoke tests are flaky

See original GitHub issue

We have ~5 current flakes in the smoke tests: => https://github.com/GoogleChrome/lighthouse/issues/11341#issuecomment-691560587 <=

First, yarn smoke perf reliable fails for me locally.

$ node lighthouse-cli/test/smokehouse/frontends/smokehouse-bin.js perf
Running ONLY smoketests for: perf

perf smoketest starting…

perf smoketest results:
Doing a run of 'http://localhost:10200/preload.html'...
1 assertion(s) failed.
$ node lighthouse-cli/index.js http://localhost:10200/preload.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/config.json

  ✘ difference at uses-rel-preconnect audit.warnings.length
              expected: 1
                 found: 2

          found result:
      {
        "id": "uses-rel-preconnect",
        "title": "Preconnect to required origins",
        "description": "Consider adding `preconnect` or `dns-prefetch` resource hints to establish early connections to important third-party origins. [Learn more](https://web.dev/uses-rel-preconnect/).",
        "score": 1,
        "scoreDisplayMode": "numeric",
        "warnings": [
          "A preconnect <link> was found for \"https://fonts.googleapis.com\" but was not used by the browser. Check that you are using the `crossorigin` attribute properly.",
          "A preconnect <link> was found for \"https://fonts.gstatic.com\" but was not used by the browser. Only preconnect to important origins that the page will certainly request."
        ]
      }

  Correctly passed 10 assertions
  Failed 1 assertion


Doing a run of 'http://localhost:10200/perf/perf-budgets/load-things.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/perf-budgets/load-things.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/config.json
  Correctly passed 5 assertions


Doing a run of 'http://localhost:10200/perf/fonts.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/fonts.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/config.json
  Correctly passed 5 assertions


Doing a run of 'http://localhost:10200/perf/trace-elements.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/trace-elements.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/config.json
[0] failed chrome version check, pruning expectation: {
  "_minChromiumMilestone": 86,
  "name": "anim",
  "failureReasonsMask": 8224,
  "unsupportedProperties": [
    "background-color"
  ]
} Actual Chromium version: 84
  Correctly passed 7 assertions


Doing a run of 'http://localhost:10200/perf/trace-elements.html?missing'...
1 assertion(s) failed.
$ node lighthouse-cli/index.js http://localhost:10200/perf/trace-elements.html?missing --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/config.json

  ✘ difference at layout-shift-elements audit.scoreDisplayMode
              expected: "notApplicable"
                 found: "informative"

          found result:
      {
        "id": "layout-shift-elements",
        "title": "Avoid large layout shifts",
        "description": "These DOM elements contribute most to the CLS of the page.",
        "score": null,
        "scoreDisplayMode": "informative",
        "displayValue": "1 element found",
        "details": {
          "type": "table",
          "headings": [
            {
              "key": "node",
              "itemType": "node",
              "text": "Element"
            },
            {
              "key": "score",
              "itemType": "numeric",
              "granularity": 0.001,
              "text": "CLS Contribution"
            }
          ],
          "items": [
            {
              "node": {
                "type": "node",
                "path": "",
                "selector": "h1",
                "nodeLabel": "Please don't move me",
                "snippet": "<h1>",
                "boundingRect": {
                  "top": 0,
                  "bottom": 0,
                  "left": 0,
                  "right": 0,
                  "width": 0,
                  "height": 0
                }
              },
              "score": 0.0023515625
            }
          ]
        }
      }

  Correctly passed 4 assertions
  Failed 1 assertion


Doing a run of 'http://localhost:10200/perf/animations.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/animations.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/config.json
[non-composited-animations] failed chrome version check, pruning expectation: {
  "_minChromiumMilestone": 86,
  "score": null,
  "displayValue": "1 animated element found",
  "details": {
    "items": [
      {
        "node": {
          "type": "node",
          "path": "2,HTML,1,BODY,1,DIV",
          "selector": "body > div#animated-boi",
          "nodeLabel": "div",
          "snippet": "<div id=\"animated-boi\">"
        },
        "subItems": {
          "items": [
            {
              "failureReason": "Unsupported CSS Property: width"
            },
            {
              "failureReason": "Unsupported CSS Property: height",
              "animation": "alpha"
            },
            {
              "failureReason": "Unsupported CSS Property: background-color",
              "animation": "beta"
            }
          ]
        }
      }
    ]
  }
} Actual Chromium version: 84
  Correctly passed 3 assertions


perf smoketest complete.
  4 test(s) passing
  2 test(s) failing

34 expectations passing
2 expectations failing
We have 1 failing smoketests: perf
error Command failed with exit code 1.

Only tried with stable Chrome.

example for TraceElements: https://github.com/GoogleChrome/lighthouse/runs/1039274323?check_suite_focus=true#step:8:276 https://github.com/GoogleChrome/lighthouse/pull/11302/checks?check_run_id=1058028788#step:8:277