Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Upload - Assertion failure for assertions with level error not actual checking passed value

See original GitHub issue

Describe the bug When uploading assertion results to LHCI server. It is incorrectly setting the github status check and marking it as failed.

Looking at the code It is checking for ‘error’ level but then not checking the actual passed parameter. So it seems that if any assertions set to error will always fail the Github status check.

To Reproduce Steps to reproduce the behavior:

Collect tests
Audit Tests
Upload Tests with GithubToken
See Error

Expected behavior When the assertion is marked as not passed then it should mark the github status as failed. Not just because it’s been set as an error level assertion.

Logs/Screenshots If applicable, add LHCI logs or screenshots to help explain your problem.

+ ls .lighthouseci
assertion-results.json
lhr-1615540607976.html
lhr-1615540607976.json
lhr-1615540612869.html
lhr-1615540612869.json
lhr-1615540617782.html
lhr-1615540617782.json
+ cat .lighthouseci/assertion-results.json
[
  {
    "name": "minScore",
    "expected": 0.98,
    "actual": 0.99,
    "values": [
      0.99,
      0.99,
      0.99
    ],
    "operator": ">=",
    "passed": true,
    "auditProperty": "performance",
    "auditId": "categories",
    "level": "error",
    "url": "http://run-server:5000/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "cumulative-layout-shift",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Cumulative Layout Shift",
    "auditDocumentationLink": "https://web.dev/cls/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "first-contentful-paint",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "First Contentful Paint",
    "auditDocumentationLink": "https://web.dev/first-contentful-paint/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "interactive",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Time to Interactive",
    "auditDocumentationLink": "https://web.dev/interactive/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 0.97,
    "values": [
      0.97,
      0.97,
      0.97
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "largest-contentful-paint",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Largest Contentful Paint",
    "auditDocumentationLink": "https://web.dev/lighthouse-largest-contentful-paint/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "speed-index",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Speed Index",
    "auditDocumentationLink": "https://web.dev/speed-index/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "total-blocking-time",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Total Blocking Time",
    "auditDocumentationLink": "https://web.dev/lighthouse-total-blocking-time/"
  }
]+ npm run test:perf:upload -- --githubToken=$GITHUB_TOKEN

> @domain-internal/cypress@0.0.0-377 test:perf:upload /drone/src/github.domain.net/web-engineering/cypress
> lhci upload "--githubToken={{GITHUB_TOKEN}}"

Saving CI project cypress (78b524fa-0249-4c64-a084-82f6b61c5e56)
Saving CI build ({{BUILDTOKEN}})
Saved LHR to https://lighthouse-ci.domain.io/ (a754f305-4152-4ab7-89f6-d8f1228f6293)
Saved LHR to https://lighthouse-ci.domain.io/ (f547531f-38cc-4ae5-9b3c-149b6fc7a522)
Saved LHR to https://lighthouse-ci.domain.io/ (54ebf667-27a4-48bb-afa1-6ff8849c8257)
Done saving build results to Lighthouse CI
View build diff at https://lighthouse-ci.domain.io/app/projects/cypress/compare/{{BUILD_TOKEN}}
GitHub token found, attempting to set status...
GitHub accepted "failure" status for "lhci/url/".

Environment (please complete the following information):