Investigate mutation testing

This is an exploration bug, not a specific proposal yet. Right now the compute time is way too high to run even in a daily cron job.

Mutation testing is another aspect of code coverage that tries to evaluate the quality of code coverage by changing parts of a code base and then checking if the test suite fails in response to that change. The idea is that just because a line of code is shown as covered by a test doesn’t mean that test is really exercising the line of code.

e.g. if if (val < 10) { /* whatever*/} is tested with the number 5, both the conditional and the following block are considered covered, but the case when val is greater than 10 is never exercised. A mutation testing library will try a few mutations, including flipping the conditional to val > 10 and replacing it with a true constant value, both of which would reveal the implicit path that wasn’t being tested.

This (along with fuzzing!) could prove invaluable as we start opening up more parts of the codebase to outside configuration (plugins, budgets, stack packs, etc).

I used https://stryker-mutator.io/ for this. It was easy to get going, though there was some tweaking necessary to get results (it spawns multiple workers as does jest, which ends up swamping the disk and CPU, and the default test timeout per mutation is too short for us). The test branch is here: https://github.com/GoogleChrome/lighthouse/compare/master...stryker (just yarn and yarn stryker run)

The only issues are that our tests are relatively slow (just yarn unit-core can take 30 seconds) and there are lots of possible mutations. Running config.js, gather-runner.js, and runner.js took about 5 hours on an 8 core machine.

Type checking before running unit tests can eliminate a whole bunch of the mutants, but is an extra 7-10 seconds on every run if the types are ok. jest can also hinder here, as its test ordering puts some of our heaviest jsdom-based tests first when they’re the least likely to fail for anything but mutating report code.

---

Results for config.js, gather-runner.js, and runner.js

html: https://quilted-jewel.glitch.me/ (their html report is pretty nice)

plain text

Ran all tests for this mutant.
0. [Survived] StringLiteral
lighthouse-core/config/config.js:7:26
-   const defaultConfigPath = './default-config.js';
+   const defaultConfigPath = "";

Ran all tests for this mutant.
16. [Survived] Block
lighthouse-core/config/config.js:35:25
-     passes.forEach(pass => {
-       pass.gatherers.forEach(gathererDefn => {
-         const gatherer = gathererDefn.instance;
-         const isGatherRequiredByAudits = requiredGatherers.has(gatherer.name);
-         if (!isGatherRequiredByAudits) {
-           const msg = `${gatherer.name} gatherer requested, however no audit requires it.`;
-           log.warn('config', msg);
-         }
-       });
-     });
+     passes.forEach(pass => {});

Ran all tests for this mutant.
17. [Survived] Block
lighthouse-core/config/config.js:36:43
-       pass.gatherers.forEach(gathererDefn => {
-         const gatherer = gathererDefn.instance;
-         const isGatherRequiredByAudits = requiredGatherers.has(gatherer.name);
-         if (!isGatherRequiredByAudits) {
-           const msg = `${gatherer.name} gatherer requested, however no audit requires it.`;
-           log.warn('config', msg);
-         }
-       });
+       pass.gatherers.forEach(gathererDefn => {});

Ran all tests for this mutant.
19. [Survived] IfStatement
lighthouse-core/config/config.js:39:10
-         if (!isGatherRequiredByAudits) {
+         if (false) {

Ran all tests for this mutant.
21. [Survived] Block
lighthouse-core/config/config.js:39:37
-         if (!isGatherRequiredByAudits) {
-           const msg = `${gatherer.name} gatherer requested, however no audit requires it.`;
-           log.warn('config', msg);
-         }
+         if (!isGatherRequiredByAudits) {}

Ran all tests for this mutant.
22. [Survived] StringLiteral
lighthouse-core/config/config.js:40:20
-           const msg = `${gatherer.name} gatherer requested, however no audit requires it.`;
+           const msg = "";

Ran all tests for this mutant.
24. [Survived] StringLiteral
lighthouse-core/config/config.js:41:17
-           log.warn('config', msg);
+           log.warn("", msg);

Ran all tests for this mutant.
88. [Survived] BinaryExpression
lighthouse-core/config/config.js:105:20
-     const auditName = auditPath ||
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+     const auditName = auditPath && auditDefinition && auditDefinition.meta && auditDefinition.meta.id;

Ran all tests for this mutant.
89. [Survived] ConditionalExpression
lighthouse-core/config/config.js:105:20
-     const auditName = auditPath ||
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+     const auditName = false;

Ran all tests for this mutant.
90. [Survived] ConditionalExpression
lighthouse-core/config/config.js:105:20
-     const auditName = auditPath ||
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+     const auditName = true;

Ran all tests for this mutant.
91. [Survived] BinaryExpression
lighthouse-core/config/config.js:106:5
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+       (auditDefinition && auditDefinition.meta || auditDefinition.meta.id);

Ran all tests for this mutant.
92. [Survived] ConditionalExpression
lighthouse-core/config/config.js:106:5
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+       (false);

Ran all tests for this mutant.
93. [Survived] ConditionalExpression
lighthouse-core/config/config.js:106:5
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+       (true);

Ran all tests for this mutant.
95. [Survived] ConditionalExpression
lighthouse-core/config/config.js:106:5
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+       (false && auditDefinition.meta.id);

Ran all tests for this mutant.
96. [Survived] ConditionalExpression
lighthouse-core/config/config.js:106:5
-       (auditDefinition && auditDefinition.meta && auditDefinition.meta.id);
+       (true && auditDefinition.meta.id);

Ran all tests for this mutant.
127. [Survived] ConditionalExpression
lighthouse-core/config/config.js:123:6
-     if (typeof auditDefinition.meta.failureTitle !== 'string' &&
+     if (true &&

Ran all tests for this mutant.
153. [Survived] BinaryExpression
lighthouse-core/config/config.js:150:17
-     gathererName = gathererName || gathererInstance.name || 'gatherer';
+     gathererName = (gathererName || gathererInstance.name) && 'gatherer';

Ran all tests for this mutant.
154. [Survived] BinaryExpression
lighthouse-core/config/config.js:150:17
-     gathererName = gathererName || gathererInstance.name || 'gatherer';
+     gathererName = gathererName && gathererInstance.name || 'gatherer';

Ran all tests for this mutant.
155. [Survived] ConditionalExpression
lighthouse-core/config/config.js:150:17
-     gathererName = gathererName || gathererInstance.name || 'gatherer';
+     gathererName = false || 'gatherer';

Ran all tests for this mutant.
158. [Survived] StringLiteral
lighthouse-core/config/config.js:150:58
-     gathererName = gathererName || gathererInstance.name || 'gatherer';
+     gathererName = gathererName || gathererInstance.name || "";

Ran all tests for this mutant.
208. [Survived] Block
lighthouse-core/config/config.js:211:52
-     if (typeof base === 'undefined' || base === null) {
-       return extension;
-     } else if (typeof extension === 'undefined') {
+     if (typeof base === 'undefined' || base === null) {} else if (typeof extension === 'undefined') {

Ran all tests for this mutant.
222. [Survived] StringLiteral
lighthouse-core/config/config.js:217:50
-       if (!Array.isArray(base)) throw new TypeError(`Expected array but got ${typeof base}`);
+       if (!Array.isArray(base)) throw new TypeError("");

Ran all tests for this mutant.
225. [Survived] IfStatement
lighthouse-core/config/config.js:220:10
-         if (!merged.some(candidate => isDeepEqual(candidate, item))) merged.push(item);
+         if (true) merged.push(item);

Ran all tests for this mutant.
232. [Survived] IfStatement
lighthouse-core/config/config.js:225:8
-       if (typeof base !== 'object') throw new TypeError(`Expected object but got ${typeof base}`);
+       if (false) throw new TypeError(`Expected object but got ${typeof base}`);

Ran all tests for this mutant.
237. [Survived] StringLiteral
lighthouse-core/config/config.js:225:54
-       if (typeof base !== 'object') throw new TypeError(`Expected object but got ${typeof base}`);
+       if (typeof base !== 'object') throw new TypeError("");

Ran all tests for this mutant.
239. [Survived] StringLiteral
lighthouse-core/config/config.js:226:49
-       if (Array.isArray(base)) throw new TypeError('Expected object but got Array');
+       if (Array.isArray(base)) throw new TypeError("");

Ran all tests for this mutant.
255. [Survived] ConditionalExpression
lighthouse-core/config/config.js:229:31
-           (key === 'settings' && typeof base[key] === 'object');
+           (key === 'settings' && true);

Ran all tests for this mutant.
257. [Survived] IfStatement
lighthouse-core/config/config.js:252:8
-       if (typeof item === 'object') {
+       if (false) {

Ran all tests for this mutant.
262. [Survived] Block
lighthouse-core/config/config.js:252:34
-       if (typeof item === 'object') {
-         // Return copy of instance and prototype chain (in case item is instantiated class).
-         return Object.assign(
-           Object.create(
-             Object.getPrototypeOf(item)
-           ),
-           item
-         );
-       }
+       if (typeof item === 'object') {}

Ran all tests for this mutant.
293. [Survived] StringLiteral
lighthouse-core/config/config.js:335:25
-       const status = {msg: 'Create config', id: 'lh:init:config'};
+       const status = {msg: "", id: 'lh:init:config'};

Ran all tests for this mutant.
407. [Survived] ConditionalExpression
lighthouse-core/config/config.js:534:46
-         } else if ('implementation' in audit && typeof audit.implementation.audit === 'function') {
+         } else if ('implementation' in audit && true) {

Ran all tests for this mutant.
417. [Survived] ConditionalExpression
lighthouse-core/config/config.js:537:36
-         } else if ('path' in audit && typeof audit.path === 'string') {
+         } else if ('path' in audit && true) {

Ran all tests for this mutant.
426. [Survived] ConditionalExpression
lighthouse-core/config/config.js:540:37
-         } else if ('audit' in audit && typeof audit.audit === 'function') {
+         } else if ('audit' in audit && true) {

Ran all tests for this mutant.
481. [Survived] ConditionalExpression
lighthouse-core/config/config.js:598:9
-           (settings.throttlingMethod !== 'devtools' && settings.throttlingMethod !== 'provided')) {
+           (false)) {

Ran all tests for this mutant.
484. [Survived] ConditionalExpression
lighthouse-core/config/config.js:598:9
-           (settings.throttlingMethod !== 'devtools' && settings.throttlingMethod !== 'provided')) {
+           (false && settings.throttlingMethod !== 'provided')) {

Ran all tests for this mutant.
485. [Survived] BinaryExpression
lighthouse-core/config/config.js:598:53
-           (settings.throttlingMethod !== 'devtools' && settings.throttlingMethod !== 'provided')) {
+           (settings.throttlingMethod !== 'devtools' && settings.throttlingMethod === 'provided')) {

Ran all tests for this mutant.
486. [Survived] ConditionalExpression
lighthouse-core/config/config.js:598:53
-           (settings.throttlingMethod !== 'devtools' && settings.throttlingMethod !== 'provided')) {
+           (settings.throttlingMethod !== 'devtools' && false)) {

Ran all tests for this mutant.
489. [Survived] ConditionalExpression
lighthouse-core/config/config.js:598:53
-           (settings.throttlingMethod !== 'devtools' && settings.throttlingMethod !== 'provided')) {
+           (settings.throttlingMethod !== 'devtools' && true)) {

Ran all tests for this mutant.
492. [Survived] ConditionalExpression
lighthouse-core/config/config.js:602:44
-       const defaultPass = passes.find(pass => pass.passName === 'defaultPass');
+       const defaultPass = passes.find(pass => true);

Ran all tests for this mutant.
522. [Survived] StringLiteral
lighthouse-core/config/config.js:654:22
-         throw new Error('Cannot set both skipAudits and onlyAudits');
+         throw new Error("");

Ran all tests for this mutant.
540. [Survived] StringLiteral
lighthouse-core/config/config.js:668:17
-           log.warn('config', `unrecognized category in 'onlyCategories': ${categoryId}`);
+           log.warn("", `unrecognized category in 'onlyCategories': ${categoryId}`);

Ran all tests for this mutant.
541. [Survived] StringLiteral
lighthouse-core/config/config.js:668:27
-           log.warn('config', `unrecognized category in 'onlyCategories': ${categoryId}`);
+           log.warn('config', "");

Ran all tests for this mutant.
546. [Survived] BinaryExpression
lighthouse-core/config/config.js:677:45
-           return !!auditRefs.find(candidate => candidate.id === auditId);
+           return !!auditRefs.find(candidate => candidate.id !== auditId);

Ran all tests for this mutant.
547. [Survived] ConditionalExpression
lighthouse-core/config/config.js:677:45
-           return !!auditRefs.find(candidate => candidate.id === auditId);
+           return !!auditRefs.find(candidate => false);

Ran all tests for this mutant.
550. [Survived] ConditionalExpression
lighthouse-core/config/config.js:677:45
-           return !!auditRefs.find(candidate => candidate.id === auditId);
+           return !!auditRefs.find(candidate => true);

Ran all tests for this mutant.
553. [Survived] StringLiteral
lighthouse-core/config/config.js:681:63
-           const parentKeyName = skipAuditIds.includes(auditId) ? 'skipAudits' : 'onlyAudits';
+           const parentKeyName = skipAuditIds.includes(auditId) ? "" : 'onlyAudits';

Ran all tests for this mutant.
554. [Survived] StringLiteral
lighthouse-core/config/config.js:681:78
-           const parentKeyName = skipAuditIds.includes(auditId) ? 'skipAudits' : 'onlyAudits';
+           const parentKeyName = skipAuditIds.includes(auditId) ? 'skipAudits' : "";

Ran all tests for this mutant.
555. [Survived] StringLiteral
lighthouse-core/config/config.js:682:17
-           log.warn('config', `unrecognized audit in '${parentKeyName}': ${auditId}`);
+           log.warn("", `unrecognized audit in '${parentKeyName}': ${auditId}`);

Ran all tests for this mutant.
556. [Survived] StringLiteral
lighthouse-core/config/config.js:682:27
-           log.warn('config', `unrecognized audit in '${parentKeyName}': ${auditId}`);
+           log.warn('config', "");

Ran all tests for this mutant.
558. [Survived] IfStatement
lighthouse-core/config/config.js:683:17
-         } else if (auditIds.includes(auditId) && categoryIds.includes(foundCategory)) {
+         } else if (true) {

Ran all tests for this mutant.
559. [Survived] BinaryExpression
lighthouse-core/config/config.js:683:17
-         } else if (auditIds.includes(auditId) && categoryIds.includes(foundCategory)) {
+         } else if (auditIds.includes(auditId) || categoryIds.includes(foundCategory)) {

Ran all tests for this mutant.
562. [Survived] StringLiteral
lighthouse-core/config/config.js:684:17
-           log.warn('config', `${auditId} in 'onlyAudits' is already included by ` +
+           log.warn("", `${auditId} in 'onlyAudits' is already included by ` +

Ran all tests for this mutant.
563. [Survived] StringLiteral
lighthouse-core/config/config.js:684:27
-           log.warn('config', `${auditId} in 'onlyAudits' is already included by ` +
+           log.warn('config', "" +

Ran all tests for this mutant.
564. [Survived] StringLiteral
lighthouse-core/config/config.js:685:12
-               `${foundCategory} in 'onlyCategories'`);
+               "");

Ran all tests for this mutant.
568. [Survived] BinaryExpression
lighthouse-core/config/config.js:695:10
-         if (filterByIncludedCategory && filterByIncludedAudit) {
+         if (filterByIncludedCategory || filterByIncludedAudit) {

Ran all tests for this mutant.
615. [Survived] BinaryExpression
lighthouse-core/config/config.js:776:25
-           const passName = pass.passName || 'unknown pass';
+           const passName = pass.passName && 'unknown pass';

Ran all tests for this mutant.
616. [Survived] ConditionalExpression
lighthouse-core/config/config.js:776:25
-           const passName = pass.passName || 'unknown pass';
+           const passName = false;

Ran all tests for this mutant.
617. [Survived] ConditionalExpression
lighthouse-core/config/config.js:776:25
-           const passName = pass.passName || 'unknown pass';
+           const passName = true;

Ran all tests for this mutant.
618. [Survived] StringLiteral
lighthouse-core/config/config.js:776:42
-           const passName = pass.passName || 'unknown pass';
+           const passName = pass.passName || "";

Ran all tests for this mutant.
619. [Survived] StringLiteral
lighthouse-core/config/config.js:777:17
-           log.warn('config', `Trace not requested by an audit, dropping trace in ${passName}`);
+           log.warn("", `Trace not requested by an audit, dropping trace in ${passName}`);

Ran all tests for this mutant.
620. [Survived] StringLiteral
lighthouse-core/config/config.js:777:27
-           log.warn('config', `Trace not requested by an audit, dropping trace in ${passName}`);
+           log.warn('config', "");

Ran all tests for this mutant.
624. [Survived] IfStatement
lighthouse-core/config/config.js:784:10
-         if (pass.recordTrace) return true;
+         if (false) return true;

Ran all tests for this mutant.
638. [Survived] StringLiteral
lighthouse-core/config/config.js:801:25
-       const status = {msg: 'Requiring audits', id: 'lh:config:requireAudits'};
+       const status = {msg: "", id: 'lh:config:requireAudits'};

Ran all tests for this mutant.
686. [Survived] StringLiteral
lighthouse-core/config/config.js:876:25
-       const status = {msg: 'Requiring gatherers', id: 'lh:config:requireGatherers'};
+       const status = {msg: "", id: 'lh:config:requireGatherers'};

Ran all tests for this mutant.
702. [Survived] ConditionalExpression
lighthouse-core/config/config.js:895:21
-               options: gathererDefn.options || {},
+               options: false,

Ran all tests for this mutant.
703. [Survived] ConditionalExpression
lighthouse-core/config/config.js:895:21
-               options: gathererDefn.options || {},
+               options: true,

Ran all tests for this mutant.
711. [Survived] StringLiteral
lighthouse-core/config/config.js:902:26
-             throw new Error('Invalid expanded Gatherer: ' + JSON.stringify(gathererDefn));
+             throw new Error("" + JSON.stringify(gathererDefn));

Ran all tests for this mutant.
719. [Survived] StringLiteral
lighthouse-core/config/config.js:943:38
-           (category ? `${category}: ` : '') +
+           (category ? `${category}: ` : "Stryker was here!") +

Ran all tests for this mutant.
726. [Survived] StringLiteral
lighthouse-core/config/config.js:947:36
-         throw new Error(errorString + ')');
+         throw new Error(errorString + "");

Ran all tests for this mutant.
730. [Survived] StringLiteral
lighthouse-core/config/config.js:958:34
-       throw new Error(errorString + ` and '${relativePath}')`);
+       throw new Error(errorString + "");

Ran all tests for this mutant.
737. [Survived] Block
lighthouse-core/gather/gather-runner.js:73:78
-     static async loadBlank(driver, url = constants.defaultPassConfig.blankPage) {
-       const status = {msg: 'Resetting state with about:blank', id: 'lh:gather:loadBlank'};
-       log.time(status);
-       await driver.gotoURL(url, {waitForNavigated: true});
-       log.timeEnd(status);
-     }
+     static async loadBlank(driver, url = constants.defaultPassConfig.blankPage) {}

Ran all tests for this mutant.
738. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:74:25
-       const status = {msg: 'Resetting state with about:blank', id: 'lh:gather:loadBlank'};
+       const status = {msg: "", id: 'lh:gather:loadBlank'};

Ran all tests for this mutant.
740. [Survived] ObjectLiteral
lighthouse-core/gather/gather-runner.js:76:30
-       await driver.gotoURL(url, {waitForNavigated: true});
+       await driver.gotoURL(url, {});

Ran all tests for this mutant.
741. [Survived] BooleanSubstitution
lighthouse-core/gather/gather-runner.js:76:49
-       await driver.gotoURL(url, {waitForNavigated: true});
+       await driver.gotoURL(url, {waitForNavigated: false});

Ran all tests for this mutant.
743. [Survived] ObjectLiteral
lighthouse-core/gather/gather-runner.js:90:59
-       const finalUrl = await driver.gotoURL(passContext.url, {
-         waitForFCP: passContext.passConfig.recordTrace,
-         waitForLoad: true,
-         passContext,
-       });
+       const finalUrl = await driver.gotoURL(passContext.url, {});

Ran all tests for this mutant.
744. [Survived] BooleanSubstitution
lighthouse-core/gather/gather-runner.js:92:19
-         waitForLoad: true,
+         waitForLoad: false,

Ran all tests for this mutant.
748. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:104:25
-       const status = {msg: 'Initializing…', id: 'lh:gather:setupDriver'};
+       const status = {msg: "", id: 'lh:gather:setupDriver'};

Ran all tests for this mutant.
753. [Survived] Block
lighthouse-core/gather/gather-runner.js:121:37
-     static async disposeDriver(driver) {
-       const status = {msg: 'Disconnecting from browser...', id: 'lh:gather:disconnect'};
-
-       log.time(status);
-       try {
-         await driver.disconnect();
-       } catch (err) {
-         // Ignore disconnecting error if browser was already closed.
-         // See https://github.com/GoogleChrome/lighthouse/issues/1583
-         if (!(/close\/.*status: (500|404)$/.test(err.message))) {
-           log.error('GatherRunner disconnect', err.message);
-         }
-       }
-       log.timeEnd(status);
-     }
+     static async disposeDriver(driver) {}

Ran all tests for this mutant.
755. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:122:25
-       const status = {msg: 'Disconnecting from browser...', id: 'lh:gather:disconnect'};
+       const status = {msg: "", id: 'lh:gather:disconnect'};

Ran all tests for this mutant.
756. [Survived] Block
lighthouse-core/gather/gather-runner.js:125:8
-       try {
-         await driver.disconnect();
-       } catch (err) {
+       try {} catch (err) {

Ran all tests for this mutant.
757. [Survived] Block
lighthouse-core/gather/gather-runner.js:127:18
-       } catch (err) {
-         // Ignore disconnecting error if browser was already closed.
-         // See https://github.com/GoogleChrome/lighthouse/issues/1583
-         if (!(/close\/.*status: (500|404)$/.test(err.message))) {
-           log.error('GatherRunner disconnect', err.message);
-         }
-       }
+       } catch (err) {}

Ran all tests for this mutant.
758. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:130:10
-         if (!(/close\/.*status: (500|404)$/.test(err.message))) {
+         if (false) {

Ran all tests for this mutant.
759. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:130:10
-         if (!(/close\/.*status: (500|404)$/.test(err.message))) {
+         if (true) {

Ran all tests for this mutant.
761. [Survived] Block
lighthouse-core/gather/gather-runner.js:130:62
-         if (!(/close\/.*status: (500|404)$/.test(err.message))) {
-           log.error('GatherRunner disconnect', err.message);
-         }
+         if (!(/close\/.*status: (500|404)$/.test(err.message))) {}

Ran all tests for this mutant.
762. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:131:18
-           log.error('GatherRunner disconnect', err.message);
+           log.error("", err.message);

Ran all tests for this mutant.
784. [Survived] ConditionalExpression
lighthouse-core/gather/gather-runner.js:157:8
-           netErr === 'net::ERR_NAME_RESOLUTION_FAILED' ||
+           false ||

Ran all tests for this mutant.
786. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:157:19
-           netErr === 'net::ERR_NAME_RESOLUTION_FAILED' ||
+           netErr === "" ||

Ran all tests for this mutant.
798. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:183:27
-       const bpStatus = {msg: `Running beforePass methods`, id: `lh:gather:beforePass`};
+       const bpStatus = {msg: "", id: `lh:gather:beforePass`};

Ran all tests for this mutant.
812. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:199:13
-           msg: `Retrieving setup: ${gatherer.name}`,
+           msg: "",

Ran all tests for this mutant.
822. [Survived] ConditionalExpression
lighthouse-core/gather/gather-runner.js:225:22
-       const isPerfRun = !settings.disableStorageReset && recordTrace && config.useThrottling;
+       const isPerfRun = true && config.useThrottling;

Ran all tests for this mutant.
825. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:228:11
-         msg: 'Loading page & waiting for onload',
+         msg: "",

Ran all tests for this mutant.
826. [Survived] ArrayLiteral
lighthouse-core/gather/gather-runner.js:230:12
-         args: [gatherers.map(g => g.instance.name).join(', ')],
+         args: [],

Ran all tests for this mutant.
827. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:230:54
-         args: [gatherers.map(g => g.instance.name).join(', ')],
+         args: [gatherers.map(g => g.instance.name).join("")],

Ran all tests for this mutant.
834. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:245:26
-       const pStatus = {msg: `Running pass methods`, id: `lh:gather:pass`};
+       const pStatus = {msg: "", id: `lh:gather:pass`};

Ran all tests for this mutant.
842. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:252:13
-           msg: `Retrieving in-page: ${gatherer.name}`,
+           msg: "",

Ran all tests for this mutant.
852. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:282:27
-         const status = {msg: 'Retrieving trace', id: `lh:gather:getTrace`};
+         const status = {msg: "", id: `lh:gather:getTrace`};

Ran all tests for this mutant.
855. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:289:11
-         msg: 'Retrieving devtoolsLog & network records',
+         msg: "",

Ran all tests for this mutant.
863. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:302:16
-         log.error('GatherRunner', pageLoadError.message, passContext.url);
+         log.error("", pageLoadError.message, passContext.url);

Ran all tests for this mutant.
866. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:314:27
-       const apStatus = {msg: `Running afterPass methods`, id: `lh:gather:afterPass`};
+       const apStatus = {msg: "", id: `lh:gather:afterPass`};

Ran all tests for this mutant.
867. [Survived] ObjectLiteral
lighthouse-core/gather/gather-runner.js:316:53
-       await driver.setThrottling(passContext.settings, {useThrottling: false});
+       await driver.setThrottling(passContext.settings, {});

Ran all tests for this mutant.
868. [Survived] BooleanSubstitution
lighthouse-core/gather/gather-runner.js:316:69
-       await driver.setThrottling(passContext.settings, {useThrottling: false});
+       await driver.setThrottling(passContext.settings, {useThrottling: true});

Ran all tests for this mutant.
873. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:322:13
-           msg: `Retrieving: ${gatherer.name}`,
+           msg: "",

Ran all tests for this mutant.
895. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:378:24
-           throw new Error(`${gathererName} failed to provide an artifact.`);
+           throw new Error("");

Ran all tests for this mutant.
899. [Survived] BinaryExpression
lighthouse-core/gather/gather-runner.js:401:25
-       const IsMobileHost = hostUserAgent.includes('Android') || hostUserAgent.includes('Mobile');
+       const IsMobileHost = hostUserAgent.includes('Android') && hostUserAgent.includes('Mobile');

Ran all tests for this mutant.
922. [Survived] StringLiteral
lighthouse-core/gather/gather-runner.js:416:58
-         URL: {requestedUrl: options.requestedUrl, finalUrl: ''},
+         URL: {requestedUrl: options.requestedUrl, finalUrl: "Stryker was here!"},

Ran all tests for this mutant.
930. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:474:12
-           if (!isFirstPass) {
+           if (false) {

Ran all tests for this mutant.
931. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:474:12
-           if (!isFirstPass) {
+           if (true) {

Ran all tests for this mutant.
932. [Survived] PrefixUnaryExpression
lighthouse-core/gather/gather-runner.js:474:12
-           if (!isFirstPass) {
+           if (isFirstPass) {

Ran all tests for this mutant.
933. [Survived] Block
lighthouse-core/gather/gather-runner.js:474:26
-           if (!isFirstPass) {
-             // Already on blank page if driver was just set up.
-             await GatherRunner.loadBlank(driver, passConfig.blankPage);
-           }
+           if (!isFirstPass) {}

Ran all tests for this mutant.
934. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:480:12
-           if (isFirstPass) {
+           if (false) {

Ran all tests for this mutant.
935. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:480:12
-           if (isFirstPass) {
+           if (true) {

Ran all tests for this mutant.
937. [Survived] Block
lighthouse-core/gather/gather-runner.js:480:25
-           if (isFirstPass) {
-             baseArtifacts.WebAppManifest = await GatherRunner.getWebAppManifest(passContext);
-           }
+           if (isFirstPass) {}

Ran all tests for this mutant.
941. [Survived] ConditionalExpression
lighthouse-core/gather/gather-runner.js:489:10
-             entry.method === 'Network.requestWillBeSent' &&
-             !!entry.params.request.headers['User-Agent']
+             true

Ran all tests for this mutant.
948. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:493:12
-           if (userAgentEntry && !baseArtifacts.NetworkUserAgent) {
+           if (true) {

Ran all tests for this mutant.
949. [Survived] BinaryExpression
lighthouse-core/gather/gather-runner.js:493:12
-           if (userAgentEntry && !baseArtifacts.NetworkUserAgent) {
+           if (userAgentEntry || !baseArtifacts.NetworkUserAgent) {

Ran all tests for this mutant.
958. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:503:12
-           if (isFirstPass) {
+           if (true) {

Ran all tests for this mutant.
959. [Survived] BooleanSubstitution
lighthouse-core/gather/gather-runner.js:506:24
-             isFirstPass = false;
+             isFirstPass = true;

Ran all tests for this mutant.
960. [Survived] PrefixUnaryExpression
lighthouse-core/gather/gather-runner.js:509:27
-         const resetStorage = !options.settings.disableStorageReset;
+         const resetStorage = options.settings.disableStorageReset;

Ran all tests for this mutant.
961. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:510:10
-         if (resetStorage) await driver.clearDataForOrigin(options.requestedUrl);
+         if (false) await driver.clearDataForOrigin(options.requestedUrl);

Ran all tests for this mutant.
962. [Survived] IfStatement
lighthouse-core/gather/gather-runner.js:510:10
-         if (resetStorage) await driver.clearDataForOrigin(options.requestedUrl);
+         if (true) await driver.clearDataForOrigin(options.requestedUrl);

Ran all tests for this mutant.
982. [Survived] StringLiteral
lighthouse-core/runner.js:34:33
-         const runnerStatus = {msg: 'Runner setup', id: 'lh:runner:run'};
+         const runnerStatus = {msg: "", id: 'lh:runner:run'};

Ran all tests for this mutant.
984. [Survived] ObjectLiteral
lighthouse-core/runner.js:44:31
-         Sentry.captureBreadcrumb({
-           message: 'Run started',
-           category: 'lifecycle',
-           data: sentryContext && sentryContext.extra,
-         });
+         Sentry.captureBreadcrumb({});

Ran all tests for this mutant.
985. [Survived] StringLiteral
lighthouse-core/runner.js:45:17
-           message: 'Run started',
+           message: "",

Ran all tests for this mutant.
987. [Survived] StringLiteral
lighthouse-core/runner.js:46:18
-           category: 'lifecycle',
+           category: "",

Ran all tests for this mutant.
988. [Survived] ConditionalExpression
lighthouse-core/runner.js:47:14
-           data: sentryContext && sentryContext.extra,
+           data: false,

Ran all tests for this mutant.
990. [Survived] ConditionalExpression
lighthouse-core/runner.js:47:14
-           data: sentryContext && sentryContext.extra,
+           data: true,

Ran all tests for this mutant.
997. [Survived] IfStatement
lighthouse-core/runner.js:64:12
-           if (!requestedUrl) {
+           if (false) {

Ran all tests for this mutant.
998. [Survived] Block
lighthouse-core/runner.js:64:27
-           if (!requestedUrl) {
-             throw new Error('Cannot run audit mode on empty URL');
-           }
+           if (!requestedUrl) {}

Ran all tests for this mutant.
999. [Survived] StringLiteral
lighthouse-core/runner.js:65:26
-             throw new Error('Cannot run audit mode on empty URL');
+             throw new Error("");

Ran all tests for this mutant.
1017. [Survived] ConditionalExpression
lighthouse-core/runner.js:71:47
-           if (typeof runOpts.url !== 'string' || runOpts.url.length === 0) {
+           if (typeof runOpts.url !== 'string' || false) {

Ran all tests for this mutant.
1020. [Survived] StringLiteral
lighthouse-core/runner.js:72:26
-             throw new Error(`You must provide a ur l to the r unner. '${runOpts.url}' provided.`);
+             throw new Error("");

Ran all tests for this mutant.
1021. [Survived] StringLiteral
lighthouse-core/runner.js:79:26
-             throw new Error('The url provided should have a proper protocol and hostname.');
+             throw new Error("");

Ran all tests for this mutant.
1036. [Survived] StringLiteral
lighthouse-core/runner.js:101:34
-         const resultsStatus = {msg: 'Generating results...', id: 'lh:runner:generate'};
+         const resultsStatus = {msg: "", id: 'lh:runner:generate'};

Ran all tests for this mutant.
1053. [Survived] ObjectLiteral
lighthouse-core/runner.js:161:41
-         await Sentry.captureException(err, {level: 'fatal'});
+         await Sentry.captureException(err, {});

Ran all tests for this mutant.
1055. [Survived] StringLiteral
lighthouse-core/runner.js:161:49
-         await Sentry.captureException(err, {level: 'fatal'});
+         await Sentry.captureException(err, {level: ""});

Ran all tests for this mutant.
1061. [Survived] BinaryExpression
lighthouse-core/runner.js:182:48
-       const runnerEntry = timingEntries.find(e => e.name === 'lh:runner:run');
+       const runnerEntry = timingEntries.find(e => e.name !== 'lh:runner:run');

Ran all tests for this mutant.
1062. [Survived] ConditionalExpression
lighthouse-core/runner.js:182:48
-       const runnerEntry = timingEntries.find(e => e.name === 'lh:runner:run');
+       const runnerEntry = timingEntries.find(e => false);

Ran all tests for this mutant.
1063. [Survived] ConditionalExpression
lighthouse-core/runner.js:182:48
-       const runnerEntry = timingEntries.find(e => e.name === 'lh:runner:run');
+       const runnerEntry = timingEntries.find(e => true);

Ran all tests for this mutant.
1064. [Survived] StringLiteral
lighthouse-core/runner.js:182:59
-       const runnerEntry = timingEntries.find(e => e.name === 'lh:runner:run');
+       const runnerEntry = timingEntries.find(e => e.name === "");

Ran all tests for this mutant.
1072. [Survived] ConditionalExpression
lighthouse-core/runner.js:183:43
-       return {entries: timingEntries, total: runnerEntry && runnerEntry.duration || 0};
+       return {entries: timingEntries, total: false || 0};

Ran all tests for this mutant.
1085. [Survived] StringLiteral
lighthouse-core/runner.js:216:25
-       const status = {msg: 'Analyzing and running audits...', id: 'lh:runner:auditing'};
+       const status = {msg: "", id: 'lh:runner:auditing'};

Ran all tests for this mutant.
1100. [Survived] StringLiteral
lighthouse-core/runner.js:264:11
-         msg: `Evaluating: ${i18n.getFormatted(audit.meta.title, 'en-US')}`,
+         msg: "",

Ran all tests for this mutant.
1119. [Survived] StringLiteral
lighthouse-core/runner.js:280:19
-             log.warn('Runner',
+             log.warn("",

Ran all tests for this mutant.
1120. [Survived] StringLiteral
lighthouse-core/runner.js:281:14
-                 `${artifactName} gatherer, required by audit ${audit.meta.id}, did not run.`);
+                 "");

Ran all tests for this mutant.
1125. [Survived] ObjectLiteral
lighthouse-core/runner.js:291:49
-             Sentry.captureException(artifactError, {
-               tags: {gatherer: artifactName},
-               level: 'error',
-             });
+             Sentry.captureException(artifactError, {});

Ran all tests for this mutant.
1126. [Survived] ObjectLiteral
lighthouse-core/runner.js:292:18
-               tags: {gatherer: artifactName},
+               tags: {},

Ran all tests for this mutant.
1127. [Survived] StringLiteral
lighthouse-core/runner.js:293:19
-               level: 'error',
+               level: "",

Ran all tests for this mutant.
1129. [Survived] StringLiteral
lighthouse-core/runner.js:296:19
-             log.warn('Runner', `${artifactName} gatherer, required by audit ${audit.meta.id},` +
+             log.warn("", `${artifactName} gatherer, required by audit ${audit.meta.id},` +

Ran all tests for this mutant.
1130. [Survived] StringLiteral
lighthouse-core/runner.js:296:29
-             log.warn('Runner', `${artifactName} gatherer, required by audit ${audit.meta.id},` +
+             log.warn('Runner', "" +

Ran all tests for this mutant.
1131. [Survived] StringLiteral
lighthouse-core/runner.js:297:12
-               ` encountered an error: ${artifactError.message}`);
+               "");

Ran all tests for this mutant.
1134. [Survived] BooleanSubstitution
lighthouse-core/runner.js:303:27
-             error.expected = true;
+             error.expected = false;

Ran all tests for this mutant.
1136. [Survived] StringLiteral
lighthouse-core/runner.js:318:30
-         log.warn(audit.meta.id, `Caught exception: ${err.message}`);
+         log.warn(audit.meta.id, "");

Ran all tests for this mutant.
1137. [Survived] ObjectLiteral
lighthouse-core/runner.js:320:35
-         Sentry.captureException(err, {tags: {audit: audit.meta.id}, level: 'error'});
+         Sentry.captureException(err, {});

Ran all tests for this mutant.
1138. [Survived] ObjectLiteral
lighthouse-core/runner.js:320:42
-         Sentry.captureException(err, {tags: {audit: audit.meta.id}, level: 'error'});
+         Sentry.captureException(err, {tags: {}, level: 'error'});

Ran all tests for this mutant.
1139. [Survived] StringLiteral
lighthouse-core/runner.js:320:73
-         Sentry.captureException(err, {tags: {audit: audit.meta.id}, level: 'error'});
+         Sentry.captureException(err, {tags: {audit: audit.meta.id}, level: ""});

Ran all tests for this mutant.
1188. [Survived] BinaryExpression
lighthouse-core/runner.js:394:32
-       return fileList.filter(f => /\.js$/.test(f) && f !== 'gatherer.js').sort();
+       return fileList.filter(f => /\.js$/.test(f) || f !== 'gatherer.js').sort();

Ran all tests for this mutant.
1191. [Survived] ConditionalExpression
lighthouse-core/runner.js:394:32
-       return fileList.filter(f => /\.js$/.test(f) && f !== 'gatherer.js').sort();
+       return fileList.filter(f => true).sort();

Ran all tests for this mutant.
1193. [Survived] ConditionalExpression
lighthouse-core/runner.js:394:51
-       return fileList.filter(f => /\.js$/.test(f) && f !== 'gatherer.js').sort();
+       return fileList.filter(f => /\.js$/.test(f) && true).sort();

Ran all tests for this mutant.
1194. [Survived] StringLiteral
lighthouse-core/runner.js:394:57
-       return fileList.filter(f => /\.js$/.test(f) && f !== 'gatherer.js').sort();
+       return fileList.filter(f => /\.js$/.test(f) && f !== "").sort();

Ran all tests for this mutant.
1204. [Survived] StringLiteral
lighthouse-core/runner.js:409:36
-       return path.join(process.cwd(), 'latest-run');
+       return path.join(process.cwd(), "");

Ran all tests for this mutant.
Ran 1.00 tests per mutant on average.
-------------------------|---------|----------|-----------|------------|----------|---------|
File                     | % score | # killed | # timeout | # survived | # no cov | # error |
-------------------------|---------|----------|-----------|------------|----------|---------|
All files                |   86.72 |     1045 |         0 |        160 |        0 |       0 |
 config/config.js        |   90.40 |      659 |         0 |         70 |        0 |       0 |
 gather/gather-runner.js |   78.72 |      185 |         0 |         50 |        0 |       0 |
 runner.js               |   83.40 |      201 |         0 |         40 |        0 |       0 |
-------------------------|---------|----------|-----------|------------|----------|---------|
INFO Stryker Done in 269 minutes 20 seconds.

A good number of these are ignorable (a lot of mutating log statements that we don’t (and usually don’t want to) test), but there are definitely some things it caught that are useful.

I’m currently about halfway through running mutations of all the audits, which is projected to take about 24 hours in total 😃

Some other background:

• State of Mutation Testing at Google (2018)
• An Experimental Determination of Sufficient Mutant Operators (1996)

[paulirish]

Results for url-shim: https://extra-small-cactus.surge.sh/

[brendankenny]

results for lighthouse-core/lib/: https://unequaled-sink.surge.sh/

Good numbers overall, and not really a surprise that files like page-functions.js are a little low.

A ton of timeouts in this run, which stryker counts as a good thing (tests failed to complete -> mutation didn’t get around our tests). I set the timeout to 80s, though, so it really is likely that most of these timeouts were caught by the run never completing (e.g. switching loop conditionals in the dependency graph is a good way to end up looping forever).