`projen test` updates snapshots by default

Given a TypeScriptAppProject, yarn test does the following:

$ npx projen test
....
🤖 test | jest --passWithNoTests --all --updateSnapshot

The --updateSnapshot is dangerous: if I run this in CI, yarn test will not actually validate my snapshots, but happily update them in place even if my tests should by all rights have failed.

The answer is not “anti-tamper protection”: that will tell me that files have changed, it will not give me the nice error message and diff that jest would have given me if the snapshot test was allowed to fail in the first place. Plus, you can’t assume that the tests will always be run with the synthesized GitHub workflow that includes anti-tamper protection: it might equally well run in a CodePipeline/CodeBuild setup, and that job might not even have access to the git repo to compare for any diffs.

[5nafu]

@jtnz

Today I tried improving on @caviliar 's solution with:

Thanks for your suggestion, but unfortunately, this did not work for us. In the latest Version of projen, the step.exec is read only.

We used your suggestion and migrated it to:

// Create new task eslint:update that has the same exec as eslint
const eslintTask = project.tasks.tryFind('eslint');
const eslintUpdate = project.tasks.addTask('eslint:update', {
  exec: eslintTask!.steps[0]!.exec!,
});
const testUpdate = project.tasks.tryFind('test:update');
testUpdate!.spawn(eslintUpdate);

// Remove '--fix' (write) of eslint task
const newEslintCommand = eslintTask!.steps[0]!.exec!.replace(' --fix', '');
eslintTask!.reset(newEslintCommand);

const testTask = project.tasks.tryFind('test');
// Remove '--updateSnapshot' from test task
const newTestCommand = testTask!.steps[0]!.exec!.replace(' --updateSnapshot', '');
console.log(newTestCommand);
testTask!.reset(newTestCommand);
testTask!.spawn(eslintTask!);

Additionally for others trying this out, please add it before project.synth(); 😁

[Chriscbr]

Here’s a proposal - let me know if this makes sense.

Current way things work:

1. There’s a test task which intentionally updates files. There’s also a test:update task (in JS-based project types), but it currently does the same thing.
2. The mutableBuild option determines whether to add a GitHub workflow step to push snapshot changes / eslint fixes back to your PR as a new commit (if you’re not using GitHub, it does nothing).

Proposed way things should work:

1. test task does not intentionally update files. test:update task does intentionally update files (by passing the appropriate flags to jest and eslint etc).
2. The mutableBuild option is split into mutableBuildTask and mutableBuildWorkflow. mutableBuildTask will determine whether the default build task runs test or test:update (irrespective of whether you’re running them locally or in CI/CD). mutableBuildWorkflow will do what mutableBuild did, but will also do nothing if mutableBuildTask is false.

In the future, mutableBuildTask: false should also cause running npx projen in CICD to fail immediately if it detects any configuration files that need to be updated, but this requires some more technical changes to projen’s synthesis engine.