Audit consistent error handling DX for all CT frameworks / dev severs

Issues created and their effect on GA

Issue	Needed for GA	Breaking
fix: angular and nuxt ct tests now fail on uncaught exceptions	x	(yes but not really since the frameworks are in alpha)
Normalize DevServer Config Resolution	?	x
Normalize DevServer Compilation Error Reporting
Hovering over mount in command log does not show the mounted component in AUT	x
Better CT onboarding experience for projects with ESLint
Add Export to commands.ts for projects with isolatedModules tsconfig flag
Nuxt auto import component does not work in Component Testing
Component name not displaying properly when importing Nuxt page component
CT Mount Audit	x	?

Errors

ESLint Warning

it("eslint warning", () => {
  5 + 5; // Unused expression
  cy.mount(<App />);
});

Cypress

Framework	Terminal Log	Console Log	Reporter Error	Code Frame	DevServer Overlay	Test Failed
angular
create-react-app	x
next
nuxt
react-vite
vue-cli

App

Framework	Terminal Log	Console Log	DevServer Overlay
angular
create-react-app	x
next
nuxt
react-vite
vue-cli

• create-react-app

ESLint Error

it("eslint error", () => {
  let thing = "Hello World!"; // Unused var
  cy.mount(<App />);
});

Cypress

Framework	Terminal Log	Console Log	Reporter Error	Code Frame	DevServer Overlay	Test Failed
angular
create-react-app	x
next
nuxt
react-vite
vue-cli

App

Framework	Terminal Log	Console Log	DevServer Overlay
angular
create-react-app	x		x
next
nuxt
react-vite
vue-cli

• create-react-app

Typescript Warning

it("typescript warning", () => {
  let thing: number = "Hello World"; // number is not assignable to string

  cy.mount(<App thing={thing} />);
});

Cypress

Framework	Terminal Log	Console Log	Reporter Error	Code Frame	DevServer Overlay
angular	x	x
create-react-app	x
next
nuxt	x	x
react-vite
vue-cli	x	x

App

Framework	Terminal Log	Console Log	DevServer Overlay
angular	x	x	x
create-react-app	x		x
next
nuxt	x
react-vite
vue-cli	x		x

• angular

• create-react-app

• nuxt

• vue-cli

Typescript Error (Compilation Error)

it('typescript error', () => {
  {}/()aa>

  cy.mount(<App />);
})

Cypress

Framework	Terminal Log	Console Log	Reporter Error	Code Frame	DevServer Overlay	Test Failed
angular	x	x	x			x
create-react-app	x		x			x
next	x	x	x	x		x
nuxt	x	x	x			x
react-vite	x		x		x	x
vue-cli	x	x	x			x

App

Framework	Terminal Log	Console Log	DevServer Overlay
angular	x	x	x
create-react-app	x		x
next	x	x	x
nuxt	x	x	x
react-vite	x	x	x
vue-cli	x		x

• angular

• create-react-app

• next

• nuxt

• react-vite

• vue-cli

Runtime Error (App)

it("runtime error in component", () => {
  cy.mount(<App />);
  cy.get("button").click(); // Wired up to button that throws error
});

Cypress

Framework	Terminal Log	Console Log	Reporter Error	Code Frame	DevServer Overlay	Test Failed
angular		x
create-react-app		x	x	x		x
next		x	x	x		x
nuxt		x
react-vite		x	x			x
vue-cli		x	x	x*		x

App

Framework	Terminal Log	Console Log	DevServer Overlay
angular		x
create-react-app		x
next		x	x
nuxt		x	x*
react-vite		x
vue-cli		x

• angular

• create-react-app

• next

• nuxt

• react-vite

• vue-cli

Runtime Error (Test)

it("runtime error in spec", () => {
  throw new Error("uh oh");
  cy.mount(<App />);
});

Cypress

Framework	Terminal Log	Console Log	Reporter Error	Code Frame	DevServer Overlay	Test Failed
angular			x	x		x
create-react-app			x	x		x
next			x	x		x
nuxt			x	x		x
react-vite			x	x		x
vue-cli			x	x		x

• angular

• create-react-app

• next

• nuxt

• react-vite

• vue-cli

Summary

Error Type	Summary
ESLint warning	ESLint warnings had no effect on the outcome of the test. create-react-app is the only framework that has ESLint integrated into the compilation. There are ENV variables that can be used to change this behavior (ESLINT_NO_DEV_ERRORS, DISABLE_ESLINT_PLUGIN), but the defaults align with the behavior of apps that don’t have ESLint integrated.
ESLint Error	ESLint errors had no effect on the outcome of the test.
Typescript Warning	Typescript Warnings had no effect on the outcomes of the test.
Typescript/Compilation Error	Typescript/Compilation Error caused every test to fail. Inline with expected behavior
Runtime Error (App)	Framework dependent. Desired behavior is that a runtime error in the component under test should fail the test, but Angular and Nuxt components swallowed the error.
Runtime Error (Test)	Runtime Errors caused all apps to report a failed test. Inline with expected behavior.

Webpack vs vite

The behavior of Webpack and Vite were functionally the same (disregarding framework outliers). The only difference noted was the error reporting due to compilation errors:

• Webpack: Reports error in Reporter, AUT is blank
• Vite: Reports error in Reporter, AUT has error overlay
  • The error in the reporter for Vite is barebones e.g. Failed to request module. The error in the AUT overlay contains what actually went wrong.

On compilation error, Webpack will: serve the asset containing the error, but replace the modules contents with throw new Error(<node-stack-trace>).

On compilations error, Vite will: not serve the asset meaning the request will fail. The error is sent to the frontend via websocket and displayed in the overlay.

Though the behavior differs, the content is the same. Normalizing this behavior will not result in a breaking change.

Outliers

• Angular: Runtime error in app did not cause test to fail
• Nuxt: Runtime error in app did not cause test to fail
• Vue-CLI: Relative Code Frame path did not link to correct location
• Next: Code-frame was shown for compilation error

DevServer API

Webpack DevServer Config API:

type LowLevelFrameworks = 'react' | 'vue' | 'svelte'
type HighLevelFrameworks = 'angular' | 'create-react-app' | 'next' | 'nuxt' | 'vue-cli'

interface WebpackDevServerConfig {
  framework: LowLevelFrameworks | HighLevelFrameworks,
  bundler: 'webpack',
  webpackConfig: any,
  options?: Record<any, any>
}

Vite DevServer Config API:

type LowLevelFrameworks = 'react' | 'vue' | 'svelte'

interface ViteDevServerConfig {
  framework: LowLevelFrameworks
  bundler: 'vite',
  viteConfig: any
}

Config Resolution

An inconsistency in how configs are sourced was found between webpack-dev-server and vite-dev-server when using the low-level frameworks (vite-dev-server does not have support for higher order frameworks, but will soon). The config resolution is outlined below for each:

• Webpack:
  • devServer.webpackConfig = {} | undefined
  • if (!devServer.webpackConfig) devServer.webpackConfig = require(‘<project-root>/webpack.config.js’)
  • merge(webpack.devServerConfig, cypressWebpackConfig)
• Vite:
  • devServer.viteConfig = {} | undefined
  • let userConfig = require(‘<project-root>/vite.config.js’)
  • merge(userConfig, viteConfig, cypressViteConfig)

They look similar, but differ by how the auto-detected webpack/vite config is used. If a user provides a devServer.webpackConfig property, we will not detect or merge their <project-root>/webpack.config.js. If a user provides a devServer.viteConfig, these options will be merged with their auto-detected <project-root>/webpack.config.js.

webpackConfig/viteConfig Type

Currently, these are types as any. Some users have put strings here, thinking Cypress will resolve a file-system path for them. An improvement could be Record<any, any> to at least denote it is an object.

The outcome should be making a recommendation on how to address the consistency in the next sprint.

Onboarding

General Issues:

1. When hovering over Mount in the command log, the AUT shows a blank view. This should probably display the mounted component

angular

I did npm install -g @angular/cli. I made a new app and a simple component - works great. I included SCSS and Routing.

On-boarding worked as expected - all the files were generated in the correct places.

nuxt

Nuxt’s on-boarding experience did not have any configuration issues. Typescript and ESLint configurations did not need modified. It was able to utilize Create Spec from Vue Component.

The issues experienced during the on-boarding experience were:

1. Page shows component name in the command log as <f .../> instead of the component name (See nuxt screenshots above)
2. The user has to add any children component into the components object in the exported object. This is not required when running the app in dev mode.

vue-cli

Vue cli’s on-boarding experience had a couple steps to get eslint and typescript happy. Typescript config needed to be updated for TS projects by adding the cypress config to the tsconfig.json file. ESLint config needed to be updated for non typescript projects by adding the eslint-cypress-plugin library.

next

create-next-app

On-boarding Next using vanilla js did not have any configuration issues. Tests ran as expected on the first attempt.

create-next-app --typescript

create-next-app using typescript requires export {} to be added to commands.ts due to the tsconfig isolatedModules flag.

react

create-react-app

On-boarding React using vanilla js did not have any configuration issues. Tests ran as expected on the first attempt.

create-react-app --typescript

On-boarding React using typescript needed to add export {} to commands.ts due to the tsconfig isolatedModules flag. IDEs are not happy with jest + cypress testing. This may require reorganization and configuration changes if you need to keep existing jest tests in your project

[lmiller1990]

TODO: Verify what areas/things to do to verify (eg, what do we need to check).

[lmiller1990]

I’m yet to clarify if this is E2E or just CT, but we’ve got some common warnings in the console across all frameworks (and possibly E2E testing).

It would be nice to get rid of these (at least, the warnings and error) even if they are harmless.