Preprocessor swallows errors and ignores ENOENT for bundled files

bug

Cypress 3.0.1 on Mac

I have been playing with a spec preprocessor based on Parcel and noticed a curious thing. If the bundler is configured incorrectly and actually does NOT bundle files but returns a non-existing path, the test runner swallows the error and happily exits with code 0, while showing (and ignoring) ENOENT error

Example output

  Running: spec.js...                                                                      (1 of 1) 
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js
basename spec
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js
basename index
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'


Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js'


Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'




  0 passing (2ms)


  (Results)

  ┌─────────────────────────┐
  │ Tests:        0         │
  │ Passing:      0         │
  │ Failing:      0         │
  │ Pending:      0         │
  │ Skipped:      0         │
  │ Screenshots:  0         │
  │ Video:        true      │
  │ Duration:     0 seconds │
  │ Spec Ran:     spec.js   │
  └─────────────────────────┘


  (Video)

  - Started processing:   Compressing to 32 CRF
  - Finished processing:  /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/videos/spec.js.mp4 (0 seconds)


====================================================================================================

  (Run Finished)


      Spec                                                Tests  Passing  Failing  Pending  Skipped 
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔ spec.js                                     3ms        -        -        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    All specs passed!                             3ms        -        -        -        -        -  

The bundles files are returned like this

const onFile = (file) => {
  const { filePath, shouldWatch, outputPath } = file
  if (!shouldWatch) {
    console.log('bundle file once without watching to %s', outputPath)
    return bundleOnce(filePath, outputPath).then(() => outputPath)
  }
  ...
}

Here is the relevant verbose logs

  cypress:server:socket watch test file integration/spec.js +0ms
  cypress:server:socket will watch test file path cypress/integration/spec.js +0ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js +0ms
  cypress:server:appdata path: /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +1ms
  cypress:server:plugins execute plugin event 'file:preprocessor' with args: EventEmitter { domain: null, _events: { rerun: [Function] }, _eventsCount: 1, _maxListeners: undefined, filePath: '/Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js', outputPath: '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js', shouldWatch: false } undefined undefined +0ms
  cypress:server:plugins call event file:preprocessor for invocation id inv1 +3ms
  cypress:server:plugins:child execute plugin event: file:preprocessor ({ eventId: 2, invocationId: 'inv1' }) +2s
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js
  cypress:server:controllers:spec request for cypress/support/index.js +12ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/support/index.js +1ms
  cypress:server:appdata path: /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js +0ms
  cypress:server:plugins execute plugin event 'file:preprocessor' with args: EventEmitter { domain: null, _events: { rerun: [Function] }, _eventsCount: 1, _maxListeners: undefined, filePath: '/Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/support/index.js', outputPath: '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js', shouldWatch: false } undefined undefined +0ms
  cypress:server:plugins call event file:preprocessor for invocation id inv2 +1ms
  cypress:server:controllers:spec request for cypress/integration/spec.js +0ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js +0ms
  cypress:server:preprocessor headless and already processed +0ms
  cypress:server:plugins:child execute plugin event: file:preprocessor ({ eventId: 2, invocationId: 'inv2' }) +52ms
bundle file once without watching to /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js
  cypress:server:plugins promise resolved for id 'inv1' with value /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +444ms
  cypress:server:controllers:spec send /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +0ms
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'


  cypress:server:plugins promise resolved for id 'inv2' with value /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js +5ms
  cypress:server:controllers:spec send /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js +0ms
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/support/index.js'


  cypress:server:controllers:spec request for cypress/integration/spec.js +2ms
  cypress:server:preprocessor getFile /Users/gleb/git/cypress-example-recipes/examples/blogs__parcel_preprocessor/cypress/integration/spec.js +0ms
  cypress:server:preprocessor headless and already processed +0ms
  cypress:server:controllers:spec send /Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js +0ms
Error: ENOENT: no such file or directory, stat '/Users/gleb/Library/Application Support/Cypress/cy/production/projects/blogs__parcel_preprocessor-677867adc8d7253c8b90ac225b617bcd/bundles/cypress/integration/spec.js'


  cypress:server:timers clearing timer id 11 from queue { '11': { args: [], ms: 85000, cb: [Function] } } +5ms
  cypress:server:timers queuing timer id 12 after 85000 ms +0ms
  cypress:server:timers child received timer id 12 +474ms
  cypress:server:timers clearing timer id 12 from queue { '12': { args: [], ms: 85000, cb: [Function] } } +9ms
  cypress:server:timers queuing timer id 13 after 85000 ms +0ms
  cypress:server:project onSetRunnables +1ms
  cypress:server:timers child received timer id 13 +10ms
  cypress:server:project runnables null +0ms
  cypress:server:reporter trying to load reporter: spec +0ms
  cypress:server:reporter spec is Mocha reporter +0ms
  cypress:server:timers clearing timer id 13 from queue { '13': { args: [], ms: 85000, cb: [Function] } } +12ms
  cypress:server:timers queuing timer id 14 after 85000 ms +0ms
  cypress:server:timers child received timer id 14 +12ms
  cypress:server:project onMocha start +0ms
  cypress:server:reporter got mocha event 'start' with args: [ { start: '2018-06-05T03:35:06.763Z' } ] +1ms

  cypress:server:timers clearing timer id 14 from queue { '14': { args: [], ms: 85000, cb: [Function] } } +2ms
  cypress:server:timers queuing timer id 15 after 85000 ms +0ms
  cypress:server:project onMocha end +0ms
  cypress:server:timers child received timer id 15 +3ms
  cypress:server:reporter got mocha event 'end' with args: [ { end: '2018-06-05T03:35:06.766Z' } ] +0ms

  0 passing (2ms)

the folder was there, just the file that was supposed to be there does not exist when we return it from preprocessor

[bahmutov]

Yes, but we are ignoring the error which is the wrong way to go about it. External code should be least trusted in this case.

Sent from my iPhone

On Jun 5, 2018, at 13:45, Chris Breiding [e-mail] wrote:

Yeah, I guess we don’t validate the path returned to make sure there’s a file there. So it ends up 404ing in the browser which results in 0 tests. I suppose we should fs.stat the output path returned and fail if it’s not there. But to some degree we need to trust that the preprocessor does the right thing (like erroring when it can’t bundle). Because even if the file is there, it could be empty or not javascript or just not the right thing that should have been produced by your source spec file.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[bahmutov]

Same problem happens on CI as well: https://circleci.com/gh/cypress-io/cypress-example-recipes/6174