Proposal: Support for Cross Browser Testing

UPDATES: February 7th, 2020

📦 We have added support for Firefox and MS Edge in Cypress v4.0.0.

• 📣 Read Announcement
• 📘 Read Cross Browser Testing Guide
• 📋 Changelog
• 🛠 Migration Guide

UPDATES: Oct 20th, 2018

I gave more updates on the work we’ve been doing for cross browser support and other features on a webinar we gave.

https://youtu.be/FfqD1ExUGlw?t=2079

UPDATES: Sept 8th, 2018

Cross browser testing is now a front and center major focus here at Cypress. We’ve been experimenting over the last few months and have a working implementation to add native event support for Chrome, Firefox, Edge, and IE11. We also believe adding Mobile Safari and Android should also be straightforward - with Safari actually being the most difficult to get.

The architecture by which we are doing cross browser testing is a best-of-breed design where we tap into the native API’s for Chrome and Firefox (avoiding the use of webdriver) and for Edge and IE11 we only use a minimal set of Webdriver API’s - none of which are the traditional sendKeys or value endpoints, which avoids the network latency problem entirely. We plan to eventually publish this architecture so people understand how Cypress is different than Webdriver. None of the other core components of the Cypress architecture are affected by this change, it is simply how to enable native events in a consistent manner across all browsers without introducing the latency / flaky / non-determinism of Webdriver.

In additional to all these things, we are also adding Sauce Labs support (at a minimum) and will also have a proof of concept prepared for that.

As we add support for native events we are subsequently adding support for the browsers at the same time to ensure all of our algorithms are normalized correctly. However, you’ll likely first see us release native event support in Chrome + Firefox, followed by IE11 sometime after that.

…below is the original post from 2016…

Does Cypress support cross browser testing?

Currently the answer is: No

Right now, we only support Chrome* variants such as Chrome, Chromium, and Canary.

Will you ever support other browsers?

Yes.

We will be able to support other Desktop Browsers and Mobile Browsers (using Webviews).

We have built Cypress from day 1 with cross browser testing in mind, and are very conscientious to ensure the API’s we write today will work in the future.

The Technical Details

Why do you only support Chrome today?

• Cypress taps into the underlying browser Automation API’s of Chrome to do things like modify cookies, take screenshots, focusing the browser window, etc.
• We also launch the browser using very specific command line flags to turn off things that get in the way of automated testing.
• We have synchronized these Automation API’s when Cypress runs headlessly (in Electron). This means running headlessly behaves identically to a real headed Chrome browser.

What do you mean by “Automation API’s”?

• Automation API’s describe the functions that a browser exposes to do things outside of the Javascript Sandbox.
• Currently Cypress comes with its own universal Driver which implements most of the cy commands using Javascript - in the same run loop as your application. This is superior in the sense that Cypress can understand, know, wait for and modify its behavior based on how your application reacts to its commands in real time.
• The downside is its impossible to fully automate a browser entirely in Javascript. Javascript is a security sandbox which restricts what you can do.
• To fully automate a browser you must extend beyond the sandbox and tap into the underlying Automation API’s which do not have these restrictions.

What will it take to support cross browsers?

• We have to port commands which need to extend outside of the sandbox to each other browser we are attempting to support.

How will you port these Automation API’s?

• Either manually by hand
• Or by using the existing browser drivers

What do you mean by “existing browser drivers”?

• Each browser has built its own driver to the Webdriver Spec.
• Here are some examples of drivers:
• Google Chrome Driver (ChromeDriver)
• Micosoft Edge Driver
• SafariDriver
• Mozilla GeckoDriver (Firefox)
• Internet Explorer Driver
• Appium

Why would you port these by hand?

• Because using the existing drivers comes with a host of problems.

What’s wrong with the existing browser drivers?

• They are built for and thus limited to the Webdriver spec.

What is the Webdriver spec?

• It’s a HTTP API spec which defines a common interface to drive all browsers consistently.
• It enables developers to abstract away the details of how each different browser implements its Automation API’s

What is the problem with Webdriver?

• It’s slow
• It’s async
• It’s stateless
• It’s exposed over an HTTP server
• It’s a lowest common denominator
• It’s limited in scope and features
• There are small inconsistencies
• There are various driver bugs

What do you mean by “lowest common denominator”?

• The spec is essentially limited to only including “all the things that every browser can do”.
• This leaves a tremendous amount of power and capability off the table that more modern browsers provide.

Can you give me some examples?

• In Webdriver you cannot clear all cookies for all domains - you can only clear cookies for the current domain context under test.
• There is no way to tell the browser to clear its cache.
• Chromedriver cannot run tests with Dev Tools open.
• Oftentimes the drivers may miss elements you’re trying to interact with.
• You cannot easily control the application under test.
• You don’t have native access to window, you’re limited to stringifying objects over the wire.

Does Cypress currently use Webdriver?

• No. We wrote our own universal driver.

What would be the benefit of using Webdriver?

• We could possibly use the existing drivers to port over various commands without having to write the manual integrations by hand.
• This would allow us to support cross browsers much faster.

If you use Webdriver won’t Cypress also inherit these limitations?

• In most cases - no.
• In some cases - possibly.

How can you avoid these limitations?

• By using our universal driver to do as much automation as possible and only go outside of the browser for a limited set of commands.
• The whole point of Cypress is to exceed the boundaries of the current testing ecosystem.
• Because we automate the browser with a completely different architecture we are in a position to extend beyond the Webdriver Spec and also normalize its behavior consistently across all browsers.

Can you give me an example of this?

• Cypress can clear all cookies of all domains instead of just the current domain under test.
• We perform work outside of the browser but then synchronize it seamlessly like with cy.request.
• We have many commands which work outside of the browser which don’t need to be ported.

But won’t you still be limited by Webdrivers flakiness?

• No. Even if we were to issue native events through Webdriver, because your application is always tested through Cypress, that means Cypress can respond synchronously to events that take place, and modify its behavior accordingly.
• The vast majority of commands do not ever have to leave the browser anyway.

Can you give me an example of what you’re talking about?

• Imagine this scenario: you have a modal opening which animates in, and you want to click the “Submit” button.
• In Webdriver, because its an HTTP API which often involves multiple network hops, there is usually a significant delay.
• Let’s assume you’re attempting to issue a native “click” to the submit button.
• When you issue the “click” command, the browser driver calculates the center coordinates of the button at say: {clientX: 50, clientY: 100}.
• It then issues the native click event with those coordinates.
• By the time your application “receives” this event, it could possibly “miss” the button altogether.
• Your test fails because your button was never clicked, and the modal was never dismissed.
• What happened?
• Because the modal was animating it, its coordinates were changing rapidly. Webdriver has no idea or concept of this, it only takes a coordinate snapshot at the moment in time and because it asynchronously dispatches the native event, it misses.

Why isn’t this a problem with Cypress?

• Because Cypress is in the same run loop as your application, its impossible for it to “miss”.
• It can synchrously run calculations, and it already automatically figures out if an element is animating beyond a certain threshold - and therefore wait until it stops.
• If it did issue a native event, it could detect that this event has yet to happen, and if the element moved from the coordinate, immediately attempt to recover by either canceling the click event or issuing another click at new coordinates.

When do you think you’ll support other browsers?

• We really don’t know, its not a priority for us right now.

Why isn’t it a priority? This is super important!

• We disagree, cross browser testing is not as important as you think it is.

…

The reasoning and examples behind this are a WIP. We’ll be updating this doc when they are ready.

[brian-mann]

We will be adding cross browser support likely by EOY. @chrisbreiding next project is that.

The support will be:

• Firefox
• Safari
• Edge

No IE support is planned. It is substantially more difficult.

[brian-mann]

First passing test running Cypress in Firefox…