Make functional tests more reliable
See original GitHub issueRegularly failing tests:
- BadSiteUnitTest: LOG_ERROR is not defined - test rewritten
- BadSiteUnitTest: ZAP_HUD_FILES is not defined
- BadSiteUnitTest: Expected condition failed: waiting for element
- FramesPageUnitTest
- PageAlertsPageUnitTest: 100 <1> but was: <0> - hopefully fixed?
Some of the functional tests (tutorial) fail sometimes, e.g.:
org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest > cannotAccessApiKeyWhenLoadingUtilsAsScript(FirefoxDriver) FAILED
org.openqa.selenium.JavascriptException: ReferenceError: LOG_ERROR is not defined
Build info: version: '3.13.0', revision: '2f0d292', time: '2018-06-25T15:24:21.231Z'
System info: host: 'travis-job-acbae740-7b82-4d89-9d65-e682edd3a1f2', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-101-generic', java.version: '1.8.0_151'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 64.0, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.23.0, moz:headless: true, moz:processID: 5738, moz:profile: /tmp/rust_mozprofile.RP2aKE..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.4.0-101-generic, rotatable: false, setWindowRect: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
Session ID: ad53072a-f6f6-4e5e-a008-bf11c07ff06c
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485)
at org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest.cannotAccessApiKeyWhenLoadingUtilsAsScript(BadSiteUnitTest.java:132)
org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest > cannotAccessApiKeyWhenLoadingUtilsAsScript(FirefoxDriver) FAILED
org.openqa.selenium.TimeoutException: Expected condition failed: waiting for element ([[FirefoxDriver: firefox on LINUX (66975787-1a03-4eae-b8d0-8a28b235e83f)] -> id: left-panel]) to become stale (tried for 20 second(s) with 500 milliseconds interval)
Build info: version: '3.13.0', revision: '2f0d292', time: '2018-06-25T15:24:21.231Z'
System info: host: 'travis-job-9f26b00f-c973-416a-a13c-86c968b31ddd', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-101-generic', java.version: '11.0.1'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 64.0, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.23.0, moz:headless: true, moz:processID: 5516, moz:profile: /tmp/rust_mozprofile.3VCarq..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.4.0-101-generic, rotatable: false, setWindowRect: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
Session ID: 66975787-1a03-4eae-b8d0-8a28b235e83f
at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:81)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:271)
at org.zaproxy.zap.extension.hud.ui.uimap.HUD.openUrlWaitForHud(HUD.java:160)
at org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest.cannotAccessApiKeyWhenLoadingUtilsAsScript(BadSiteUnitTest.java:127)
org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest > testSidePanelsHiddenAndRevealed(FirefoxDriver) FAILED
org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:40)
at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:35)
at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:185)
at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.checkPanelHidden(FramesPageUnitTest.java:147)
at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanesHidden(FramesPageUnitTest.java:153)
at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanelsHiddenAndRevealed(FramesPageUnitTest.java:108)
org.zaproxy.zap.extension.hud.ui.firefox.tutorial.PageAlertsPageUnitTest > testTaskAndNextButton(FirefoxDriver) FAILED
org.opentest4j.AssertionFailedError: expected: <1> but was: <0>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:54)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:195)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:184)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:179)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:508)
at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.PageAlertsPageUnitTest.testTaskAndNextButton(PageAlertsPageUnitTest.java:100)
(The above stack traces might not be exactly the same in develop, but the assertions should be the same.)
Issue Analytics
- State:
- Created 5 years ago
- Comments:14 (11 by maintainers)
Top Results From Across the Web
Best Practices for Functional Testing | Global App Testing
Read our guide to discover the differences between non-functional and functional software testing, how to test software functionality and best practices.
Read more >Best Practices for Functional Testing - Test IO
Develop reusable test cases. It can be a lengthy process to develop test cases, making them reusable helps to spare time and reduce...
Read more >Functional Testing: A Complete Guide with Types and Example
An In-Depth Comprehensive Functional Testing Tutorial with Types, Techniques, and Examples: What is Functional Testing?
Read more >Functional vs non-functional software testing - CircleCI
Non-functional testing assesses application properties that aren't critical to functionality but contribute to the end-user experience.
Read more >What is Functional Testing? Types & Examples - Guru99
FUNCTIONAL TESTING is a type of software testing that validates the software system against the functional requirements/specifications.
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Just added a checklist to the first comment
The change in #425 addresses a failure in
FramesPageUnitTest
that happened when waiting for the panels to be hidden/visible (because of the overlap it could open the Alerts dialogue instead of pressing the button to hide/show the frames).