question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Make functional tests more reliable

See original GitHub issue

Regularly failing tests:

  • BadSiteUnitTest: LOG_ERROR is not defined - test rewritten
  • BadSiteUnitTest: ZAP_HUD_FILES is not defined
  • BadSiteUnitTest: Expected condition failed: waiting for element
  • FramesPageUnitTest
  • PageAlertsPageUnitTest: 100 <1> but was: <0> - hopefully fixed?

Some of the functional tests (tutorial) fail sometimes, e.g.:

org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest > cannotAccessApiKeyWhenLoadingUtilsAsScript(FirefoxDriver) FAILED
    org.openqa.selenium.JavascriptException: ReferenceError: LOG_ERROR is not defined
    Build info: version: '3.13.0', revision: '2f0d292', time: '2018-06-25T15:24:21.231Z'
    System info: host: 'travis-job-acbae740-7b82-4d89-9d65-e682edd3a1f2', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-101-generic', java.version: '1.8.0_151'
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 64.0, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.23.0, moz:headless: true, moz:processID: 5738, moz:profile: /tmp/rust_mozprofile.RP2aKE..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.4.0-101-generic, rotatable: false, setWindowRect: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
    Session ID: ad53072a-f6f6-4e5e-a008-bf11c07ff06c
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
        at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
        at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
        at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
        at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
        at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:548)
        at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:485)
        at org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest.cannotAccessApiKeyWhenLoadingUtilsAsScript(BadSiteUnitTest.java:132)
org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest > cannotAccessApiKeyWhenLoadingUtilsAsScript(FirefoxDriver) FAILED
    org.openqa.selenium.TimeoutException: Expected condition failed: waiting for element ([[FirefoxDriver: firefox on LINUX (66975787-1a03-4eae-b8d0-8a28b235e83f)] -> id: left-panel]) to become stale (tried for 20 second(s) with 500 milliseconds interval)
    Build info: version: '3.13.0', revision: '2f0d292', time: '2018-06-25T15:24:21.231Z'
    System info: host: 'travis-job-9f26b00f-c973-416a-a13c-86c968b31ddd', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-101-generic', java.version: '11.0.1'
    Driver info: org.openqa.selenium.firefox.FirefoxDriver
    Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 64.0, javascriptEnabled: true, moz:accessibilityChecks: false, moz:geckodriverVersion: 0.23.0, moz:headless: true, moz:processID: 5516, moz:profile: /tmp/rust_mozprofile.3VCarq..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.4.0-101-generic, rotatable: false, setWindowRect: true, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
    Session ID: 66975787-1a03-4eae-b8d0-8a28b235e83f
        at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:81)
        at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:271)
        at org.zaproxy.zap.extension.hud.ui.uimap.HUD.openUrlWaitForHud(HUD.java:160)
        at org.zaproxy.zap.extension.hud.ui.firefox.badsite.BadSiteUnitTest.cannotAccessApiKeyWhenLoadingUtilsAsScript(BadSiteUnitTest.java:127)
org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest > testSidePanelsHiddenAndRevealed(FirefoxDriver) FAILED
    org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
        at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:40)
        at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:35)
        at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:185)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.checkPanelHidden(FramesPageUnitTest.java:147)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanesHidden(FramesPageUnitTest.java:153)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.FramesPageUnitTest.testSidePanelsHiddenAndRevealed(FramesPageUnitTest.java:108)
org.zaproxy.zap.extension.hud.ui.firefox.tutorial.PageAlertsPageUnitTest > testTaskAndNextButton(FirefoxDriver) FAILED
    org.opentest4j.AssertionFailedError: expected: <1> but was: <0>
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:54)
        at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:195)
        at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:184)
        at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:179)
        at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:508)
        at org.zaproxy.zap.extension.hud.ui.firefox.tutorial.PageAlertsPageUnitTest.testTaskAndNextButton(PageAlertsPageUnitTest.java:100)

(The above stack traces might not be exactly the same in develop, but the assertions should be the same.)

Issue Analytics

  • State:open
  • Created 5 years ago
  • Comments:14 (11 by maintainers)

github_iconTop GitHub Comments

1reaction
psiinoncommented, Feb 5, 2019

Just added a checklist to the first comment

0reactions
thc202commented, Feb 22, 2019

The change in #425 addresses a failure in FramesPageUnitTest that happened when waiting for the panels to be hidden/visible (because of the overlap it could open the Alerts dialogue instead of pressing the button to hide/show the frames).

Read more comments on GitHub >

github_iconTop Results From Across the Web

Best Practices for Functional Testing | Global App Testing
Read our guide to discover the differences between non-functional and functional software testing, how to test software functionality and best practices.
Read more >
Best Practices for Functional Testing - Test IO
Develop reusable test cases. It can be a lengthy process to develop test cases, making them reusable helps to spare time and reduce...
Read more >
Functional Testing: A Complete Guide with Types and Example
An In-Depth Comprehensive Functional Testing Tutorial with Types, Techniques, and Examples: What is Functional Testing?
Read more >
Functional vs non-functional software testing - CircleCI
Non-functional testing assesses application properties that aren't critical to functionality but contribute to the end-user experience.
Read more >
What is Functional Testing? Types & Examples - Guru99
FUNCTIONAL TESTING is a type of software testing that validates the software system against the functional requirements/specifications.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found