Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ensure not throwing error

See original GitHub issue

Setting I have a very basic configuration file with 3 ensure checks.

config:
  target: <omitted-target>
  phases:
    - duration: 20
      arrivalRate: 5
      rampTo: 10
      name: Sustained load
  ensure:
    p95: 100
    max: 200
    maxErrorRate: 1

scenarios:
  - name: "Visit Landing"
    flow:
      - get:
          url: "/"

I run artillery/2.0.0-10 win32-x64 node-v14.15.3.

What’s going wrong? Running this test I get the following report summary.

--------------------------------
Summary report @ 12:48:14(+0100)
--------------------------------

vusers.created_by_name.Visit Landing: .......................................... 148
vusers.created.total: .......................................................... 148
vusers.completed: .............................................................. 148
vusers.session_length:
  min: ......................................................................... 197.4
  max: ......................................................................... 422.1
  median: ...................................................................... 247.2
  p95: ......................................................................... 320.6
  p99: ......................................................................... 347.3
http.request_rate: ............................................................. 8/sec
http.requests: ................................................................. 148
http.codes.200: ................................................................ 148
http.responses: ................................................................ 148
http.response_time:
  min: ......................................................................... 101
  max: ......................................................................... 274
  median: ...................................................................... 138.4
  p95: ......................................................................... 194.4
  p99: ......................................................................... 242.3
Log file: test.json

As you can see the ensure condition p95 = 194.4 > 100 = ensure.p95 is violated. Unfortunately, there is no error thrown stating that the respective ensure condition has failed.

What’s expected? I would expect to get an error like ensure condition failed: ensure.p95 < 100

Issue Analytics

State:
Created 2 years ago
Comments:7 (2 by maintainers)

Top GitHub Comments

2reactions

hassycommented, Jan 25, 2022

@jclusso oops, a case of documentation being updated before a new release and going out of sync 🤦‍♂️ Could you try again with the latest version of artillery and artillery-plugin-ensure please?

Example script to try with:

config:
  target: "https://www.artillery.io"
  plugins:
    ensure: {}
  phases:
    - duration: 10
      arrivalRate: 1
  ensure:
    thresholds:
      - http.response_time.p95: 1000
    conditions:
      - expression: http.response_time.p99 < 100 and vusers.created <= 10
        strict: false
      - expression: vusers.failed == 0
      - expression: http.codes.200 > 10
    p99: 1000
    maxErrorRate: 0
scenarios:
  - flow:
      - get:
          url: "/"
      - get:
          url: "/docs"
      - get:
          url: "/integrations"

0reactions

lazysealscommented, Feb 24, 2022

Sorry for the late response. I can confirm that the issue is resolved with @hassy’s response. Thank you!

Top Results From Across the Web

How to test that no exception is thrown? - java - Stack Overflow

The first unit of work should test whether or not a valid user is being returned in the case of valid and invalid...

How to test a function that's expected to throw error in jest…

Say, I want to write a test for the function below and want to ensure I test if it actually fails when the...

Best Practices for exceptions - .NET - Microsoft Learn

Learn best practices for exceptions, such as using try/catch/finally, handling common conditions without exceptions, and using predefined .

10 Handling PL/SQL Errors

Exceptions cannot propagate across remote procedure calls done through database links. A PL/SQL block cannot catch an exception raised by a remote subprogram....

Error handling and automatic retries in AWS Lambda

Function – Your function's code throws an exception or returns an error object. ... If you retry, ensure that your function's code can...