Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Test failes when using negativ values in delta operator

See original GitHub issue

Description of Issue/Question

Delta values are not calculated correctly when using negative values e.g. -10% or postive values e.g +10% It looks like delta tests are behaving just like when using only the percentage 10%

Setup

(Please provide relevant configs, xml snapshots and related files (Be sure to remove sensitive info).)

Pre snapshot:

<route-summary-information>
<as-number>44745</as-number>
<router-id>10.255.255.250</router-id>
<route-table>
<table-name>inet.0</table-name>
<destination-count>141</destination-count>
<total-route-count>141</total-route-count>
<active-route-count>100</active-route-count>
<holddown-route-count>1</holddown-route-count>
<hidden-route-count>1</hidden-route-count>
<protocols>
<protocol-name>Direct</protocol-name>
<protocol-route-count>9</protocol-route-count>
<active-route-count>8</active-route-count>
</protocols>
<protocols>
<protocol-name>Local</protocol-name>
<protocol-route-count>7</protocol-route-count>
<active-route-count>7</active-route-count>
</protocols>
<protocols>
<protocol-name>OSPF</protocol-name>
<protocol-route-count>124</protocol-route-count>
<active-route-count>123</active-route-count>
</protocols>
<protocols>
<protocol-name>Static</protocol-name>
<protocol-route-count>1</protocol-route-count>
<active-route-count>1</active-route-count>
</protocols>
</route-table>
</route-summary-information>

Post snapshot:

<route-summary-information>
<as-number>44745</as-number>
<router-id>10.255.255.250</router-id>
<route-table>
<table-name>inet.0</table-name>
<destination-count>140</destination-count>
<total-route-count>140</total-route-count>
<active-route-count>111</active-route-count>
<holddown-route-count>0</holddown-route-count>
<hidden-route-count>1</hidden-route-count>
<protocols>
<protocol-name>Direct</protocol-name>
<protocol-route-count>9</protocol-route-count>
<active-route-count>8</active-route-count>
</protocols>
<protocols>
<protocol-name>Local</protocol-name>
<protocol-route-count>7</protocol-route-count>
<active-route-count>7</active-route-count>
</protocols>
<protocols>
<protocol-name>OSPF</protocol-name>
<protocol-route-count>123</protocol-route-count>
<active-route-count>123</active-route-count>
</protocols>
<protocols>
<protocol-name>Static</protocol-name>
<protocol-route-count>1</protocol-route-count>
<active-route-count>1</active-route-count>
</protocols>
</route-table>
</route-summary-information>

Testfile:

tests_include:
  - check_route_summary

check_route_summary:
- command: show route summary
- iterate:
    xpath: //route-summary-information/route-table
    id: ./table-name
    ignore-null: true
    tests:
      - delta: active-route-count, +10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
      - delta: active-route-count, -10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "

Config file:

hosts:
  - device: 10.1.1.1
    username : ansible
tests:
   - test_routes-2.yml

Test result:

jsnapy --port 22 --check pre post -f jsnapy-host.cfg -v

jsnapy.cfg file location used : /etc/jsnapy
Configuration file location used : /etc/jsnapy
*************************** Device: 10.1.1.1 ***************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <111.0> 
FAIL | All "active-route-count" is not with in delta difference of +10% [ 0 value matched / 1 value failed ]
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <111.0> 
FAIL | All "active-route-count" is not with in delta difference of -10% [ 0 value matched / 1 value failed ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Failed
Total No of tests passed: 0
Total No of tests failed: 2 
Overall Tests failed!!!

As you see, both tests are failing instead of just one which has the delta set to +10%

If I change the <active-route-count> in the post snapshot to <active-route-count>89</active-route-count> e.g -10% of the pre value, both tests fail again:

jsnapy --port 22 --check pre  post -f jsnapy-host.cfg -v
jsnapy.cfg file location used : /etc/jsnapy
Configuration file location used : /etc/jsnapy
*************************** Device: 10.1.1.1 ***************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <89.0> 
FAIL | All "active-route-count" is not with in delta difference of +10% [ 0 value matched / 1 value failed ]
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <89.0> 
FAIL | All "active-route-count" is not with in delta difference of -10% [ 0 value matched / 1 value failed ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Failed
Total No of tests passed: 0
Total No of tests failed: 2 
Overall Tests failed!!!

When using following testfile with only 10%:

tests_include:
  - check_route_summary

check_route_summary:
- command: show route summary
- iterate:
    xpath: //route-summary-information/route-table
    id: ./table-name
    ignore-null: true
    tests:
      - delta: active-route-count, 10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>,

the test fails for every value lower/higher then 10% from the pre snapshot value (here 100)