Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Experiment freezes after it's been interrupted by the safeguards

See original GitHub issue

Describe the bug When the tolerance of one of the probes is falsy, sometimes the experiment just freezes. It’s unreliable.

Runtime versions

$ python --version
Python 3.8.0

$ chaos info core
NAME                VERSION   
CLI                 1.9.0     
Core library        1.19.0  

$ chaos info extensions
NAME                                    VERSION   LICENSE                       DESCRIPTION                                       
chaostoolkit-addons                     0.1.3     UNKNOWN                       Addons for your Chaos Toolkit experiments

To Reproduce

Create a exp.json
Put this code in there:

{
  "version": "1.0.0",
  "title": "test",
  "description": "test",
  "configuration": {
    "myarg": "exp.json",
    "another_arg": "another.json"
  },
  "tags": [
    "network",
    "lala"
  ],
  "controls": [
    {
      "name": "Safeguards",
      "provider": {
        "type": "python",
        "module": "chaosaddons.controls.safeguards",
        "arguments": {
          "probes": [
            {
              "name": "My probe",
              "description": "My best probe",
              "tolerance": {
                "type": "regex",
                "pattern": "exp.jsonnn",
                "target": "stdout"
              },
              "type": "probe",
              "provider": {
                "type": "process",
                "path": "echo",
                "arguments": "${myarg}"
              },
              "pauses": {
                "after": 1
              }
            }
          ]
        }
      }
    }
  ],
  "steady-state-hypothesis": {
    "title": "Check something",
    "probes": [
      {
        "name": "My probe",
        "description": "My best probe",
        "tolerance": {
          "type": "regex",
          "pattern": "exp.json",
          "target": "stdout"
        },
        "type": "probe",
        "provider": {
          "type": "process",
          "path": "echo",
          "arguments": "${myarg}"
        }
      }
    ]
  },
  "method": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      },
      "pauses": {
        "after": 1
      }
    }
  ],
  "rollbacks": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      },
      "pauses": {
        "after": 1
      }
    }
  ]
}

Run command multiple times chaos run exp.json --rollback-strategy always

Expected behavior It should complete the experiment.

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:13 (7 by maintainers)

Top GitHub Comments

1reaction

Lawouachcommented, Mar 3, 2021

I managed to reproduce this but this may take some work to figure what’s going on. Remember that the harsh interruption provided by CTK is not deterministic because it interacts with the Python VM in a way that isn’t friendly.

I need to see if I can profile this piece of code.

0reactions

CharlieMoon37commented, Aug 6, 2021

Issue resolved, branch merged but issue was not closed

Top Results From Across the Web

Chapter 5-Protecting Your System: Physical Security, from ...

Note that it's not freezing temperatures that damage disks, but the condensation that forms when they thaw out.

IAEA SAFEGUARDS GLOSSARY

Since 1987, IAEA safeguards have become more effective and efficient, mainly through the series of strengthening measures approved by the IAEA Board of....

The victims of unethical human experiments and coerced ...

The project discounted claims of abuse when no experiment or research was involved, or when victims having misunderstood compensation schemes for experiments ......

Coronavirus: Covid News: Israeli Study Says Second Booster ...

But scientists criticized the study's methodology, and the benefits of a second booster for young, healthy adults are less clear. Some experts ...

How Texas failed to protect its power grid against extreme ...

Three years after the 2011 storms, the Texas electric grid faced another major cold weather test when a polar vortex swept across the...