Failing action should fail experiment

I have set up the following experiment:

{
    "version": "1.0.0",
    "title": "Memory leak",
    "description": "We shold be able to have a memoryleak in a pod",
    "tags": [
        "basecase"
    ],
    "steady-state-hypothesis": {
        "title": "All deployments are OK",
        "probes": [
            {
                "type": "probe",
                "tolerance": true,
                "provider": {
                  "type": "python",
                  "module": "chaosk8s.probes",
                  "func": "all_microservices_healthy"
                },
                "name": "all-microservices-healthy"
            }
        ]
    },
    "method": [
        {
            "type": "action",
            "provider": {
              "type": "python",
              "module": "chaosk8s.actions",
              "arguments": {
                "spec_path": "./memleak-deployment.yaml"
              },
              "func": "start_microservice"
            },
            "name": "start-microservice"
        }
    ],
    "rollbacks": [
        {
            "type": "action",
            "provider": {
              "type": "python",
              "module": "chaosk8s.actions",
              "arguments": {
                "name": "memory-leak"
              },
              "func": "kill_microservice"
            },
            "name": "kill-microservice"
          }
    ]
}```
For some reason reason, I get a failure in the "start_microservice" action (but that is a separate issue). The problem is that the experiment does not fail:

![image](https://user-images.githubusercontent.com/1544931/60712491-71927a80-9f17-11e9-95c5-0ea87b73cd94.png)


I expect it to fail, and it is important that it does.

[Lawouach]

I think there is some truth in that last statement. The trick is to define “failing condition” though. Is it failing because the system is broken? Is it failing because the action doesn’t work as it should (like buggy input…)?

How to evaluate the action is working in the expected boundaries from when it actually misbehaves?

I find that discussion quite exciting to be honest and it fits some larger discussion around taking the whole chaos experiment further via the Open Chaos project (https://openchaos.io/).

I’ll cc @russmiles on this one.

[github-actions[bot]]

This Issue was closed because it was not reactivated after 7 days of being marked Stale.