Use text files to maintain test definitions

Context

One important aspect of automated tests is that they are part of the application. So, when the application changes, some tests might change as well. It is important that those changes can be reviewed by peers to ensure it’s testing what needs to be tested. However, if we keep application code and test definitions apart, verifying those changes becomes a challenge.

Therefore, it is very important that the test definitions can be versioned in the same repository as the application. This allows our users to be able to:

1. Keep track of all changes in the test suite;
2. Review changes before they are applied to the project;

I think it would be extremely beneficial to the project if we stored all test definitions in a text format (probably YAML) so our users could have all benefits of using git.

Proposed solution

Instead of using a database to store all definitions, we start using yaml files instead. At the current state of the application, we could represent all our resources using the following yaml schemas:

# Test schema
nameOfTheResource:
    name: string
    execution:
        endpoint: string
        method: string
        headers:
            - name: string
              value: string
        body: string
    assertions:
        - name: string
          attribute: string
          comparison: string
          value: string

The problem is that the definition file might become huge if we store all tests and definitions in the same place. We can improve that experience by enabling the user to define multiple files and import them from the main file.

Let’s imagine we organize our files like this:

my_project/
├─ tracetest.yaml
├─ tracetest/
│  ├─ tests/
│  │  ├─ create_user_test.yaml
│  │  ├─ update_user_test.yaml
│  │  ├─ get_user_test.yaml
│  │  ├─ delete_user_test.yaml

Considering that tracetest.yaml is the main definitions file. We could define it as:

# my_project/tracetest.yml
tracetest:
    config:
        # Tracetest would scan all files in this folder recursively and generate 
        # one big file by merging all files from that directory into one.
        tracetestDir: ./tracetest/ 
    tests:
        - createUserTest
        - getUserTest
        - updateUserTest
        - deleteUserTest



# my_project/tracetest/tests/create_user_test.yml
createUserTest:
    # Test definitions here


# my_project/tracetest/tests/update_user_test.yml
updateUserTest:
    # Test definitions here


# my_project/tracetest/tests/delete_user_test.yml
deleteUserTest:
    # Test definitions here


# my_project/tracetest/tests/get_user_test.yml
getUserTest:
    # Test definitions here

#159 proposes the creation of a new type of resource. Referencing existing resources are extremely important for that idea to work.

Note: This approach allows us to reuse configuration files if needed. E.g: using the same spanSelector to more than one test.

Test execution

Execution of tests would still happen in the server backend because it will need to be able to fetch logs from jaeger or tempo in order to work. Those resources usually live inside a k8s cluster and are not accessible from the internet. Thus, the backend would have to be running inside that cluster in order to connect to those services.

To be continued.

Questions to be answered

1. How can we make we won’t kill our UI by using text-based definitions?

[schoren]

That is a good point @mathnogueira. I don’t like the terraform state approach, because it adds a lot of complexity to the CLI.

I think that for a first version we can get away with having the UUID as part of the file. I imagine the most common case at the beginning would be to create the test in the UI, and then export it to track it as part of a gitops or similar flow.

[schoren]

We did some work on implementing the selectors/assertions in the backend. To apply assertions, we implemented the follwing sturcture:

testDefinition:
  "selector": [
    "assertion 1",
    "assertion 2"
    ....
  ]

That structure is internal to the assertions package, but could easily be translated to a yaml file:

testName:
  span[tracetest.span.type="http"]:  // selector
    - tracetest.span.duration < 2000 // assertion 1
    - http.response.status_code = 200  // assertion 1

Adding the triggerting transaction info might look like this:

testName:
  trigger:
    method: POST
    url: http://example.com/do/stuff
    body: |
       {
         "key": "value",
         "foo": "bar"
       }
    headers:
      - Content-Type: application/json 
  span[tracetest.span.type="http"]:  // selector
    - tracetest.span.duration < 2000 // assertion 1
    - http.response.status_code = 200  // assertion 1