Support for "internal variables" in tests

Example case:

I am testing a create product flow. It starts with an API call, which triggers a DB insert. The new ID is then passed to some other internal services. I want to assert that all traces after the DB insert include that new ID.

Our current implementation does not allow for this, because within an assertion, you can only reference attributes in the matching span:

id: VZvc1g44g
name: Test Create
trigger: # ...
specs:
# all spans should have the created ID
- selector: span[]
  assertions:
    - attr:myapp.created_id = ? # whatever we use here, will reference the current span

Proposals:

Use outputs internally.

If the outputs are processed before the specs, we could do something like this:

id: VZvc1g44g
name: Test Create
trigger: # ...

outputs:
- name: createdID
  selector: span[name="db insert']
  value: attr:myapp.created_id

specs:
# all spans should have the created ID
- selector: span[]
  assertions:
    - attr:myapp.created_id = ${env:createdID}

This would work fine on isolation, but would create (and maybe even override) an Environment Variable when running in the context of a transaction.

Add a variables section

Variables would work as outputs scoped to a test. The difference is that instead of being exported to the env when running in a transaction, these would only live within a test.

id: VZvc1g44g
name: Test Create
trigger: # ...

outputs:
- name: THIS_IS_EXPORTED
  selector: span[name="root']
  value: attr:myapp.some_value

variables:
- name: createdID # this is not exported
  selector: span[name="db insert']
  value: attr:myapp.created_id

specs:
# all spans should have the created ID
- selector: span[]
  assertions:
    - attr:myapp.created_id = ${var:createdID}
    - attr:myapp.other_attr = ${env:THIS_IS_EXPORTED} # exported could be referenced here too

Other possible implementations:

Exportable variables

Replace outputs with variables, add an export bool flag:

variables:
- name: createdID
  export: false # could be the default
  selector: span[name="db insert']
  value: attr:myapp.created_id
- name: THIS_IS_EXPORTED
  export: true
  selector: span[name="root']
  value: attr:myapp.some_value


specs:
# all spans should have the created ID
- selector: span[]
  assertions:
    - attr:myapp.created_id = ${var:createdID}
    # exported variables needs to be referenced via `env` to be consistent with consuming exported vars from other tests
    - attr:myapp.other_attr = ${env:THIS_IS_EXPORTED} 

Outputs are reference to variables

This is similar to terraform’s module outputs. In terraform, a module can define all the internal variables it wants, but only the defined outputs can be referenced from other modules.

In our case, outputs values would be merged with the env variables for the next transaction.

variables:
- name: createdID
  selector: span[name="db insert']
  value: attr:myapp.created_id
- name: someValue
  selector: span[name="root']
  value: attr:myapp.some_value

outputs:
- name: THIS_IS_EXPORTED
  variable: someValue

specs:
# all spans should have the created ID
- selector: span[]
  assertions:
    - attr:myapp.created_id = ${var:createdID}
    # exported variables needs to be referenced via `env` to be consistent with consuming exported vars from other tests
    - attr:myapp.other_attr = ${env:THIS_IS_EXPORTED} 

[schoren]

I think that flow makes the most sense of any options. In that case, your tests can only reference internal variables or env vars. A definition would look like this:

id: VZvc1g44g
name: Test Create
trigger: # ...

variables:
- name: createdID # this is not exported
  selector: span[name="db insert']
  value: attr:myapp.created_id

specs:
# all spans should have the created ID
- selector: span[]
  assertions:
    - attr:myapp.created_id = ${var:createdID}
    # - attr:myapp.other_attr = ${env:THIS_IS_EXPORTED} # not valid, this value is not set until end of test

outputs:
- name: THIS_IS_EXPORTED
  selector: span[name="root']
  value: attr:myapp.some_value

# allow using vars here too
- name: CREATED_ID
  value: var:createdID 

Should we allow the last output syntax? It enables value reusing between vars/outputs, not sure if it has any drawbacks.

[mathnogueira]

I don’t think there are issues with it because those variables are read-only during the assertion phase. I’d say: if it’s easy, let’s allow it, otherwise let’s see how it feels without it and if it’s too verbose, we add it.