Improve trace pooling process

Problem

When Tracetest triggers a test and waits for the trace to be available, it executes a very simple condition to ensure the trace is complete. Today, the logic is very simple: if a trace is polled twice in a row and the number of spans didn’t change, consider it complete. However, this is not true for long-running operations.

With the current implementation, if we had one test that takes a long time to generate the complete trace, we would have to adjust the pooling configuration to make sure to add enough time between polling executions to prevent false-positive complete traces. However, this would slow down all other tests executed by that tracetest instance.

Possible solution

Enable the polling condition to be set at a test level

Instead of hardcoding the condition, we could let a test define when a trace is complete and can be asserted. If this is not configured in the test, we would use the default implementation.

Test with default polling configuration

When the polling object is not defined, the default polling strategy would be by counting the number of spans and check if they are equal. If they are equal in two subsequent polling executions, the trace is marked as complete.

name: My test
trigger:
    # how to trigger the test

spec:
    # assertions

Test using the same default polling strategy, but with different parameter

name: My test
trigger:
    # how to trigger the test

polling:
    strategy: sameSpanCount
    sameSpanCount:
        executions: 3 # if the number of spans is the same in 3 subsequent executions, the trace is marked as complete

spec:
    # assertions

Test using an assertion to stop polling

# slow-test.yaml
name: My slow test
trigger:
   # how to trigger the test

polling:
   # polling would only stop when a span named "insert pokemon into database" is present in the trace
   strategy: assert
   assert:
       selector: span[name = "insert pokemon into database"
       assertion: tracetest.selected_spans.count = 1

spec:
    # assertions

[kdhamric]

@olha23 Yes, we should. Eventually, we will allow people to define multiple profiles as given above. For that, they would need to be able to specify all the fields: name, strategy, executions, retry delay, and timeout. I am not sure when we will tackle this work - it would not hurt to begin to mock some of it.

[olha23]

@kdhamric from UI perspective we should allow users to define the number of times a trace can be polled before being marked as “complete” in the test configuration?