CLI without ID proposal

Our current CLI relies on test IDs to run (instead of create and run) tests, so it saves the test ID to the definition file. This makes the tests non portable; meaning, a test created on one instance will fail to run on a different instance.

To fix this, we can add a name that acts as a user defined ID. This will exist in parallel to our current Test ID, which is used in the web UI.

Example:

To run a new test:

tracetest test run --definition ./file.yml --name 'my-unique-name'

This will result in a new test being created, with the User Defined ID set to my-unique-name. The ./file.yml definition file is left untouched. Successive calls to the same server will result in a test run without a new test being created.

Now, if the same command is executed with the CLI pointing to a different instance, the same process will repeat: first call will create a new test, successive calls creates new runs for the same test.

A name must be unique within an instance, and be a valid, one line string. It will be used to uniquely identify a test in a way that is agnostic to the instance.

Example of valid names:

• some test name
• some-test-name
• represent/some/dir/structure

Run an entire directory

With this new approach, we could have a run -r ./somedir/ command, that can recursively read a directory, and create a run for each yaml file found, using the file name without the extension as the name

Example structure:

tracetest/
|- cart
|  |- add-to-cart.yaml
|  |- checkout.yaml
|- signup
|  |- success.yaml
|  |- error.yaml

Example command:

tracetest run -r ./tracetest/

This command would be equivalent to running the following commands:

tracetest run --definition ./tracetest/cart/add-to-cart.yaml --name cart/add-to-cart
tracetest run --definition ./tracetest/cart/checkout.yaml --name cart/checkout
tracetest run --definition ./tracetest/signup/success.yaml --name signup/success
tracetest run --definition ./tracetest/signup/error.yaml --name signup/error

[schoren]

1. name: I think this was not a great word choice on my part. My idea was to left the current implementation untouched, and just add new things. So, this new name should NOT be the same as the current, web displayed name. I thought of using id, but that still conflicts with existing fields. Maybe we could use key?

2. version: text files don’t support version at the moment, and I think it should stay that way. The idea of text files is to enable git based workflows, so running a text file should update the test version, and text files should always be considered “latest”. I don’t imagine a workflow where a user would want to run an older version of a test from the CLI

[kdhamric]

I would think we would allow a user to specify either the id or the name to run a test via the command line.