New process and guidelines for integration tests (replaces `demo-project` and `homepage_sample_code` tests in GitLab CI

From a call with @tayloramurphy and @DouweM, we started thinking of replacing demo-project with a set of sample Meltano projects in meltano/meltano that can serve as integration tests, perhaps along with specific scripts that would run - such as the invoking of specific plugins.

I’m sure there are better and more sophisticated approaches, but one approach would be a convention have a meltano.yml, a script.sh, and a meltano.after.yml which should match to the results. Another approach would be a set of pytest tests that perform a series of operations against (a copy of?) the source meltano.yml file and confirm the results (or errors), possibly including specific telemetry events that are expected to be sent over the course of those script commands.

[pandemicsyn]

We’ve chatted a bit in the past about having more examples in the docs as well, and we had some issues with doc drift leading up 2.0. I’ve been thinking about that a bit and wanted to float an idea. Using a setup like @aaronsteers described, I think we can actually kill two birds with one stone.

We can get more examples for users - that are actually maintained and always work, and new and improved integration tests. The only real change would be that instead of having a script.sh file, we treat our test orchestrations as living and usable Markdown docs. Users can use them as examples, and we can use them as scripts to execute via mdsh.

Contrived example, but if you wanted test and document how to transition from using elt to run you might have something like:

# transition-from-elt-to-run.md

This example shows how to transition an `etl` task with a custom state-id to a `job` executed via `run`. 
To follow along with this example, download link to meltano yml to a fresh project and run:

```
meltano install
```

Then assuming you had an `elt` job invoked like so:

```shell
meltano elt --state-id=my-custom-id tap-gitlab target-postgres
```

You would first need to rename the id to match meltano's internal pattern:

```shell
meltano state copy my-custom-id tap-gitlab-to-target-postgres
```

Then you can create a job and execute it using `meltano run`:


```shell
meltano job add my-new-job --task="tap-gitlab target-postgres"
meltano run my-new-job
```

Compiling this via mdsh will yield a bash script that we can invoke from our integration tests:

meltano install
meltano elt --state-id=my-custom-id tap-gitlab target-postgres
meltano state copy my-custom-id tap-gitlab-to-target-postgres
meltano job add my-new-job --task="tap-gitlab target-postgres 
meltano run my-new-job

So we could structure our tests something like:

/meltano/docs/example-library
|- kitchen-sink - a canonical kitchen sink project that shows all options in one file
   |- meltano.yaml
   |- kitchen-sink.md
|- env-config-aliases
   | - meltano.yaml
   | - all-the-ways-to-configure-things-demo.md
|- performing-work
   | - meltano.yaml
   | - meltano-job-run-demo.md
... more examples and awesome-sauce demo's...

meltano/integration/example-library
|- kitchen-sink 
   |- expected-meltano.yaml
   |- validation.sh 
|- env-config-aliases
   | - expected-meltano.yaml
   | - validation.sh
| - performing-work
   | - expected-meltano.yaml
   | - expected-output.jsonl
   | - expected-log-line-matches.txt
   | - validation.sh
... more examples and awesome-sauce test's...

Where validation.sh compiles the md, executes it (just like the script.sh in @aaronsteers original proposal), and performs any additional needed validation steps (diffing files, greping log lines for matches, etc).

@aaronsteers @afolson curious how y’all feel about this. It would basically guarantee that our example library always works, and since all these come with a meltano.yml, users can download them and actually follow along.

[aaronsteers]

@pandemicsyn - Ditto what @tayloramurphy said. Your proposal checks all the boxes I was looking for (plus a few more):

1. Arbitrary starting point in the meltano.yml of each case.
2. A generic script which performs a series of actions (essentially a sequence of meltano cli commands).
3. A generic contract (validation.sh) that each sample can use for custom validation.
4. An ‘expected output’ version of the meltano.yml that ensures the project is exactly as we expect it to be after all operations are completed. (For instance, would catch config landing in the wrong place even if all other behaviors were functioning as expected.)
5. We are able to use both the ‘before’ and ‘after’ versions of meltano.yml to validate against JSON Schema rules.
6. The examples and tests are both trivial in terms of effort to maintain and create, and can be reasonably included in the definition of done for all new features. (Basically these could also be the demo script.)
7. Tests are fast and easy to parallelize.
8. From all of the above in place, it would be trivial to adapt a TDD driven development approach, where we (or Product) write the examples and test cases before development even begins. 🎉

In short, I love it. I especially love your proposal to use something like mdsh, since that basically makes these dual-purpose tests and tutorials. 👌