Reliable FCS benchmarks which run on every release

We need to come up with reliable FCS benchmarks, and run them as part of dotnet/performance, ideally every release (on every signed build for current release branch).

Some ideas for scenarios we want to test:

• VS related (may be better solved with telemetry, rather than benchmarks):
  • Cold:
    • Time to colouring and breadcrumbs in the newly built projects of different sizes.
    • Time to intellisense in an arbitrary document in a newly build project of different sizes.
    • Time to build, time to rebuild.
    • Time to produce diagnostics on error (type errors, syntax errors, etc).
    • Time to full typecheck of project graphs of both connected and independent.
  • Hot (re-check):
    • Same as above but in a prebuilt project of various sizes.
• General (FCS):
  • Static tests:
    • Typechecking of projects of various sizes.
    • Typechecking of project graphs of both connected and independent.
  • Runtime tests (pretty much testing our codegen).
    • Time to main, we need to include different “configs” - ngen’d/crossgen’d, w/ and w/o mibc.

Some metrics we are interested in: time spent in scenarios, allocations (how much do we promote, is there anything ending up in LOH, etc), time spent in GC, etc.

More info and context: https://github.com/dotnet/fsharp/discussions/12526 & https://github.com/dotnet/fsharp/pull/13419

[0101]

@kerams it should be running now, and data being collected. Still have to figure out how to get to them.

[0101]

I created some parsing and type checking benchmarks on FSharpPlus and FsToolkit.ErrorHandling which take reasonable amount of time but should also hopefully catch most regressions.

https://github.com/dotnet/performance/pull/2592

They can be run locally, with local build of FSC plugged in via project file. I’ll think about making that easier somehow.