Deprecate and unlist package version 3.0.0 and change release habits

Last week we released new major version 3.0.0

Unfortunately the version has got an ugly bug has described by our amazing @daveMueller https://github.com/coverlet-coverage/coverlet/issues/1037#issuecomment-758365055

We did a great improvement on coverage testing, today we have a way to check if we’re able to recognize pattern and verify line numbers and branch numbers. I’m very happy of the quality of our coverage. With this version we’re very close to OpenCover as reported by @JSkimming https://github.com/coverlet-coverage/coverlet/issues/1037#issuecomment-759255949 one of the best coverage solution around.

Anyway the bug is escaped from our UTs and integrations because it manifests only on multi-document project and we’re testing 1 document at time(we check coverage for specific scenario and we generate 1 asm for 1 class and we tests only that class so only 1 cs file). Also offering nightly build to our friends didn’t help https://github.com/coverlet-coverage/coverlet/issues/1037#issuecomment-758031177 and I’m ok with it, it’s a nightly not an “official preview”.

In the spirit of “learning from our(my) errors” I propose to add one more step before official release. I think that when the new fix/improvements are “enough”(also if we have 1 big change) we should release an official preview to nuget and maybe ask to our friends of coverlet community to give it a try.

We could also open an coverlet Twitter account to ask some collaboration for preview testing.

Coverlet is becoming the choice for xplat .NET Coverage, we have a lot of download https://nugettrends.com/packages?months=12&ids=coverlet.collector&ids=coverlet.msbuild&ids=coverlet.console and users ~61K(github numbers) and we’re on MS docs https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage?tabs=windows with the privilege of VSTest strict integration --collect "XPlat Code Coverage", for all these reason I think we need to do a step further also on this part of “chain” to improve the quality and move @tonerdo’s baby to the adolescence😄

Please let me know what you think @tonerdo @petli

[petli]

After thinking a bit about this I think I agree with @JSkimming. In general bugs will be found faster if they get pulled into PRs by dependabot across the ecosystem, rather than just by a subset of preview users. If people automatically also merge those PRs without reviewing them is up to them.

This particular bug, by affecting almost every single user, means that we got a bunch of duplicate issues reported which added some noise, but rather than that a bug not being found at all.

[pkrukp]

Anyway the bug is escaped from our UTs and integrations because it manifests only on multi-document project and we’re testing 1 document at time(we check coverage for specific scenario and we generate 1 asm for 1 class and we tests only that class so only 1 cs file).

I have not seen any mentions about improving automatic tests. “multi-document project” is the normal case, nobody has projects with only one document. Did you consider adding such test?