v2020.4.1b1: Released sources don't contain necessary test data

Issue description

The latest released sources (v2020.4.1b1 and v2020.4.1a2) don’t contain necessary test data.

The release tarballs and zip files on GitHub don’t contain the directory tests/pypi, probably because it’s git submodule.

The submodule leads to repo https://github.com/sarugaku/pipenv-test-artifacts, which is problematic:

1. It does not contain any releases, so I’ll have to download the sources corresponding to a commit.
2. The repo is not owned by pypa.

The sdist from PyPI does not contain tests at all.

Expected result

1. Some way to get the proper test data - either included in the pipenv source release, or a separate release.
2. Repos with pipenv (test) sources owned by PyPA.

Actual result

Not able to run tests from provided sources.

[frostming]

I am willing to help here. I found the tests very unusual for a python project. I think it’s quite a hurdle for developers to get started.

Yeah, Pipenv is using a highly black-boxed testing method. The mocking level is too low IMO. There are mainly two parts of testing: resolution and installation. For resolution, most resolvers(poetry, piptools, resolvelib) are using mocked package sources, it should be easy to learn from.

[techalchemy]

For resolution, most resolvers(poetry, piptools, resolvelib)

It really comes down to the space pipenv is claiming to occupy because it crosses virtualenv boundaries, so despite the fact that it relies on other libraries in most cases to do each individual piece (e.g. piptools/resolvelib or pip) we still need to ensure that the outcome we expect actually occurs (e.g. after installing some package, it is ultimately present in the place we installed it).

Most of the fundamentals in question wind up being moved into other libraries (requirementslib or vistir for example) which perform lower level testing and a bit more in the way of mocking – and resolvelib which is the direction we should move with our own resolver as well.

So in one sense we are testing the ‘right’ things, i.e. the high level user flow producing the necessary outcomes. However, we aren’t doing nearly enough unit testing and, given enough time, we could easily avoid using real libraries to produce the errors we are testing for. Using real libraries with a fake local pypi instance was just faster since users were encountering real issues. Those libraries have dependencies, however, and that means including those in the test suite, etc. So yeah, mocking things would be great.