Log debug and informational messages to stderr only

What’s the problem this feature will solve?

Commands that generate output for the user (pip list, pip freeze) must be concerned with debug output.

Describe the solution you’d like

As mentioned in #6099:

1. all log messages should go to stderr instead of stdout
2. command API related output should use a dedicated interface that outputs to stdout

Any issue that seeks to make the CLI more usable would benefit from this change. This also avoids issues like #6024.

Alternative Solutions

Originally, ad-hoc redirection of logger output to stderr was used to preserve command output. This could be used again, but it would make logging output from commands inconsistent in pip.

Additional Context

• #6099

[cjerdonek]

I was planning to do this between the 19.2 and 19.3 release but would be okay with someone else doing it. (I had already done it for subprocess logging in PR #6339.)

I was thinking this should be implemented in three PR’s as follows:

1. Switch all command API output to use a function called something like write_output() whose initial implementation just delegates to logging.INFO (so this would be a refactoring PR with no behavior change). This PR will let us review where all the API output is happening.

2. Change the implementation of write_output() from using a logging call to writing directly to stdout.

3. Change the logging configuration to use stderr streams for everything instead of stdout for some. This step will involve simplifying our logging config and in particular merging the subprocess logger introduced in PR #6339 with the main logger, as they would now both be using the same stderr stream.

[cjerdonek]

print is probably fine, too. We can always change it if we need to. Also, you don’t necessarily want to copy the subprocess logger. It’s just an example of a different logger being used.