Update E2E to fail on PHP errors/warnings/notices

Feature Description

Our E2E tests are intentionally set up to fail if a console log is made unexpectedly. The same has not been true however for PHP log entries which are just as valuable to catch.

Luckily, setting up debug logging in WP has been supported for a very long time, we only need to configure our E2E environment to do this and then to listen for changes during tests. This sounds somewhat complicated, but it isn’t.

Docker already exposes container logs via its API (along with everything else that can be done via the command line) which can be consumed as a stream in our Node E2E Jest process. Then we only need to add a dedicated container for tailing the WP debug.log file which will be essentially piped to the log for that container, so we can subscribe specifically to this one log. Then we just need to handle the stream and data from it in Jest as part of normal setup/teardown.

See my POC branch here https://github.com/google/site-kit-wp/compare/poc/e2e-docker-logs

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

• PHP errors of any kind (fatal/warning/notice/deprecation) raised during an E2E test should cause that test to fail similar to an unexpected console log on the client
  • The only difference here is that PHP errors should never be expected, so we don’t need to add support for allowing them as we have in JS
• The specific errors that were raised should be exposed in the test failure
• Errors raised in one test should not cause other tests to fail

Implementation Brief

• Finish and merge https://github.com/google/site-kit-wp/pull/3936
  • Mostly done in the POC already
  • Fix existing PHP notice being raised in tests
  • Final cleanup/polish

Test Coverage

• No additional tests to be added, the changes here will expand the coverage to include observation of the debug

Visual Regression Changes

• N/A

QA Brief

• Create a new Pull Request in the “Ready for review” state so the E2E tests run in CI.
• Make a change to one of the PHP files used in E2E that causes a PHP error to be raised (e.g. trigger_error( 'Hello debug.log!' );)
• Ensure that E2E tests that cause errors to be raised fail accordingly and others that do not raise errors in the same run do not fail

Changelog entry

• N/A

[FlicHollis]

Hi @aaemnnosttv just adding your comment here that this may be blocked until we raise the minimum version of WP that we require. I will remove this from the sprint until we have a further update.

[FlicHollis]

Hi @aaemnnosttv Moving this back to EB as this has not been added to Sprint 68