[TODO Roundup] Logging overhaul

Summary

Mephisto doesn’t yet have a good system for logging of information. As such we’d like to add loggers via the python Logger class. At a high level, modules should each have their own loggers. Initially we should be able to configure the log level across the whole application (from DEBUG to CRITICAL), and should be able to write both to terminal and to file. NOTE - the logger by default should be configured to use NullHandler, and mephisto should have a setting to toggle this to StreamHandler (for writing to terminal) or FileHandler (for writing to file).

Some helpful information for logging best practices can be found across the web (1, 2, 3…)

Requirements:

• Create a core.logger class for Mephisto to store any initializers and helper methods required for creating loggers.
• Add a logger object to every class in the core, data_model, server, and providers modules.
• Replace existing print statements with logs, and add additional marked logs.
• Add logging to unintended except blocks

Missing logs

• Do a check to see if the database schema is not as expected, and log this to the user.
• Update the wrapper around the call to heroku functions to only surface the “You have too many apps” message if there are actually too many applications, otherwise surface the contained error. This can be tested by creating a heroku account, launching 5 apps without shutting them down, and then launching another.
• Do propper logging around failed MTurk Approve and Reject calls, described here

Wishlist

• Peruse the flow of the Supervisor, Operator, and TaskLauncher classes to see if there are important control flow operations that might be improved with logging.
• Add logging to the data model for similar important control flow paths, things that might require inspection. (Essentially, while debugging other parts of Mephisto, when you find the important variable to introspect, add that to DEBUG logging. Important control flow goes into INFO, and then errors above that).
• Update the logging in print_run_details to actually have useful details about the progress of tasks currently being run.
• Write debug level logging to file, and provide a utility for cleaning up old Mephisto logs.

[pringshia]

I arrived here via this TODO.

The error message here can be misleading: https://github.com/facebookresearch/Mephisto/blob/2067a756d71aeb1f2f0368402c42fdf84184b3d7/mephisto/abstractions/architects/heroku_architect.py#L371-L377

@jxmsML encountered an error, which was suppressed and mislogged. The error was:

Creating ⬢ jingxu23-Ebb27b-2149-763c5a327… ! ▸ Name must start with a letter, end with a letter or digit and can only contain lowercase letters, digits, and dashes.

whereas they were presented with:

Exception: You have hit your limit on concurrent apps with heroku, which are required to run multiple concurrent tasks. Please wait for some of your existing tasks to complete. If you have no tasks running, login to heroku and delete some of the running apps or verify your account to allow more concurrent apps

[JackUrb]

Covered in #690