[testing] trainer tests fail - 2 issues

trainer tests seem to have several issues:

USE_CUDA=1 pytest tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer

        if self.api_key is None:
>           raise ValueError(
                "Comet.ml requires an API key. Please provide as the "
                "first argument to Experiment(api_key) or as an environment"
                " variable named COMET_API_KEY "
            )

the test should skip and not fail if the user doesn’t have Comet.ml

1. wandb seems to still be problematic when there is an error in the test.

Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 180, in console_main
    code = main()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 157, in main
    ret = config.hook.pytest_cmdline_main(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 289, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 284, in wrap_session
    config._ensure_unconfigure()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 920, in _ensure_unconfigure
    fin()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 632, in stop_global_capturing
    self._global_capturing.pop_outerr_to_orig()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 522, in pop_outerr_to_orig
    out, err = self.readouterr()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 563, in readouterr
    out = self.out.snap()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 481, in snap
    self.tmpfile.seek(0)
OSError: [Errno 29] Illegal seek

I have to add WANDB_DISABLED=true to overcome those. There was an issue about it earlier, but I am not able to find it.

Full report:

FAILED tests/test_trainer.py::TrainerIntegrationTest::test_can_resume_training - ValueError: Comet.ml requires an API key. Please provide as the first argument...
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_can_resume_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_evaluate - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_evaluate - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_flos_extraction - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_flos_extraction - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_load_best_model_at_end - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_load_best_model_at_end - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_model_init - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_model_init - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_num_train_epochs_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_num_train_epochs_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_predict - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_predict - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_save_checkpoints - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_save_checkpoints - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_train_and_eval_dataloaders - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_train_and_eval_dataloaders - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_lm - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_lm - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_iterable_dataset - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_iterable_dataset - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_with_datasets - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_with_datasets - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_training_arguments_are_left_untouched - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_training_arguments_are_left_untouched - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_add_remove_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_add_remove_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_event_flow - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_event_flow - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_init_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_init_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_utils.py::TrainerUtilsTest::test_distributed_tensor_gatherer - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_utils.py::TrainerUtilsTest::test_distributed_tensor_gatherer - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_find_code_in_transformers - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_find_code_in_transformers - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_is_copy_consistent - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_is_copy_consistent - OSError: [Errno 29] Illegal seek

Thank you!

Environment info

- `transformers` version: master
- Platform: Linux-4.15.0-118-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201014 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

@sgugger

[sgugger]

Yes I think raising a warning should be better than a hard error since it does not mean the training has to stop, and it blocks the tests. If you can suggest a PR with those changes, I’d happily review it.

[stas00]

and wandb needed the latest version - which fixed the OSError: [Errno 29] Illegal seek issue