codecarbon plugin issues

https://github.com/huggingface/transformers/pull/12304 added the codecarbon plugin and there are multiple issues with it:

1. it needs to be documented in

https://github.com/huggingface/transformers/blob/2772d3e79d66925cf4adeaffd8be610f0ab177b6/src/transformers/training_args.py#L316-L319

along all the other plugins

1. It doesn’t respect user’s logging level. It needs to read the set for the current rank log-level and pass it explicitly to the object instance here:

https://github.com/huggingface/transformers/blob/2772d3e79d66925cf4adeaffd8be610f0ab177b6/src/transformers/integrations.py#L782

via the log_level argument (but which expects a string like “warning” and not the real logging.WARNING which is normally used, so one needs to remap from real logging level to the string CC expects.

1. same logs are logged more than once in different formats:

[codecarbon INFO @ 19:33:14] Tracking Nvidia GPU via pynvml
[codecarbon INFO @ 19:33:14] Tracking Intel CPU via RAPL interface
08/23/2021 19:33:14 - INFO - codecarbon - Tracking Nvidia GPU via pynvml
08/23/2021 19:33:14 - INFO - codecarbon - Tracking Intel CPU via RAPL interface

1. it breaks the training as it can’t find some server.

Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='get.geojs.io', port=443): Max retries exceeded with url: /v1/ip/geo.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fcd95312700>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/core/util.py", line 10, in suppress
    yield
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/emissions_tracker.py", line 348, in stop
    emissions_data = self._prepare_emissions_data()
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/emissions_tracker.py", line 367, in _prepare_emissions_data
    geo: GeoMetadata = self._get_geo_metadata()
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/emissions_tracker.py", line 612, in _get_geo_metadata
    return GeoMetadata.from_geo_js(self._data_source.geo_js_url)
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/external/geography.py", line 83, in from_geo_js
    response: Dict = requests.get(url, timeout=0.5).json()
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='get.geojs.io', port=443): Max retries exceeded with url: /v1/ip/geo.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fcd95312700>: Failed to establish a new connection: [Errno 111] Connection refused'))
08/23/2021 19:33:20 - WARNING - codecarbon - stopping.

This part of CC never worked for me. It always fails here for me and not just in this integration. Only Offline version of the tracker works w/o this failure. Could we use the offline tracker by default instead?

---

To reproduce - I run:

python examples/pytorch/translation/run_translation.py --model_name_or_path google/mt5-small --do_train --source_lang en --target_lang ro --dataset_name wmt16 --dataset_config_name ro-en --output_dir output_dir --per_device_train_batch_size=4 --logging_step 2 --save_steps 0 --fp16 --max_train_samples 10 --save_total_limit 0 --overwrite_output_dir --save_strategy no

Thank you.

@JetRunner, @sgugger

[JetRunner]

pinging @JetRunner

will add when I’m back from the leave

[JetRunner]

Got it. My guess is the geojs service is not available in some countries. Let me see how we can add an offline option.