Unicode decoding issue on Windows

When trying to decode Unicode characters on Windows, gitlint can crash. This can easily be shown by trying to lint the commit 3ee281eec4ef4325ce90d27f6f368a6b95818cfe of the gitlint commit history.

(.venv) C:\Users\Administrator\gitlint>gitlint --commits 3ee281eec4ef4325ce90d27f6f368a6b95818cfe
Traceback (most recent call last):
  File "C:\Users\Administrator\gitlint\gitlint\.venv\Scripts\gitlint-script.py", line 11, in <module>
    load_entry_point('gitlint', 'console_scripts', 'gitlint')()
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\administrator\gitlint\gitlint\cli.py", line 180, in cli
    ctx.invoke(lint)
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\administrator\gitlint\gitlint\cli.py", line 211, in lint
    gitcontext = GitContext.from_local_repository(lint_config.target, ctx.obj[2])
  File "c:\users\administrator\gitlint\gitlint\git.py", line 199, in from_local_repository
    raw_commit = _git("log", sha, "-1", long_format, _cwd=repository_path).split("\n")
  File "c:\users\administrator\gitlint\gitlint\git.py", line 28, in _git
    result = sh.git(*command_parts, **git_kwargs)  # pylint: disable=unexpected-keyword-arg
  File "c:\users\administrator\gitlint\gitlint\shell.py", line 45, in git
    return _exec(*args, **kwargs)
  File "c:\users\administrator\gitlint\gitlint\shell.py", line 65, in _exec
    stdout = ustr(result[0])
  File "c:\users\administrator\gitlint\gitlint\utils.py", line 27, in ustr
    return unicode(obj, DEFAULT_ENCODING)  # pragma: no cover # noqa
  File "C:\Users\Administrator\gitlint\gitlint\.venv\lib\encodings\cp1252.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1: character maps to <undefined>

[metya]

Is it here I should write about a bug?

when I install gitlint hook by pre-commit in linux it works normally, but in windows it failed with error

gitlint..................................................................Failed
- hook id: gitlint
- exit code: 1

Traceback (most recent call last):
  File "C:\Users\metya\scoop\apps\miniconda3\4.7.12.1\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\metya\scoop\apps\miniconda3\4.7.12.1\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\Scripts\gitlint.EXE\__main__.py", line 7, in <module>
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\click\core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\gitlint\cli.py", line 204, in cli
    log_system_info()
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\gitlint\cli.py", line 51, in log_system_info
    LOG.debug("Git version: %s", git_version())
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\gitlint\git.py", line 57, in git_version
    return _git("--version").replace(u"\n", u"")
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\gitlint\git.py", line 33, in _git
    result = sh.git(*command_parts, **git_kwargs)  # pylint: disable=unexpected-keyword-arg
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\gitlint\shell.py", line 45, in git
    return _exec(*args, **kwargs)
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\gitlint\shell.py", line 65, in _exec
    stdout = ustr(result[0])
  File "c:\users\metya\.cache\pre-commit\repog8u71i9k\py_env-default\lib\site-packages\gitlint\utils.py", line 88, in ustr
    return obj.decode(DEFAULT_ENCODING)
LookupError: unknown encoding: C

Even if the commit message is perfectly valid.

[jorisroovers]

@metya, can you try setting LC_ALL to UTF-8 and let me know if that works for you?

# Regular windows CMD
Set LC_ALL=UTF-8

# git-bash/Cygwin
export LC_ALL=UTF-8

# Now try again

The reason this happens is because git sets LC_CTYPE=C on Windows when invoking /bin/sh, and Python can’t find the C encoding. I’ve added a workaround in #158, adding some logic for gitlint to more smartly fall back to UTF-8. This will go out as part of the 0.14.0 release within the next month.

This doesn’t solve all unicode issues on Windows (I’ve spend more time on it but no silver bullets…yet), but hopefully it should keep gitlint from crashing.