Support for unicode utf-8 charmaps

Using mailmerge to send mass emails is better than using any gmail extensions, since i have the ability to use advanced markdown and html however while working with a large (5k+) csv file of job applicants, i came across an issue that people had entered their names in a localized language (eg. mandarin, etc) that sometimes didn’t fit the ascii charmaps.

It would be useful to open the csv file as encoding=utf8 with errors=ignore to avoid errors like this: 'charmap' codec can't decode byte 0x9d in position 1710: character maps to <undefined> instead of just returning true if strings are ascii.

[thecodelearner]

@seshrs awesome! I will try this.

[thecodelearner]

GitHub doesn’t support csv uploads so here’s a txt file of the csv: mailmerge_database_txt.txt

edit: uploaded the mailmerge_database.csv here, just in case we have any issues with csv as a txt file https://github.com/thecodelearner/tmp-files/blob/master/mailmerge_database.csv

also, I think that my python installation on windows is broken since the same file runs perfectly on linux, but gives the UnicodeDecodeError on Windows

—> Running mailmerge on Windows:

PS7 mailmerge-client> mailmerge
Traceback (most recent call last):
  File "c:\users\pc\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\pc\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python38\Scripts\mailmerge.exe\__main__.py", line 7, in <module>
  File "c:\users\pc\appdata\local\programs\python\python38\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\pc\appdata\local\programs\python\python38\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\pc\appdata\local\programs\python\python38\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\pc\appdata\local\programs\python\python38\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\pc\appdata\local\programs\python\python38\lib\site-packages\mailmerge\__main__.py", line 114, in main
    for _, row in enumerate_range(csv_database, start, stop):
  File "c:\users\pc\appdata\local\programs\python\python38\lib\site-packages\mailmerge\__main__.py", line 288, in enumerate_range
    for i, value in enumerate(iterable):
  File "c:\users\pc\appdata\local\programs\python\python38\lib\site-packages\mailmerge\__main__.py", line 273, in read_csv_database
    for row in reader:
  File "c:\users\pc\appdata\local\programs\python\python38\lib\csv.py", line 110, in __next__
    self.fieldnames
  File "c:\users\pc\appdata\local\programs\python\python38\lib\csv.py", line 97, in fieldnames
    self._fieldnames = next(self.reader)
  File "c:\users\pc\appdata\local\programs\python\python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 17: character maps to <undefined>

—> Running mailmerge on Linux with exact same database.csv:

root@Proton:/mnt/c/Users/PC/merge-test# cat mailmerge_database.csv
name,email
"大松 李",[e-mail]
"Juán",[e-mail]
"Jöse Felix",[e-mail]
"Paveł",[e-mail]
root@Proton:/mnt/c/Users/PC/merge-test# file mailmerge_database.csv
mailmerge_database.csv: CSV text
root@Proton:/mnt/c/Users/PC/merge-test# mailmerge --version
mailmerge, version 2.1.0
root@Proton:/mnt/c/Users/PC/merge-test# mailmerge
>>> message 1
TO: [e-mail]
SUBJECT: Testing mailmerge
FROM: My Self <[e-mail]>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Date: Tue, 14 Jul 2020 14:13:53 -0000

Hi, 大松 李

>>> message 1 sent
>>> Limit was 1 message.  To remove the limit, use the --no-limit option.
>>> This was a dry run.  To send messages, use the --no-dry-run option.