UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 78552: character maps to <undefined>

Description

pandas.CSVDataset default argument hit Unicode Error

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 78552: character maps to <undefined>

Steps

Step to reproduce

catalog.yml

sample:
  type: pandas.CSVDataSet
  filepath: data2/sample.csv

python script to create and load dataset

df['test'] = ['123']
df['test'] = '\x9d'

df.to_csv('sample.csv') # to data/sample.csv
pd.read_csv('sample.csv') # success
catalog.load('sample') # UnicodeDecodeError: 'charmap'

Kedro: 0.16.4 Window 10

[limdauto]

~I think this is a discrepancy in pandas’ API, not Kedro.~

When writing data, the pandas.CSVDataSet uses panda’s to_csv method. This method uses utf-8 as the default encoding.

On the other hand, when reading data, the pandas.CSVDataSet uses panda’s read_csv method. The subtle difference here is that this method has an optional encoding, instead of an utf-8 default. If it’s not provided, the dataset will defer to the encoding of the open() method on the filesystem, which will use the system’s default encoding. On some Windows system, it’s cp1252, hence the charmap error message.

You can fix this by either changing the encoding on the reading side or the writing side. Both are equally valid.

• On the reading side, you need to pass encoding to both load_args, which will then pass it to panda’s read_csv, and open_args_load, which will then pass it to the open() method, so both use utf-8 correctly.
• Or on the writing side, you can pass cp1252 to save_args so pandas writes your file with your system’s default encoding and no need to adjust the load args.

I hope this helps.

[lorenabalan]

@noklam I believe this is addressed in 0.17.0, as per commit referenced above. Could you please confirm if that’s the case, and if so consider closing the issue?