LightGBM error special JSON characters in feature name

Hi - great library everyone!

I’m doing tabular prediction with news article data. The LightGBM model doesn’t run with the following error:

LightGBMError: Do not support special JSON characters in feature name.
Do not support special JSON characters in feature name.

The upstream issue in LightGBM is here:https://github.com/microsoft/LightGBM/issues/2455

Basically they check for special json characters .e.g [],{}": in features names and throw an error if found. Could autogluon check for these characters and remove any offending features?

Liam

[oguzhangur96]

Hi,

I have encountred the same problem. Fixed it before the training begins with a regular expression and lambda function (pandas). Object named “data” must be a dataframe.

import re
data = data.rename(columns = lambda x:re.sub('[^A-Za-z0-9_]+', '', x))

Could provide data/column headers if needed.

[Innixma]

Thanks for the info @oguzhangur96! Right now the main difficulty here is to ensure that column names are always converted when entering the model (whether for feature importances, fit, inference) and also inversely converted back to the original upon leaving. This seems to be something that LightGBM itself should handle, and I don’t have a good understanding of why they haven’t done so.

Furthermore, it isn’t sufficient to only remove the special characters, because we have to ensure no two columns have the same name. One simple way is to just rename columns 0-n as ‘0’, ‘1’, …‘n-1’, ‘n’, but then this operation would also have to be done on all of the 99% of problems where this is not required, particularly for online-inference of single rows. I have to first benchmark inference times in these situations to ensure our online-inference speed isn’t significantly slowed due to this fix.