Converting jsonLines file into parquet

I am writing log files in jsonlines format into s3 bucket. I was able to load it into panda dataframe: df=wr.s3.read_json(path, lines=True) then if I save it back wr.s3.to_parquet( df=df, path=path2, dataset=True, mode="append" ) I am receiving error ArrowTypeError: Expected a bytes object, got a 'dict' object

If I modify my code and do for df in wr.s3.read_json(path, lines=True, chunksize=1): wr.s3.to_parquet( df=df, path=path2, dataset=True, mode="append" )

The code worked, but then I am getting one file for each log entry. This is very slow and not efficient. How can I read jsonlines file and write it to parquet?

[edvorkin]

@igorborgest after further inspection I discover that you are right, some fields with the same name have different data types. That how I got the exception. Thanks for your help.

[igorborgest]

is there I way to enforce some schema from the catalog and reject non-compliant records?

Yes there is, using the dtype argument:

dtype (Dict[str, str], optional) – Dictionary of columns names and Athena/Glue types to be casted. Useful when you have columns with undetermined or mixed data types. (e.g. {‘col name’: ‘bigint’, ‘col2 name’: ‘int’})