isAdjustedToUtc: false

Hello G-Research,

I’ve been working on a project in order to convert data from an SQL-Server database to parquet, but I reach a wall when trying to remove the UTC from the DateTime column for the writer. I’ve read through other issues and found out that isAdjustedToUtc is set to default true, however the original data i have does not use UTC. If I try using columns[i] = new Column<DateTime?>(cdt.Schema[i].Name, LogicalType.Timestamp(isAdjustedToUtc: false, timeUnit: TimeUnit.Nanos) there is an exception since the values in the database are of type DateTime and there is a mismatch with the LogicalTypeOverride.

System.NotSupportedException: 'unsupported logical system type System.Nullable1[System.DateTime] with logical type Timestamp(isAdjustedToUTC=false, timeUnit=nanoseconds, is_from_converted_type=false, force_set_converted_type=false)

I posted the same question in StackOverflow a few days ago, but haven’t gotten a conclusive answer, apart from using the ConverterFactory, and while i try using that based on the snippets provided I can’t seem to get it working converterFactory.GetConverter<DateTime, TimeSpanNanos>(columnDescriptor, byteBuffer: buffer) ;

To briefly summarize, I just want to take the DateTime values in the original database column and write them to the parquet file again as DateTime but without UTC. And since they are of the same type it feels redundant to use the LogicalFactories to cast DateTime as DateTime with no UTC. Is there an easier way or what is the best way to proceed?

I know this is not a direct issue to the library itself, and more an issue on my part, but I would really appreciate any suggestions how to handle this instance.

[vthemelis]

Oh, sorry, I misunderstood the purpose of the logical type. I thought that I would have to provide a logical type representing an array (or list as in LogicalType.List()) of DateTimes as the last argument to the Column ctor.

Looks like just passing the logical type for the leaf type of the array is enough.

Thanks!

[adamreeve]

Hi @vthemelis, can you explain more what you’re trying to achieve? It sounds like you have an array of DateTimes and you want to write them as TimeZone naive timestamps in a Parquet file, so you should just be able to specify isAdjustedToUtc: false in the logical type.

If instead you’re talking about reading DateTime values from a Parquet file and you want them to have Kind set to DateTimeKind.Unspecified you can use the ParquetSharp.ReadDateTimeKindAsUnspecified AppContext switch that was added in #288