ParquetSharp should take into account "isAdjustedToUTC" for Timestamps

Quality of life improvement.

ParquetSharp by default returns DateTimes with DateTime.Kind equal to Unspecified. ParquetSharp can be a bit smarter and use isAdjustedToUTC field from the Parquet Format to specify the DateTime.Kind as DateTime.Utc

By default, ParquetSharp uses new DateTime() which returns an DateTime.Kind.Unspecified variant.

[Nisden]

HI @adamreeve

Just a small follow up… So I have been feeding you the wrong information before.

Azure Synapse Serverless only supports isAdjustedToUTC=true And we do actually save “locally adjusted” DateTime’s to Parquet, so getting DateTimeKind.UTC is kinda of a pain.

For now I have implemented a workaround

public class CustomLogicalReadConverterFactory : LogicalReadConverterFactory
    {
        private readonly bool forceDateTimeKindUnspecified;

        public CustomLogicalReadConverterFactory(bool forceDateTimeKindUnspecified)
        {
            this.forceDateTimeKindUnspecified = forceDateTimeKindUnspecified;
        }

        public override Delegate GetConverter<TLogical, TPhysical>(ColumnDescriptor columnDescriptor, ColumnChunkMetaData columnChunkMetaData)
        {
            if (forceDateTimeKindUnspecified && typeof(TLogical) == typeof(DateTime))
            {
                var timestampType = (TimestampLogicalType)columnDescriptor.LogicalType;
                switch (timestampType.TimeUnit)
                {
                    case TimeUnit.Millis:
                        return (LogicalRead<DateTime, long>.Converter)((s, _, d, _) => LogicalRead.ConvertDateTimeMillis(s, d, DateTimeKind.Unspecified));
                    case TimeUnit.Micros:
                        return (LogicalRead<DateTime, long>.Converter)((s, _, d, _) => LogicalRead.ConvertDateTimeMicros(s, d, DateTimeKind.Unspecified));
                }
            }

            if (forceDateTimeKindUnspecified && typeof(TLogical) == typeof(DateTime?))
            {
                var timestampType = (TimestampLogicalType)columnDescriptor.LogicalType  ;
                switch (timestampType.TimeUnit)
                {
                    case TimeUnit.Millis:
                        return (LogicalRead<DateTime?, long>.Converter)(
                            (source, rep, dest, def) => LogicalRead.ConvertDateTimeMillis(source, rep, dest, def, DateTimeKind.Unspecified));
                    case TimeUnit.Micros:
                        return (LogicalRead<DateTime?, long>.Converter)(
                            (source, rep, dest, def) => LogicalRead.ConvertDateTimeMicros(source, rep, dest, def, DateTimeKind.Unspecified));
                    case TimeUnit.Nanos:
                        return (LogicalRead<TPhysical?, TPhysical>.Converter)LogicalRead.ConvertNative;
                }
            }

            return base.GetConverter<TLogical, TPhysical>(columnDescriptor, columnChunkMetaData);
        }
    }

And ill try and open up a ticket with Microsoft for an actual solution that will allow use to use isAdjustedToUtc=false

[adamreeve]

Yes the proposed change only affects reading Parquet data into dotnet DateTime values. When writing DateTimes, we currently default to using isAdjustedToUtc: true, and there isn’t really a way to infer what this should be so we’re not suggesting changing how data is written.

So from what you’re saying, it sounds like you already need to specify isAdjustedToUtc: false when writing files for compatibility with Azure Synapse Serverless. And there probably isn’t a need to make the default reading behaviour configurable then.