Parquet export trims SqlDateTime time
See original GitHub issueHello, When I try to export data to parquet from SQLServer table with a column type SqlDateTime, the column is converted to string which does not contain time data anymore, but only date.
I tried setting TypeConverterFormatSpec, but it didn’t help. What am I missing?
ChoETLFrxBootstrap.IsSandboxEnvironment = true;
var cmd = new SqlCommand($"sql cmd", conn);
var dr = await cmd.ExecuteReaderAsync();
using var w = new ChoParquetWriter(fileName)
.Configure(c => c.Culture = CultureInfo.InvariantCulture)
.Configure(c => c.TypeConverterFormatSpec = new ChoTypeConverterFormatSpec { DateTimeFormat = "o" })
.Configure(c => c.LiteParsing = true)
.Configure(c => c.RowGroupSize = 5000)
.NotifyAfter(100000)
.OnRowsWritten((o, e) => $"Rows Loaded: {e.RowsWritten} <-- {DateTime.Now}".Print());
w.Write(dr);
Issue Analytics
- State:
- Created 7 months ago
- Comments:7 (4 by maintainers)
Top Results From Across the Web
How to write TIMESTAMP logical type (INT96) to parquet ...
INT96 timestamps use the INT96 physical type without any logical type, so don't annotate them with anything. If you are interested in the ......
Read more >Unloading semistructured data in the Parquet format
You can unload tables with SUPER data columns to Amazon S3 in the Parquet format. Amazon Redshift represents SUPER columns in Parquet as...
Read more >Loading Parquet data from Cloud Storage | BigQuery
This page provides an overview of loading Parquet data from Cloud Storage into BigQuery. Parquet is an open source column-oriented data format that...
Read more >EXPORT command
The EXPORT command exports data from a database to one of several external file formats. ... Export packages are bound by using the...
Read more >Viewing Parquet Export Events More Easily
The EXPORT TO PARQUET command exports a table, columns from a table, or query results to files in the Parquet format. When you...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
applied fix, released v1.0.1.26. try it and let me know.
Got it, type spec is respected only in POCO model reader.
Let me fix it. If you want to load them successfully, define POCO class and use the writer.
Will keep you posted with the fix.