question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Parquet export trims SqlDateTime time

See original GitHub issue

Hello, When I try to export data to parquet from SQLServer table with a column type SqlDateTime, the column is converted to string which does not contain time data anymore, but only date.

I tried setting TypeConverterFormatSpec, but it didn’t help. What am I missing?

        ChoETLFrxBootstrap.IsSandboxEnvironment = true;

        var cmd = new SqlCommand($"sql cmd", conn);
          
        var dr = await cmd.ExecuteReaderAsync();

        using var w = new ChoParquetWriter(fileName)
            .Configure(c => c.Culture = CultureInfo.InvariantCulture)
            .Configure(c => c.TypeConverterFormatSpec = new ChoTypeConverterFormatSpec { DateTimeFormat = "o" })
            .Configure(c => c.LiteParsing = true)
            .Configure(c => c.RowGroupSize = 5000)
            .NotifyAfter(100000)
            .OnRowsWritten((o, e) => $"Rows Loaded: {e.RowsWritten} <-- {DateTime.Now}".Print());

        w.Write(dr);

Issue Analytics

  • State:closed
  • Created 7 months ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
Cinchoocommented, Feb 13, 2023

applied fix, released v1.0.1.26. try it and let me know.

1reaction
Cinchoocommented, Feb 8, 2023

Got it, type spec is respected only in POCO model reader.

Let me fix it. If you want to load them successfully, define POCO class and use the writer.

Will keep you posted with the fix.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to write TIMESTAMP logical type (INT96) to parquet ...
INT96 timestamps use the INT96 physical type without any logical type, so don't annotate them with anything. If you are interested in the ......
Read more >
Unloading semistructured data in the Parquet format
You can unload tables with SUPER data columns to Amazon S3 in the Parquet format. Amazon Redshift represents SUPER columns in Parquet as...
Read more >
Loading Parquet data from Cloud Storage | BigQuery
This page provides an overview of loading Parquet data from Cloud Storage into BigQuery. Parquet is an open source column-oriented data format that...
Read more >
EXPORT command
The EXPORT command exports data from a database to one of several external file formats. ... Export packages are bound by using the...
Read more >
Viewing Parquet Export Events More Easily
The EXPORT TO PARQUET command exports a table, columns from a table, or query results to files in the Parquet format. When you...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found