Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

SparkSQL saves Timestamp/DateType's as Longs

See original GitHub issue

The DataFrameValueWriter contains the following:

case TimestampType => generator.writeNumber(value.asInstanceOf[Timestamp].getTime())
case DateType      => generator.writeNumber(value.asInstanceOf[Date].getTime())

While the TimestampType is documented as being persisted as a long, the DateType is documented as being serialized and persisted as a Date string.

Issue Analytics

State:
Created 6 years ago
Reactions:13
Comments:9

Top GitHub Comments

4reactions

Fangroucommented, Oct 19, 2017

Hello @jbaiera , I also notice this problem when I read the doc and this error appear 2 times, in Table9 and Table10 (https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-sql-type-conversion)

And my question is why we convert Timestamp/Date to long? Should we keep the same behaviors between JdkValueWriter and DataFrameValueWriter ?

Or maybe instead of converting Timestamp/Date to long or date (string format), it would be better to offer an option in configuration to let users make the choice?

Regards, Fangrou

3reactions

jambortacommented, Jul 14, 2018

I ran into the same problem today. Regardless if it a date or timestamp, the spark connector writes to ES as a number. Is there any way around this to write fields as date?

Top Results From Across the Web

How to Effectively Use Dates and Timestamps in Spark 3.0

Spark SQL defines the timestamp type as TIMESTAMP WITH SESSION TIME ZONE , which is a combination of the fields ( YEAR ,...

Does spark sql timestamp datatype actually store timezone?

sql.Timestamp * and are stored internally as longs, which are capable of storing timestamps with microsecond * precision. ref- spark-git.

Data Types - Spark 3.3.1 Documentation

TimestampType : Represents values comprising values of fields year, month, day, hour, minute, and second, with the session local time-zone. The timestamp value ......

Spark SQL Date and Timestamp Functions

All these accept input as, Date type, Timestamp type or String. ... Column, Returns the current Unix timestamp (in seconds) as a long....

java.sql.Timestamp Scala Example - ProgramCreek.com

package oharastream.ohara.connector.jdbc.datatype import java.sql. ... duration) def beWithinSeconds(seconds: Long)(expected: Timestamp) = new ...