question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add sample high/low watermark data to quick start

See original GitHub issue

We use

SELECT From_unixtime(A0.create_time) as create_time,
               C0.NAME as schema_name,
               B0.tbl_name as table_name,
               {func}(A0.part_name) as part_name,
               {watermark} as part_type
        FROM   PARTITIONS A0
               LEFT OUTER JOIN TBLS B0
                            ON A0.tbl_id = B0.tbl_id
               LEFT OUTER JOIN DBS C0
                            ON B0.db_id = C0.db_id
        WHERE  C0.NAME IN {schemas}
               AND B0.tbl_type IN ( 'EXTERNAL_TABLE', 'MANAGED_TABLE' )
               AND A0.PART_NAME NOT LIKE '%%__HIVE_DEFAULT_PARTITION__%%'
        GROUP  BY C0.NAME, B0.tbl_name
        ORDER by create_time desc

as the high metastore query to get the high watermark and low watermark for a hive table. And we have already included the datamodel in https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/models/hive_watermark.py .

We should be able to provide a sample data with csv format to quick start. UI has already had support with watermark data.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:6 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
jornhcommented, Oct 19, 2019

@Mikhail-Ivanov and @feng-tao anything else needed to close this with https://github.com/lyft/amundsendatabuilder/pull/145 merged?

1reaction
feng-taocommented, Sep 27, 2019

@Mikhail-Ivanov all yours 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

Structured Streaming Programming Guide - Apache Spark
First, let's start with a simple example of a Structured Streaming query - a streaming word count. Quick Example. Let's say you want...
Read more >
Add watermarks and background objects in Pages on Mac
In Pages on your Mac, add section layout objects, such as watermarks and logos, that appear in the same place on the background...
Read more >
Incremental or High Water Mark data Loading
In this article, we'll examine one way that we can easily perform incremental loads using data staging connectors inside Matillion ETL. The ...
Read more >
Charting in the Active Trader Pro Platforms
Place your cursor on the start of the time range you would like to analyze, and simply click and drag your cursor to...
Read more >
JasperReports - Quick Guide - Tutorialspoint
JasperCompileManager − Used to compile a JRXML report template. net.sf.jasperreports.engine.JasperFillManager − Used to fill a report with data from the data ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found