question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Adding `if_exists="override" Mode

See original GitHub issue

Is your feature request related to a problem? Please describe. is there mode to do truncate load for existing table if_exists="override", it could be useful to avoid dropping table

Describe the solution you’d like being able to override data in existing table.

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:5

github_iconTop GitHub Comments

1reaction
ismailsimsekcommented, Sep 30, 2021

correct i like to replace the data, but keep the table intact. dropping table always has risk of losing table metadata, and could cause failing queries for the time between drop and create. override is safer operation.

0reactions
jkukulcommented, Jun 27, 2022

could cause failing queries for the time between drop and create

To avoid these problems, a single “truncate” load job would work, but does risk changing the schema accidentally. I think some combination of writing to a staging table, then a SQL merge statement to delete existing rows and replace with rows from the staging table would be required to do this atomically without changing the table metadata.

Could such an implementation also be considered for the replace mode? The time between “drop” and “create” can be significant for larger tables and results in the failure of downstream queries.

Could the replace mode be re-implemented with a more atomic solution, i.e.:

  1. Write to a staging location
  2. Drop the existing table
  3. Rename the staging location to the target location

What would be the downsides?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Replace and overwrite instead of appending - python
Open the file in 'w' mode, you will be able to replace its current text save the file with new contents. Share.
Read more >
Hive - INSERT INTO vs INSERT OVERWRITE Explained
INSERT OVERWRITE is used to replace any existing data in the table or partition and insert with the new rows.
Read more >
INSERT OVERWRITE - Spark 3.0.0-preview Documentation
The INSERT OVERWRITE statement overwrites the existing data in the table using the new values. The inserted rows can be specified by value...
Read more >
E45 readonly option is set (add ! to override) - Ask Ubuntu
I'm trying to configure the extension mcrypt in my Ubuntu Server VirtualBox for work in my phpMyAdmin page. I ran vi /etc/php5/mods-available/mcrypt.ini and ......
Read more >
"Write Database overwrite mode" — RapidMiner Community
If "'overwrite' mode is more or less the equivalent to DROP IF EXISTS followed by CREATE TABLE and INSERT" then after creating new...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found