Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Support UPDATE output mode for Spark Structured Streaming
See original GitHub issueWe now support Spark Structured Streaming as of 6.x, but only in APPEND mode. It would be beneficial to support UPDATE mode, with the requirement that the user MUST provide a field to be used for a document ID.
Issue Analytics
- State:
- Created 6 years ago
- Reactions:5
- Comments:16 (7 by maintainers)
Top Results From Across the Web
OutputMode · The Internals of Spark Structured Streaming
Output mode ( OutputMode ) of a streaming query describes what data is written to a streaming sink. There are three available output...
Read more >Spark Streaming - Different Output modes explained
OutputMode in which all the rows in the streaming DataFrame/Dataset will be written to the sink every time there are some updates. Use...
Read more >Structured Streaming Programming Guide - Apache Spark
Note that there are some restrictions when you use session window in streaming query, like below: “Update mode” as output mode is not...
Read more >Spark streaming output modes - Medium
Spark streaming output modes · Append mode: It involves writing only the new incoming data to the sink. · Update mode: It involves...
Read more >Spark Structured Streaming output mode.
Complete mode: it writes all the rows. It is supported just by groupBy or groupByKey aggregations. Update mode: writes in the sink “only”...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
While the project’s official stance is that we maintain BWC with the previous major’s most recent minor release, we give it our best effort to maintain backwards compatibility with as many versions of Elasticsearch as we can.
If anyone on this ticket is still interested (I know it’s been a long wait), I’ve got a draft PR up for this. I’d appreciate any feedback before we finalize it. See #1839