question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Multiple results tabs opening when executing script under Apache Spark connector

See original GitHub issue

Description

When executing a script on an Apache Spark connection, a results window gets opened for each statement whether results are returned or not. Consider the following query:

CREATE TABLE MyTable(PersonName VARCHAR(50));
INSERT INTO MyTable(PersonName) SELECT 'John Doe';
SELECT * FROM MyTable;
DROP TABLE MyTable;

On other connections (tested on MS SQL Server), one result window is opened, which is expected since there is one SELECT statement. On Apache Spark, however, four tabs are opened, one for each statement.

MS SQL Server:

image

Apache Spark:

image

DBeaver Version

22.2.4.202211061524

Operating System

Windows 11 22H2

Database and driver

Spark 3.3.1 Apache Spark driver 1.2.1.spark2

Steps to reproduce

  1. Create an Apache Spark connection
  2. Create a script consisting of several SQL DML statements that do not produce results
  3. Run all the statements as a single script by pressing Alt-X or the Execute script button
  4. Multiple result windows are opened

Additional context

No response

Issue Analytics

  • State:open
  • Created 10 months ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
ShadelessFoxcommented, Nov 17, 2022

Wow, thank you! I believe this would be enough.

0reactions
chipmunkofdoom2commented, Nov 17, 2022

@ShadelessFox Are there any settings I can change or play around with on the driver? Or in DBeaver?

If you need a Spark dev environment, it’s fairly straightforward to stand up a single node Spark cluster in standalone mode. Just tested these commands in an LXC container running Ubuntu 22.04:

# Optional: if you don't have Java installed or this is new VM/container, install JDK and set Java home
sudo apt-get update; sudo apt-get install openjdk-11-jdk -y
echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> ~/.bashrc
source ~/.bashrc

# Download latest Spark release and unpack
wget https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz
tar -xf spark-3.3.1-bin-hadoop3.tgz
mv spark-3.3.1-bin-hadoop3 spark

# Start master node. Explicitly set listen address to 127.0.0.1
spark/sbin/start-master.sh --host 127.0.0.1

# Start worker node pointing at the master node we just created
spark/sbin/start-worker.sh spark://127.0.0.1:7077

#Start thriftserver pointing at our master node. This is how DBeaver will connect
spark/sbin/start-thriftserver.sh --master spark://127.0.0.1:7077

Next, add a Spark connection in DBeaver. For the host, use the IP/name of the machine hosting Spark. Leave port as 10000. I put default as the database, but I don’t think this matters. Leave username/password blank:

image

You should then be up and running.

I can also help with testing. I don’t have Java experience, but if there are any guides for compiling/running tests, I can follow them.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Configuration - Spark 3.1.2 Documentation
Spark properties control most application settings and are configured separately for each application. These properties can be set directly on a SparkConf ......
Read more >
Spark Web UI - Understanding Spark Execution
Spark UI is separated into below tabs. If you are running the Spark application locally, Spark UI can be accessed using the http://localhost: ......
Read more >
SQL Execution - DBeaver Documentation
The executes all queries in the script, but opens multiple result tabs. Each script query is executed in a separate thread (that is,...
Read more >
Lab 4 - Explore, transform, and load data into the Data ...
Then you will use Apache Spark to load data into the data warehouse and join ... A new SQL script tab will open...
Read more >
Authoring and running queries - Amazon Redshift
You can author queries with multiple SQL statements in one query tab. The queries are run serially and multiple results tabs open for...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found