Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error with latest jar file when trying to show dataframe

See original GitHub issue

@davidrabinowitz: Any suggestions on what I may be doing wrong here? I am running

openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)

but any time I try to run a simple select query statement i am getting this error? when I call the

df.show() or df.count() on my dataframe in pyspark

: org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
        at org.apache.spark.util.io.ChunkedByteBufferOutputStream.toChunkedByteBuffer(ChunkedByteBufferOutputStream.scala:115)
        at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:323)
        at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:138)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:91)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:35)
        at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:77)
        at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1509)
        at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1383)

Issue Analytics

State:
Created 3 years ago
Comments:14 (5 by maintainers)

Top GitHub Comments

1reaction

JordanYankovichcommented, Feb 3, 2021

Hey David,

I was able to get around this issue by changing the Docker base image I was using.

I downloaded spark v3.0.1 w/ Hadoop 3.2 and later from spark.apache.org to build a docker image and that fixed my issue. Downloading directly from https://github.com/apache/spark and building an image from there seemed to cause issues.

0reactions

charlesa101commented, Feb 5, 2021

Thanks, @davidrabinowitz, and @JordanYankovich! that fixed my issue as well

The base image was built with what @JordanYankovich recommended, we needed to customize the base image