java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter

Hi1

I am using spark-redshift to save dataframe into redshift but facing following exception.

java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
        at org.apache.avro.mapreduce.AvroKeyRecordWriter.<init>(AvroKeyRecordWriter.java:55)
        at org.apache.avro.mapreduce.AvroKeyOutputFormat$RecordWriterFactory.create(AvroKeyOutputFormat.java:79)
        at org.apache.avro.mapreduce.AvroKeyOutputFormat.getRecordWriter(AvroKeyOutputFormat.java:105)
        at com.databricks.spark.avro.AvroOutputWriter.<init>(AvroOutputWriter.scala:82)
        at com.databricks.spark.avro.AvroOutputWriterFactory.newInstance(AvroOutputWriterFactory.scala:31)
        at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:234)
        at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
        at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:88)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

[hafizmujadid]

i am using spark-avro latest version but i am facing same issue. Spark-redshift is really pissing me off 😦

[hafizmujadid]

There are multiple options to do so.

1. you can use same approach as Spark-RedShift and save your data from in S3 bucket and then run copy command using redshift jdbc driver.

2. Use SaveAsHadoopDataSet as in following code

   val dbUrl = "jdbc:postgresql://" + endpoint + ":" + port + "/" +
   

   database + “?user=” + username + “&password=” + password val records = df.map(row => (TableRow(row.toString, columnTypes), null)) val jobConf = new JobConf() DBConfiguration.configureDB(jobConf, “org.postgresql.Driver”, dbUrl, username, password) DBOutputFormat.setOutput(jobConf, table, columns: _*) records.saveAsHadoopDataset(jobConf)

On Mon, May 23, 2016 at 12:46 AM, WillSidebottom [e-mail] wrote:

OK, are you still writing data to Redshift from spark? If so, how are you doing it? Thanks again!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/databricks/spark-redshift/issues/129#issuecomment-220851987

Regards: HAFIZ MUJADID