Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

R script processing fails execution on Apache Airflow

See original GitHub issue

Describe the issue See below. Looks like R scripts in general are not yet supported on AA.

To Reproduce Steps to reproduce the behavior:

Create an R script.
Create an Apache Airflow RTC
Submit the R script using above RTC
Check the task log file

[2021-04-08 20:58:24,774] {pod_launcher.py:156} INFO - b"[I 20:58:24.773] 'an_r_script-0408135736':'an_r_script' - Packages installed (13.682 secs)\n"
[2021-04-08 20:58:24,779] {pod_launcher.py:156} INFO - b'Traceback (most recent call last):\n'
[2021-04-08 20:58:24,780] {pod_launcher.py:156} INFO - b'  File "bootstrapper.py", line 395, in <module>\n'
[2021-04-08 20:58:24,780] {pod_launcher.py:156} INFO - b'    main()\n'
[2021-04-08 20:58:24,780] {pod_launcher.py:156} INFO - b'  File "bootstrapper.py", line 384, in main\n'
[2021-04-08 20:58:24,780] {pod_launcher.py:156} INFO - b'    file_op = FileOpBase.get_instance(**input_params)\n'
[2021-04-08 20:58:24,781] {pod_launcher.py:156} INFO - b'  File "bootstrapper.py", line 57, in get_instance\n'
[2021-04-08 20:58:24,781] {pod_launcher.py:156} INFO - b"    raise ValueError('Unsupported file type: {}'.format(filepath))\n"
[2021-04-08 20:58:24,781] {pod_launcher.py:156} INFO - b'ValueError: Unsupported file type: an_r_script.r\n'
[2021-04-08 20:58:25,822] {pod_launcher.py:173} INFO - Event: an-r-script-1ed4d6271f374226a7abe2fa51b94fb6 had an event of type Failed
[2021-04-08 20:58:25,822] {pod_launcher.py:284} INFO - Event with job id an-r-script-1ed4d6271f374226a7abe2fa51b94fb6 Failed
[2021-04-08 20:58:25,839] {pod_launcher.py:173} INFO - Event: an-r-script-1ed4d6271f374226a7abe2fa51b94fb6 had an event of type Failed
[2021-04-08 20:58:25,839] {pod_launcher.py:284} INFO - Event with job id an-r-script-1ed4d6271f374226a7abe2fa51b94fb6 Failed
[2021-04-08 20:58:25,863] {taskinstance.py:1150} ERROR - Pod Launching failed: Pod returned a failure: failed
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 287, in execute
    'Pod returned a failure: {state}'.format(state=final_state))
airflow.exceptions.AirflowException: Pod returned a failure: failed

Expected behavior The R script is executed.

Deployment information Describe what you’ve deployed and how:

Elyra version: 2.2+
Installation source: from source
Deployment type: local execution

Issue Analytics

State:
Created 2 years ago
Comments:5 (5 by maintainers)

Top GitHub Comments

1reaction

ptitzlercommented, Apr 8, 2021

I updated the announcement thread. As far as I know this restriction has not been publicly communicated before.

0reactions

ptitzlercommented, Apr 9, 2021

@akchinSTC can you please indicate which version of the elyra/airflow container image includes the fix?

Top Results From Across the Web

Deploying R models in production with Apache Airflow and ...

Our use case involves a batch learning deployment through an Airflow DAG. Before running R scripts containing the XGBoost model, our data ...

Serialization error after successful DAG run #20875 - GitHub

Below is the error we are getting - this usually occurs after task execution and not during DAG parsing by the scheduler. Apache...

Using Spark with Airflow in R - YouTube

This video presents using Apache Airflow (a platform to programmatically author, schedule and monitor workflows) with Apache Spark and ...

Apache Airflow | How to check response coming from a ...

Try the following code: bash_command = """ set -e; Rscript /code/demo.R | grep 'status:200' ... Airflow is unable to interpret exit command.

Command Line Interface and Environment Variables Reference

Possible choices: celery, cheat-sheet, config, connections, dag-processor, dags, db, info, ... airflow dags backfill [-h] [-c CONF] [--continue-on-failures] ...