Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Please tell me how to run it with parameters.

See original GitHub issue

https://github.com/elyra-ai/airflow-notebook/blob/25eacb637043cedcaab87b1e846411cf70c767ea/etc/docker-scripts/bootstrapper.py#L205

In order to receive a variable and run the notebook, I need to run the paper mill by passing it a variable, but there doesn’t seem to be such a setting right now.

Maybe there is another way I’m not aware of?

As far as I know, passing parameters should be like this:


import papermill as pm

pm.execute_notebook(
   'path/to/input.ipynb',
   'path/to/output.ipynb',
   parameters = dict(alpha=0.6, ratio=0.1)
)

in papermill README.md

Issue Analytics

State:
Created 2 years ago
Reactions:1
Comments:11 (6 by maintainers)

Top GitHub Comments

1reaction

nanaonescommented, Jun 3, 2021

Hi @nanaones, could you describe what your deployment environment currently looks like so I can setup a reproduction for the development team here? e.g. how is spark deployed? colocated in the same k8s cluster or remote? configuration in airflow via connections? etc. thanks!

Airflow is above EKS. Jupyter Hub, Livy, YARN, and Spark are installed on EMR. I installed Elyra from JupyterHub for Jupyter lab users. This is the same configuration as the notebook in EMR used by AWS. The kernel images used were created using the following guides from AWS. (However, only sparkmagic is installed in the default Python image, and the kernel is added via jupyter-kernelspec. Send a request to EMR’s LIVY endpoint via Jupyter’s sparkmagic_config.json file in AWS EMR. )

https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-local-jupyter.html

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-jupyterhub.html

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-considerations.html

0reactions

thesuperzappercommented, Feb 14, 2022

I have raised a separate issue https://github.com/elyra-ai/elyra/issues/2472 about setting dynamic environment variables in Notebook/Script that take their values from the outputs of the upstream tasks.