question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Azure ML] score.py not being copied to container using SDK v2

See original GitHub issue
  • Package Name: azure-ai-ml
  • Package Version: 0.1.0b5
  • Operating System: Tested on both windows and macos
  • Python Version: 3.9

Describe the bug score.py is not being copied to a container, tried both relative and absolute path. Due to which container fails at runtime with ModuleNotFoundError: No module named 'main'

To Reproduce Steps to reproduce the behavior: Sample code is given below:

env = Environment(
    conda_file="./code/deploy/environment.yml",
    image="mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210727.v1",
)

blue_deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name=local_endpoint_name,
    model=model,
    environment=env,
    code_configuration=CodeConfiguration(
        code="./code/deploy", scoring_script="score.py"
    ),
    instance_type="Standard_F2s_v2",
    instance_count=1,
)

And directory listing from inside the docker container:

root@21d33d6d1768:/var/azureml-app# ls -alih
total 16K
714328 drwxr-xr-x 1 root root 4.0K Jul 22 12:52 .
714327 drwxr-xr-x 1 root root 4.0K Jul 22 12:19 ..
714329 drwxr-xr-x 2 root root 4.0K Jul 22 12:47 .empty
570216 -rw-r--r-- 1 root root  259 Jul 22 12:18 conda.yml
root@21d33d6d1768:/var/azureml-app#

And exception logs when container runs

(ewa_deployer_v2) waqas@192-168-1-13 EWA.CreditRisk.Model % docker run local-07222036358655:blue
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-07-22T15:15:25,827211584+00:00 - gunicorn/run
2022-07-22T15:15:25,847585459+00:00 - rsyslog/run
2022-07-22T15:15:25,849390709+00:00 - iot-server/run
2022-07-22T15:15:25,877277918+00:00 - nginx/run
EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...
2022-07-22T15:15:26,660859376+00:00 - iot-server/finish 1 0
2022-07-22T15:15:26,682624043+00:00 - Exit code 1 is normal. Not restarting iot-server.
Dynamic Python package installation is disabled.
Starting HTTP server
Starting gunicorn 20.1.0
Listening at: http://127.0.0.1:31311 (64)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 152
SPARK_HOME not set. Skipping PySpark Initialization.
Exception in worker process
Traceback (most recent call last):
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/opt/miniconda/envs/inf-conda-env/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/var/azureml-server/entry.py", line 1, in <module>
    import create_app
  File "/var/azureml-server/create_app.py", line 4, in <module>
    from routes_common import main
  File "/var/azureml-server/routes_common.py", line 32, in <module>
    from aml_blueprint import AMLBlueprint
  File "/var/azureml-server/aml_blueprint.py", line 30, in <module>
    import main
ModuleNotFoundError: No module named 'main'
Worker exiting (pid: 152)
Shutting down: Master
Reason: Worker failed to boot.

Expected behavior scoring file should be copied and container starts without any issue

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
rdzotzcommented, Oct 19, 2022

Is there any solution or associated root cause of this issue. I am experiencing the same when using the azure CLI v2 for deployment

0reactions
TajiHarrisMicrosoftcommented, Oct 26, 2022

@waqassiddiqi could I receive the environment.yaml and score.py file used. Also, what region was this done in?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting online endpoints deployment and scoring
Learn how to troubleshoot some common deployment and scoring errors with online endpoints.
Read more >
Manage Azure Machine Learning environments with the CLI ...
Learn how to manage Azure ML environments using Python SDK and Azure CLI extension for Machine Learning.
Read more >
Tutorial: ML pipelines with Python SDK v2 - Azure
In this tutorial, you'll use Azure Machine Learning (Azure ML) to create a production ready machine learning (ML) project, using AzureML Python ......
Read more >
Advanced entry script authoring - Azure - Microsoft Learn
Azure Machine Learning then creates an OpenAPI (Swagger) specification for the web service during deployment. Warning. You must not use ...
Read more >
Access data in a job - Azure Machine Learning | Microsoft Learn
Learn how to read and write data for your jobs with the Azure Machine Learning Python SDK v2 and the Azure Machine Learning...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found