How to properly install python modules with venv using DevOps Pipeline build? Module Not Found Error

Note: There are already several issues mentioning this problem, but several of these were closed after finding some quickfix which were not working for me. See i.e. this issue.

Issue Statement: The general problem is that it seems not to be possible for me to import packages installed during DevOps pipline in an app. In my case i want to use azure.servicebus and have to use the SDK as the bindings don’t fullfill my requirements. However, after building with devops and uploading the artifact via SCM_RUN_FROM_PACKAGE the app starts running, but is not able to import the module from neither worker_env nor from .python_packages...

When deploying from local via function app core tools the problem does not occur. However, for deploying to production I have to use DevOps pipline!

Investigative information

Please provide the following:

• Timestamp: 2020-07-15
• Function App name: functestkjskdj
• Core Tools version: 3.0.2534

Repro steps

Provide the steps required to reproduce the problem:

I use the following YAML in my Azure Pipelines Build:

trigger:
  - master
  - dev-*

variables:

  functionAppName: '$(FUNCTION_APP_NAME)'
  azureSubscription: '$(SERVICE_CONNECTION_NAME)'
  storageName: '$(FUCTION_APP_STORAGE_ACCOUNT_NAME)'
  resourceGroupName: '$(FUNCTION_APP_RESOURCE_GROUP_NAME)'

pool:
  vmImage: ubuntu-18.04
steps:
- task: UsePythonVersion@0
  displayName: "Setting python version to 3.6 as required by functions"
  inputs:
    versionSpec: '3.6' # Change Python version if necessary
    architecture: 'x64'
- script: |
    # dotnet restore # COMMENT OUT IF NOT USING FUNCTION EXTENSIONS
    # dotnet build --output './bin/' # COMMENT OUT IF NOT USING FUNCTION EXTENSIONS
    #source $(workingDirectory)/worker_venv/bin/activate
    #pip install -r requirements.txt
    #worker_venv/lib/python3.6/site-packages
    #pip install --target="$(workingDirectory)/.python_packages/lib/site-packages" -r requirements.txt
    pip install --target="worker_venv/lib/python3.6/site-packages" -r requirements.txt

- task: ArchiveFiles@2
  displayName: "Archive files"
  inputs:
    rootFolderOrFile: "$(System.DefaultWorkingDirectory)"
    includeRootFolder: false
    archiveFile: "$(System.DefaultWorkingDirectory)/build$(Build.BuildId).zip"
- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'
- task: AzureFunctionApp@1
  displayName: "Azure functions app deploy"
  inputs:
    # todo: make service connection dynamic for alpha,beta,ga
    azureSubscription: $(azureSubscription) # here the subscription or service connection name
    appType: functionAppLinux
    appName: $(functionAppName)
    packageForLinux: '$(Build.ArtifactStagingDirectory)/*.zip'

Expected behavior

Provide a description of the expected behavior.

After deployment i would expect that I can import the modules from requirements.txt from my function modules.

Actual behavior

Provide a description of the actual behavior observed.

Imports of modules from requirements.txt produce Module Not Found Error

Known workarounds

Provide a description of any known workarounds.

I was not finding any workaround except doing zip deploy via core tools which is not really usable when using CI/CD via DevOps.

Contents of the requirements.txt file:

Provide the requirements.txt file to help us find out module related issues.

azure-servicebus
numpy

Related information

Provide any related information

I shared the testapp i am using on github: https://github.com/georghildebrand/functestkjskdj Note that i am not breaking on import error. Instead i string the errors and output it at REST. I am doing this as the monitoring is very slow which makes debugging at higher iteration speed very hard.

[AvinZarlez]

Here is our working YAML for installing dependencies that is working for us.

    - bash: |
        python -m venv worker_venv
        source worker_venv/bin/activate
        pip install --target="$(workingDirectory)/.python_packages/lib/site-packages" -r requirements.txt
      workingDirectory: $(workingDirectory)
      displayName: 'Install application dependencies'

Perhaps you need to define workingDirectory as ‘$(System.DefaultWorkingDirectory)/’?

[AvinZarlez]

We just ran into this same issue.

Changing pip install -r requirements.txt in the YAML file to pip install --target="$(workingDirectory)/.python_packages/lib/site-packages" -r requirements.txt resolved it for us.

We definitely would expect to not have to specify the target in order for pip install -r requirements.txt to be deployed.