Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

SoftwareRequirement: environment modules changes to env not propagating to tool

See original GitHub issue

Expected Behavior

If I have a CWL file with a SoftwareRequirement that is resolved by an environment module, then I would expect that the environment variables set propagate to the tool execution.

Actual Behavior

They do not: only PATH is modified by the environment module

Workflow Code

Note this depends on having environment modules set up. Modify the cwl to load a module you have available:


#!/usr/bin/env cwl-runner
# -*- mode: yaml; -*-

cwlVersion: v1.1
class: CommandLineTool

        version: ["6.6.2"]

inputs: []

baseCommand: env

    type: stdout


- type: modules
  modulecmd: /usr/bin/modulecmd

cwltool runs OK:

 cwltool --beta-dependency-resolvers-configuration modules.yml env.cwl 
INFO /lustre/home/shared/dc118/sw/cwl/3.0.20201117141248/bin/cwltool 3.0.20201117141248
INFO Resolved 'env.cwl' to 'file:///lustre/home/dc118/nashvest/env.cwl'
INFO [job env.cwl] /tmp/7u0vetk_$ env > /tmp/7u0vetk_/d5514f0d054e63af38037aba76940ead71fd7324
Autoloading gcc/6.3.0
Autoloading gsl/2.6-gcc6
INFO [job env.cwl] completed success
    "output": {
        "location": "file:///lustre/home/dc118/nashvest/d5514f0d054e63af38037aba76940ead71fd7324",
        "basename": "d5514f0d054e63af38037aba76940ead71fd7324",
        "class": "File",
        "checksum": "sha1$488689425380b273e1e59a0dd3fdc884a57a842c",
        "size": 471,
        "path": "/lustre/home/dc118/nashvest/d5514f0d054e63af38037aba76940ead71fd7324"
INFO Final process status is success

Module that should have been loaded:

$ module show ncl

conflict        ncl
Autoloading gcc/6.3.0
module load     gcc/6.3.0
Autoloading gsl/2.6-gcc6
module load     gsl/2.6-gcc6
setenv          NCARG_ROOT      /lustre/sw/ncl/ncl-6.6.2
prepend-path    PATH /lustre/sw/ncl/ncl-6.6.2/bin
prepend-path    LD_LIBRARY_PATH /lustre/sw/ncl/ncl-6.6.2/lib
prepend-path    LD_LIBRARY_PATH /lustre/sw/ncl/ncl-6.6.2/lib64
prepend-path    MANPATH /lustre/sw/ncl/ncl-6.6.2/man

Actual environment

$ cat d5514f0d054e63af38037aba76940ead71fd7324 

If I change the command to say cwltool --beta-dependency-resolvers-configuration modules.yml --preserve-environment NCARG_ROOT env.cwl then the output of env is unchanged

Your Environment

  • cwltool version: 3.0.20201117141248

@mr-c and I have discussed briefly on gitter: Where Michael suggests pinging @jmchilton and @hmenager for comment?

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

rupertnashcommented, Nov 27, 2020

Maybe instead we should:

  • construct an environment as cwltool does now
  • run the shell commands produced by DependencyManager
  • capture that environment
  • use the captured environment to run the tool


rupertnashcommented, Nov 26, 2020

Galaxy docs say:

A DependencyManager attempts to resolve named and versioned dependencies by searching for them under a list of directories. Directories should be of the form:


and should each contain a file ‘’ which can be sourced to make the dependency available in the current shell environment.

A brief review of the resolvers suggests that they all act in the bold text manner. This make me think that the way cwltool is trying to use this feature (getting the PATH from the modified shell environment and then taking total control of the environment) is a bit wrong

Read more comments on GitHub >

github_iconTop Results From Across the Web

Environment Modules
The Environment Modules package is a tool that simplify shell initialization and lets users easily modify their environment during the session with ...
Read more >
Creating Your Own Environment Modules
Environment modules make it easy to use pre-installed software by updating the appropriate environment ... No changes are needed to ~/.bashrc in this...
Read more >
Environment Modules — Modules documentation
The Modules package is a tool that simplifies shell initialization and lets users easily modify their environment during a session using modulefiles.
Read more >
ansible docs modules
Override the default behavior by setting the environment variable ANSIBLE_ROLES_PATH. Each module ... Contributors develop and change modules and plugins .
Read more >
Configuring EasyBuild
For example: to configure EasyBuild to use Lmod as modules tool, the following ... not in environment variable values or command line option...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found