runway 1.18.3 does not load hook from module root directory

I am having an issue where we have a few module that have hooks and were running version 1.4.4, when we upgraded to version 1.18.3, we are getting an error ModuleNotFoundError: No module named. The first hook works fine during the deploy and the subsequent one fails. It works when running from my local PC but fails in AWS codebuild.

We added sys_path: ./ and our directory structure is like this:

module01
  templates
    template.yaml
  mod01_hooks_dir
    some_file.py
  stacker.yml

module02
  templates
    template.yaml
  mod02_hooks_dir
    some_file.py
  stacker.yml

in stacker.yml

sys_path: ./

pre_build:
  some_hook:
    mod01_hooks_dir.some_file.some_function

pre_build:
  some_hook:
    mod02_hooks_dir.some_file.some_function

Expected behavior Runway should run all hooks as expected

Screenshots

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/cfngin/hooks/utils.py", line 68, in handle_hooks
    method = load_object_from_string(hook.path, try_reload=True)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/util.py", line 554, in load_object_from_string
    importlib.import_module(module_path)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tgw_attach_hooks'
Shared resource transit_gateway is already accepted and active
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.7.6/bin/runway", line 8, in <module>
    sys.exit(cli())
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/_cli/main.py", line 31, in invoke
    return super(_CliGroup, self).invoke(ctx)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/_cli/commands/_takeoff.py", line 26, in takeoff
    ctx.forward(deploy, **kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/core.py", line 628, in forward
    return self.invoke(cmd, **kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/_cli/commands/_deploy.py", line 41, in deploy
    Runway(ctx.obj.runway_config, ctx.obj.get_runway_context()).deploy(deployments)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/__init__.py", line 59, in deploy
    "deploy", deployments if deployments is not None else self.deployments
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/__init__.py", line 234, in __run_action
    variables=self.variables,
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/components/_deployment.py", line 373, in run_list
    )[action]()
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/components/_deployment.py", line 187, in deploy
    return self.__sync("deploy")
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/components/_deployment.py", line 331, in __sync
    self.run(action, region)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/components/_deployment.py", line 244, in run
    variables=self._variables,
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/components/_module.py", line 336, in run_list
    )[action]()
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/components/_module.py", line 170, in deploy
    return self.run("deploy")
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/core/components/_module.py", line 227, in run
    inst[action]()
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/module/cloudformation.py", line 41, in deploy
    cfngin.deploy(force=self.options["environment"])
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/cfngin/cfngin.py", line 131, in deploy
    action.execute(concurrency=self.concurrency, tail=self.tail)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/cfngin/actions/base.py", line 160, in execute
    self.pre_run(**kwargs)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/cfngin/actions/build.py", line 527, in pre_run
    handle_hooks("pre_build", hooks, self.provider, self.context, dump, outline)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/cfngin/actions/build.py", line 194, in handle_hooks
    utils.handle_hooks(stage=stage, hooks=hooks, provider=provider, context=context)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/cfngin/hooks/utils.py", line 68, in handle_hooks
    method = load_object_from_string(hook.path, try_reload=True)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/site-packages/runway/util.py", line 554, in load_object_from_string
    importlib.import_module(module_path)
  File "/root/.pyenv/versions/3.7.6/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tgw_attach_hooks'

Additional context If I revert back to 1.4.4, it works fine.

[newbeeeeeee]

@ITProKyle , Thanks for following up. I have scheduled the upgrade tonight and I will update tomorrow morning.

[ITProKyle]

I am using version 3.8.5 on my computer

Great! So it shouldn’t be a mismatch in import functionality. The biggest places this occurs is python 2 vs 3.

I was not aware of using ./../ for sys_path… I do have __init__.py file in every module’s hook directory

For ./../ to work, you would also want to add an init file to to root of the Runway module directory (e.g. module01/__init__.py). There needs to be one at each level you are trying to traverse with python importing.

To get a better idea of the paths that a python import can see while Runway is importing hooks, you can add the --debug flag. It will add a lot of additional output but most importantly it will display the value of sys.path and sys.modules as they are being added to and reverted between the execution of each Runway module. You can use these values to ensure the expected path is being added.