Error to deploy spaceflights tutorial in AWS Steps Function

Description

I tried to follow the tutorial to deploy the spaceflights template project on AWS Step Functions (https://antonymilneqb.github.io/kedro-docs/10_deployment/10_aws_step_functions.html). I follow the steps written in the tutorial but I get an error when I try to run the service.

Context

Steps to Reproduce

1. python -m venv venv & source venv/bin/activate
2. pip install kedro==0.17.1 & kedro info
3. kedro new --starter=spaceflights
4. cd space (project folder)
5. create new configuration environment to prepare a compatible data catalog (conf/aws/catalog.yml) with the aws file paths
6. pip install -r src/requirements.txt
7. kedro package
8. create lambda_handler.py with a copy the code available in the tutorial and changed the row 7 in configure_project("space")
9. create Dockerfile with a copy the code available in the tutorial
10. push the project on ECR
11. pip install deploy_requirements.txt
12. go in the script deploy.py with a copy the code available in the tutorial and changed the rows 31 to s3_data_bucket_name = ("<bucket-name>") with my bucket name
13. create the file cdk.json with a copy the code available in the tutorial & run cdk deploy
14. go to aws and try to run the step function, which is deployed but it throws an error

Expected Result

The service step function starts the execution correctly.

Actual Result

The step functions throws an error in the first step, it seems that the lambda can’t find the functions implemented.

– Error Received.

{
  "errorMessage": "[Errno 38] Function not implemented",
  "errorType": "OSError",
  "requestId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "stackTrace": [
    "  File \"/home/app/lambda_handler.py\", line 7, in handler\n    configure_project(\"space\")\n",
    "  File \"/home/app/kedro/framework/project/__init__.py\", line 211, in configure_project\n    settings.configure(settings_module)\n",
    "  File \"/home/app/dynaconf/base.py\", line 223, in configure\n    self._wrapped = Settings(settings_module=settings_module, **kwargs)\n",
    "  File \"/home/app/dynaconf/base.py\", line 271, in __init__\n    self.validators.validate()\n",
    "  File \"/home/app/dynaconf/validator.py\", line 318, in validate\n    validator.validate(self.settings)\n",
    "  File \"/home/app/dynaconf/validator.py\", line 172, in validate\n    self._validate_items(settings, settings.current_env)\n",
    "  File \"/home/app/kedro/framework/project/__init__.py\", line 58, in _validate_items\n    super()._validate_items(settings, env)\n",
    "  File \"/home/app/dynaconf/validator.py\", line 183, in _validate_items\n    self.default(settings, self)\n",
    "  File \"/home/app/kedro/framework/project/__init__.py\", line 49, in validator_func\n    return getattr(importlib.import_module(module), class_name)\n",
    "  File \"/usr/local/lib/python3.8/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n",
    "  File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import\n",
    "  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n",
    "  File \"<frozen importlib._bootstrap>\", line 961, in _find_and_load_unlocked\n",
    "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
    "  File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import\n",
    "  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n",
    "  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n",
    "  File \"<frozen importlib._bootstrap>\", line 671, in _load_unlocked\n",
    "  File \"<frozen importlib._bootstrap_external>\", line 843, in exec_module\n",
    "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
    "  File \"/home/app/kedro/framework/session/__init__.py\", line 32, in <module>\n    from .session import KedroSession, get_current_session\n",
    "  File \"/home/app/kedro/framework/session/session.py\", line 54, in <module>\n    from kedro.framework.session.store import BaseSessionStore\n",
    "  File \"/home/app/kedro/framework/session/store.py\", line 73, in <module>\n    class ShelveStore(BaseSessionStore):\n",
    "  File \"/home/app/kedro/framework/session/store.py\", line 76, in ShelveStore\n    _lock = Lock()\n",
    "  File \"/usr/local/lib/python3.8/multiprocessing/context.py\", line 68, in Lock\n    return Lock(ctx=self.get_context())\n",
    "  File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 162, in __init__\n    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)\n",
    "  File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 57, in __init__\n    sl = self._semlock = _multiprocessing.SemLock(\n"
  ]
}

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used: 0.17.1
• Python version used: 3.8.10
• Operating system and version: Linux Ubuntu 20.04 LTS
• cdk version used: 1.128.0
• npm version used: 6.14.15

[AlbertoGarau]

Hello @MerelTheisenQB My tools version:

• kedro, version 0.18.3
• Python 3.10.6
• cdk 2.43.1 (build c1ebb85)
• No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy

[ankatiyar]

This issue comes from the import of ShelveStore which uses multiprocessing.lock which isn’t supported on AWS Lambda functions. ShelveStore has been moved out to its own module in #1614, therefore, will not be imported, which will resolve this issue in the upcoming release of Kedro 0.18.4.