OSError: [Errno 38] Function not implemented

Describe the bug

Using the DBT Python API, I run by using this function: dbt.main.handle_and_check()

I’m also using it with an AWS Lambda that gets deployed (zipped and then gets unzipped by serverless-python-requirements).

I’m setting the argument --profiles-dir to os.path.dirname(__file__) and --project-dir to os.path.dirname(__file__). Both of the profiles.yml and dbt_project.yml are located in the same location under ./data_warehouse/snowflake_dbt

The error I’m getting: OSError: [Errno 38] Function not implemented

Any idea how to solve the issue? when I run the lambda locally, everything works fine. I think it’s due to the combination of AWS Lambda and DBT…

Screenshots and log output

Running with dbt=0.18.1
--- Logging error ---
Traceback (most recent call last):
  File "/var/task/data_warehouse/snowflake_dbt/main.py", line 62, in handler
    results, success = run_dbt(e.name.value, e.macro, e.event_vars, e.event_args)
  File "/var/task/data_warehouse/snowflake_dbt/main.py", line 94, in run_dbt
    results, success = dbt.main.handle_and_check(params)  # pylint: disable=maybe-no-member
  File "/tmp/sls-py-req/dbt/main.py", line 202, in handle_and_check
    task, res = run_from_args(parsed)
  File "/tmp/sls-py-req/dbt/main.py", line 241, in run_from_args
    task = parsed.cls.from_args(args=parsed)
  File "/tmp/sls-py-req/dbt/task/base.py", line 156, in from_args
    return super().from_args(args)
  File "/tmp/sls-py-req/dbt/task/base.py", line 103, in from_args
    return cls(args, config)
  File "/tmp/sls-py-req/dbt/task/runnable.py", line 55, in __init__
    super().__init__(args, config)
  File "/tmp/sls-py-req/dbt/task/base.py", line 151, in __init__
    register_adapter(self.config)
  File "/tmp/sls-py-req/dbt/adapters/factory.py", line 182, in register_adapter
    FACTORY.register_adapter(config)
  File "/tmp/sls-py-req/dbt/adapters/factory.py", line 105, in register_adapter
    adapter: Adapter = adapter_type(config)  # type: ignore
  File "/tmp/sls-py-req/dbt/adapters/base/impl.py", line 162, in __init__
    self.connections = self.ConnectionManager(config)
  File "/tmp/sls-py-req/dbt/adapters/base/connections.py", line 42, in __init__
    self.lock: RLock = flags.MP_CONTEXT.RLock()
  File "/var/lang/lib/python3.7/multiprocessing/context.py", line 72, in RLock
    return RLock(ctx=self.get_context())
  File "/var/lang/lib/python3.7/multiprocessing/synchronize.py", line 187, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
  File "/var/lang/lib/python3.7/multiprocessing/synchronize.py", line 59, in __init__
    unlink_now)
OSError: [Errno 38] Function not implemented

I also set the threads to 1 in profiles.yml but no help…

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.18.1
   latest version: 0.18.1

Up to date!

Plugins:
  - bigquery: 0.18.1
  - snowflake: 0.18.1
  - redshift: 0.18.1
  - postgres: 0.18.1

The operating system you’re using:

The output of python --version: Python 3.7

[Gunnnn]

Hello! I also have the same issue when trying to run DBT on Databricks cluster. dbt command itself runs smoothly, but dbt debug gives error, mentioned in this thread.

[RoniFinTech]

Maybe the method suggested by AWS can be used to overcome this for lambdas? https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/ On the other hand, you can use multiprocessing.Pipe instead of multiprocessing.Queue to accomplish what you need without getting any errors during the execution of the Lambda function.