DBT cli throw memory exception for ffi.callback() on Mac with Apple Sillicon

Describe the bug

dbt run or dbt --version throw error Cannot allocate write+execute memory for ffi.callback().

Steps To Reproduce

On Macbook pro with M1 chip brew tap fishtown-analytics/dbt brew install dbt add default profile.yml with connection to snowflake create new project dbt init dbt run with the two sample models, or simply dbt --version

Expected behavior

Should complete successfully, instead of throwing exception. The process works fine on Intel-based Macbook pro.

Screenshots and log output

> dbt run
Running with dbt=0.19.0
Found 3 models, 8 tests, 0 snapshots, 0 analyses, 143 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

The operating system you’re using: macOS Big Sur 11.2.2

The output of python --version: 3.8.7

Additional context

Add any other context about the problem here.

[vklimontovich]

TL;DR: if you experience the error during dbt run, just add those lines to ~/.dbt/profiles.yml (see the full doc in profiles.yml here)

config:
    send_anonymous_usage_stats: False

Details:

Here’s the stack trace (MacBook Air (M1, 2020) running Mac OS 11.3.1). The exception is raised in user behavior tracking code. Quickfix will be disabling tracking (see above). I guess the proper way to handle this would be wrapping tracker.flush() with try/except. Failure to track isn’t a critical error after all

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/opt/homebrew/Cellar/python@3.8/3.8.10/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 229, in track_run
    dbt.tracking.flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/tracking.py", line 428, in flush
    tracker.flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/tracker.py", line 218, in flush
    emitter.sync_flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 182, in sync_flush
    Emitter.flush(self)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 149, in flush
    self.send_events(self.buffer)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 206, in send_events
    status_code = self.http_post(data).status_code
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/tracking.py", line 68, in http_post
    r = requests.post(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connection.py", line 343, in connect
    self.ssl_context = create_urllib3_context(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 290, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
    self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1028, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/OpenSSL/SSL.py", line 331, in __init__
    self.callback = _ffi.callback(
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

[mr-m0nst3r]

Hey guys,

I fixed this ffi problem by reinstalling the libffi using homebrew, and install cffi wheel: https://files.pythonhosted.org/packages/3e/9b/660d6da900af1976a8b4efea713a7ce9e514bf4659eff9b17f90f00be1cf/cffi-1.15.0-cp39-cp39-macosx_11_0_arm64.whl