Importing Prophet breaks Pandas/Google BigQuery

We recently introduced Prophet into our project and have discovered there’s some sort of conflict between Prophet, Pandas and Google’s BigQuery API. Not sure which library is at fault here exactly.

This works as expected (prompts for an authentication key):

import pandas as pd

df = pd.read_gbq('test', 'test')

But this gives a nightmarish set of exceptions:

import pandas as pd
from fbprophet import Prophet

df = pd.read_gbq('test', 'test')

Traceback is as follows:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 879, in _find_spec
AttributeError: 'PyxImporter' object has no attribute 'find_spec'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    df = pd.read_gbq('give me an error', 'a project id')
  File "./venv/lib/python3.6/site-packages/pandas/io/gbq.py", line 100, in read_gbq
    **kwargs)
  File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 906, in read_gbq
    dialect=dialect, auth_local_webserver=auth_local_webserver)
  File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 202, in __init__
    self.credentials = self.get_credentials()
  File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 214, in get_credentials
    credentials = self.get_application_default_credentials()
  File "./venv/lib/python3.6/site-packages/pandas_gbq/gbq.py", line 243, in get_application_default_credentials
    credentials, _ = google.auth.default(scopes=[self.scope])
  File "./venv/lib/python3.6/site-packages/google/auth/_default.py", line 287, in default
    credentials, project_id = checker()
  File "./venv/lib/python3.6/site-packages/google/auth/_default.py", line 164, in _get_gae_credentials
    from google.auth import app_engine
  File "./venv/lib/python3.6/site-packages/google/auth/app_engine.py", line 32, in <module>
    from google.appengine.api import app_identity
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 946, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 881, in _find_spec
  File "<frozen importlib._bootstrap>", line 855, in _find_spec_legacy
  File "./venv/lib/python3.6/site-packages/pyximport/pyximport.py", line 253, in find_module
    fp, pathname, (ext,mode,ty) = imp.find_module(fullname,package_path)
  File "~/.pyenv/versions/3.6.1/lib/python3.6/imp.py", line 270, in find_module
    "not {}".format(type(path)))
RuntimeError: 'path' must be None or a list, not <class '_frozen_importlib_external._NamespacePath'>

Running in a fresh virtual environment with Python 3.6.1 on macOS 10.12.3, reproduced on multiple machines with this test setup (as in, outside of our project).

Requirements after installing with pip install pandas pandas-gbq google-api-python-client fbprophet are:

cachetools==2.0.1
certifi==2017.11.5
chardet==3.0.4
cycler==0.10.0
Cython==0.27.3
fbprophet==0.2.1
google-api-python-client==1.6.4
google-auth==1.2.1
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.2.0
httplib2==0.10.3
idna==2.6
matplotlib==2.1.0
numpy==1.13.3
oauth2client==4.1.2
oauthlib==2.0.6
pandas==0.21.0
pandas-gbq==0.2.0
pyasn1==0.3.7
pyasn1-modules==0.1.5
pyparsing==2.2.0
pystan==2.17.0.0
python-dateutil==2.6.1
pytz==2017.3
requests==2.18.4
requests-oauthlib==0.8.0
rsa==3.4.2
six==1.11.0
uritemplate==3.0.0
urllib3==1.22

[bletham]

I’m able to replicate this issue.

pd.read_gbq at some point calls from google.auth import app_engine and that is where the issue is coming from. The following two lines will produce the error:

from fbprophet import Prophet
from google.auth import app_engine

Importantly, the order matters here: The following two lines actually work:

from google.auth import app_engine
from fbprophet import Prophet

It seems that google.auth is using pyximport to handle some cython code, and that is conflicting with the pystan’s use of cython in compiling the model. The conflict is really between those two packages - this code will produce the error with no fbprophet involved:

import pystan
model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
model = pystan.StanModel(model_code=model_code)  # this will take a minute
from google.auth import app_engine

We rely on the pystan model compilation pretty fundamentally so I’m not too optimistic about being able to fix this on our side. I suspect that either google.auth will have to stop using pyximport, or pystan model compilation will have to do something different to avoid this conflict. I’d recommend trying to get an upstream fix, but in the meantime we’ll leave this open in case someone else does have any ideas for how to fix this in fbprophet.

[amintapan]

To install fbprophet on AWS Sagemaker Jupyter Notebook: !conda install -c plotly plotly==3.10.0 --yes !conda install -c conda-forge fbprophet --yes from fbprophet import Prophet Prophet()