Error while using sk-learn in cythonized code

Description

Error while using sk-learn in cythonized code. A small script using sk-learn fails when it is run cythonized. It doesn’t fail when running in pure python.

Steps/Code to Reproduce

Here is the script: https://gist.github.com/yoelk/818082db9d333f87c1d5f3e0820a25c1

I’ve also uploaded it in a project to GitHub, for reproducing the problem: https://github.com/yoelk/sklearn_cython_problem Please see the README.md file for how to reproduce

Expected Results

When running the code directly (without cythonization), there no Error is thrown. I’d like it to be the same with cythonization.

Actual Results

Traceback (most recent call last):
  File "run_cython_test.py", line 16, in <module>
    sys.exit(main())
  File "src/feature_union/cli.py", line 99, in feature_union.cli.main (build/src/feature_union/cli.c:2626)
    combined_features = FeatureListUnion([("pca", pca), ("univ_select", selection)])
  File "src/feature_union/cli.py", line 26, in feature_union.cli.FeatureListUnion.__init__ (build/src/feature_union/cli.c:1237)
    super(FeatureListUnion, self).__init__(transformer_list, **kwargs)
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/pipeline.py", line 630, in __init__
    self._validate_transformers()
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/pipeline.py", line 664, in _validate_transformers
    self._validate_names(names)
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/utils/metaestimators.py", line 65, in _validate_names
    invalid_names = set(names).intersection(self.get_params(deep=False))
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/pipeline.py", line 646, in get_params
    return self._get_params('transformer_list', deep=deep)
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/utils/metaestimators.py", line 26, in _get_params
    out = super(_BaseComposition, self).get_params(deep=False)
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/base.py", line 227, in get_params
    for key in self._get_param_names():
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/base.py", line 197, in _get_param_names
    init_signature = signature(init)
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/externals/funcsigs.py", line 59, in signature
    sig = signature(obj.__func__)
  File "/home/joel/.virtualenvs/feature_union/local/lib/python2.7/site-packages/sklearn/externals/funcsigs.py", line 173, in signature
    raise ValueError('callable {0!r} is not supported by signature'.format(obj))
ValueError: callable <cyfunction FeatureListUnion.__init__ at 0x7f14db64f410> is not supported by signature

Versions

>>> import platform; print(platform.platform())
Linux-4.10.0-38-generic-x86_64-with-Ubuntu-16.04-xenial
>>> import sys; print("Python", sys.version)
('Python', '2.7.12 (default, Nov 19 2016, 06:48:10) \n[GCC 5.4.0 20160609]')
>>> import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.13.3')
>>> import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.19.1')
>>> import sklearn; print("Scikit-Learn", sklearn.__version__)
('Scikit-Learn', '0.19.0')

[jnothman]

this sounds like a strange if not inappropriate use. If you want to use scikit-learn components without modification in your obfuscated/compiled code, you can still import them and reveal their source. If on the other hand you want to modify scikit-learn code, that’s not our problem.

I’d be inclined to close this as something we don’t intend to support.

[yoelk]

I know, it worked for me as well (sorry for not writing it explicitly before). I’m porting our code to python3 as we speak As this is the solution, I’ll close this issue. I still have to check it doesn’t fall elsewhere with python3. If it does, I’ll open a new issue.

Thanks a lot for the attention!