Document how to embed Cython modules in C/C++ applications

The documentation on embedding is inherently lacking. Even the one in the CPython docs is incomplete at best.

There are some hints in the Wiki, there’s the --embed option to the cython and cythonize frontends, the embedding demo, the PyImport_AppendInittab() function in CPython and the documentation of the module init function in CPython.

All of these are incomplete and none of them gives the whole picture in one place. There should be a dedicated documentation page in the Cython docs for this, and the documentation in the CPython docs is also worth improving.

[scoder]

Please don’t use the bug tracker for general discussions. We have the cython-users mailing list for this.

[da-woods]

FWIW I have a working example of how to call PyInit_* on a multi-phase init module therefore avoiding the need to use PyImport_AppendInittab. Essentially it’s running through the multi-phase init steps manually. I’m not sure if we want to put it in any updated embedding docs, but it’s posted below in case anyone wants to document it properly:

I suspect this comes under

Most importantly, DO NOT call the module init function instead of importing the module. This is not the right way to initialise an extension module. (It was always wrong but used to work before, but since Python 3.5, it is wrong and no longer works.)

though

#include <Python.h>

#include "embed_example.h"

int main() {
    int result = 1;
    PyObject *spec = NULL, *spec_globals = NULL, *mod = NULL;
    Py_Initialize();
    PyObject *maybe_mod = PyInit_embed_example();
    if (!maybe_mod) goto bad;
    if (Py_IS_TYPE(maybe_mod, &PyModuleDef_Type)) {
        // multi-phase init
        spec_globals = PyDict_New();
        if (!spec_globals) goto bad;
        PyObject *res = PyRun_String(
            "import importlib.machinery as im\n"
            // Note that Cython doesn't actually use the loader
            // so it can be None. It'd be better to
            // provide something more useful though.
            "spec = im.ModuleSpec('embed_example', None)\n",
            Py_file_input, spec_globals, spec_globals);
        Py_XDECREF(res); // don't use res whether or not it's set
        if (!res) goto bad;
        spec = PyDict_GetItemString(spec_globals, "spec");
        if (!spec) goto bad;               
        
        mod = PyModule_FromDefAndSpec(
            (PyModuleDef*)maybe_mod,
            spec);
        if (!mod) goto bad;
        int execRes = PyModule_ExecDef(mod, (PyModuleDef*)maybe_mod);
        if (execRes) goto bad;
    } else {
        mod = maybe_mod;
    }
    
    func(); // cdef public in the module being embedded.
    
    result = 0;
    if (0) {
        bad:
        PyErr_Print();
    }
    // The moduledef isn't an owned reference so doesn't get decref'd
    Py_XDECREF(mod);
    Py_XDECREF(spec);
    Py_XDECREF(spec_globals);
    Py_Finalize();
    return result;    
}

I remain sceptical of whether embedding Cython in C/C++ applications is actually useful (because people expect it to embed their module and all its dependencies), but multi-phase init is definitely something that people fight with, and additional ways to work with it are probably good.