Support "const" modifier on fused types

cdef void c_encode_by_bits(const npy_intp n, const npy_intp m, const long * a, 
                           const long bitshift, integer * out) nogil:
    cdef npy_intp i, j 
    cdef long shift
    cdef integer value

    for i in range(n):
        value = 0
        shift = 0
        for j in range(m):
            value += a[i * m + j] << shift
            shift += bitshift
        out[i] = value
            
cdef void c_decode_by_bits(const npy_intp n, const npy_intp m, 
                           const integer * a, const long bitshift, 
                           long * out) nogil:
    cdef npy_intp i, j, k
    cdef long shift, mask = (1 << bitshift) - 1
    
    for i in range(n):
        shift = 0
        for j in range(m):
            out[i * m + j] = (<long> a[i] >> shift) & mask 
            shift += bitshift

ctypedef fused integer:
    char
    unsigned char
    short
    unsigned short
    int
    unsigned int
    long
    unsigned long

[1/1] Cythonizing locality/helpers/encode.pyx
Traceback (most recent call last):
  File "build.py", line 99, in <module>
    ext_modules = cythonize(cy_extensions) + c_extensions,
  File "/usr/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 934, in cythonize
    cythonize_one(*args)
  File "/usr/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1039, in cythonize_one
    result = compile([pyx_file], options)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Main.py", line 686, in compile
    return compile_multiple(source, options)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Main.py", line 664, in compile_multiple
    result = run_pipeline(source, options, context=context)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Main.py", line 494, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Pipeline.py", line 340, in run_pipeline
    data = phase(data)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Pipeline.py", line 53, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/ModuleNode.py", line 137, in process_implementation
    self.generate_c_code(env, options, result)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/ModuleNode.py", line 365, in generate_c_code
    self.body.generate_function_definitions(env, code)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 436, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 436, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 1754, in generate_function_definitions
    self.generate_function_header(code, with_pymethdef=with_pymethdef)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 2466, in generate_function_header
    arg_decl = arg.declaration_code()
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/PyrexTypes.py", line 3050, in declaration_code
    return self.type.declaration_code(self.cname, for_display)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/PyrexTypes.py", line 2372, in declaration_code
    for_display, dll_linkage, pyrex)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/PyrexTypes.py", line 1475, in declaration_code
    return self.const_base_type.declaration_code("const %s" % entity_code, for_display, dll_linkage, pyrex)
  File "/usr/lib/python3.6/site-packages/Cython/Compiler/PyrexTypes.py", line 1533, in declaration_code
    raise Exception("This may never happen, please report a bug")

Replacing “integer” with “long” stops the exception.

[scoder]

It looks like this is not difficult but a bit of work. Basically, in FusedNode.py, the fused_base_types must be generated without const (i.e. if type.is_const: type = type.const_base_type), but then the specialisation afterwards must look into const types to see if they are fused, and then specialise the type into the corresponding const type again. Not sure if this can be done locally in PyrexTypes.py or if it requires some non-local (but probably repetitive) code changes. PR welcome.

[rth]

We spend half a day investigating this issue with @lesteve at EuroScipy. Posting a (partial) status update in case it might be helpful.

A (possibly) minimal unit test that fails can be found in this branch, where to produce an error it is sufficient to run,

python -m Cython.Build.Cythonize cython/tests/memoryview/const_fused_type_memoryview.pyx

or alternatively,

python runtests.py -T 1772

(though the traceback in the latter case is less detailed).

Below are a few additional observation (from someone with no formal CS background or in depth understanding how Cython works, and possibly I missed some contribution docs),

• Looking at the generated C code,

  • the difference between float and const float declaration is just a few additional “const” declaration
  • the difference between float and floating (fused float type) is fairly large (though this has been tested with memoryviews: with plain scalars, it might not longer be the case)

  so the probable expected result of the PR that would fix this issue might be a slight modification of the resulting C code.

• in the overall processing pipeline the processing step which is probably affected is AnalyseDeclarationsTransform (cf @scoder’s above comment https://github.com/cython/cython/issues/1772#issuecomment-376629824).

• we had some trouble specifying breakpoints for pdb in the code as it appears that some source files are cythonized. Using git clean -xdf could be a possible workaround.

There are probably much better ways of approaching this issue for someone with better knowledge of Cython code base.