Generating specialize C helper code automatically

This issue is about generating specialized code helpers automatically. A code helper is considered specialized, if it doesn’t work on the most general form, e.g. two PyObject *, but instead e.g. on two PyUnicode_Object instead, where the task can be identified more immediately as PyUnicode_Concat, just concatenating the two strings.

In Nuitka as a result of compile time analysis during optimization we have increasingly more shape knowledge. Often with relatively concrete types. A shape is a set of characteristics that the uses of a expose, e.g. could mean it’s “iterable”, “indexable”, as example shapes. Many times however, shapes are known to be one or two types specifically, int, bool, unicode, etc. but often also one of the other, e.g. unicode or str.

Currently in Nuitka there are helper functions used for code that does all sort of operations. There are very general ones, and then the more specialized ones. The general ones use OBJECT as a type indicator in the function names (where there is specialization). As of writing this, this is only the case for ‘+’ and += (_INPLACE) are optimized for a subset of types and shapes. And there are comparison operations, <, <=, ==, !=, >=, >, but they are only specialized for INT`, i.e. Python2 integers.

These helpers are hand optimized versions of more general code that represents what e.g. PyNumber_Add does. It looks at type slots and implements special behavior like the concatenation that + also does in Python. Many of their forms are not optimized, or not yet fully optimized. The current form is more of a demonstration of feasibility than a practical approach. This will only be sufficient to get few operations optimized, but it will never scale and doing this by hand is error prone and could lead to bugs.

Instead, what we need is a code generator with sufficient type knowledge about the two arguments, so that it can make decisions automatically. For instance, having a nb_add is a given in for long, float, but not so for a list. So instead of making checks, and executing branched code, the code generator would not generate that check, or do so in a way that the C compiler will know it’s not happening.

Other examples are checks like type1 == type2. Many times, this comes down to branch decisions. And the different shapes allow some of these, some do not.

[kayhayen]

Good to hear @riya-17 and it’s easy to get stuck with things like this.

In an attempt at mentoring, let me tell you how I see these things. Many times, I find people discover a tool, let’s say a hammer. And then they desperately search for nails. And they come up with things that are not really nails, and hammer them in things, where they don’t belong. This can be a great learning exercise, but I generally am too lazy for that. And I consider laziness a virtue.

It is very easy to get lost and spend a lot of time on non-problems this way. Contrast this with searching a problem, then looking for the tool that matches it. A hammer seems to be intended to to attach things, so lets use it. Depending on what it is, it may happen that we don’t need no nails, because e.g. the way the parts we want to attach are constructed. They may just fit stiff together and hold. Maybe they get loose after some time, or we want to put more on it, then we need to search for a concept like nails, and will find it.

So my philosophy about things like Jinja is to simply put it to the intended use, and see where I hit limits. When I feel this is way to verbose, there ought to be some to save repetitive tasks, or there ought to be a way to make sure that all templates are created identical, or share the same values, then I will look into it.

This explains why first thing I do, is to paste a mock up (in our case an actually really good version that Nuitka currently uses) of one thing, see what I can do with it, that brings me closer to solving the problem.

if a need for an environment, doesn’t enter the picture, then they probably are not for us.

This approach has served me well. Also, I have had embarrassing failures of course, where reading up ahead of time, could be made me avoid big mistakes. But once I discover the need for nails, I am sure, I understand what I want to use them for. I really don’t like sitting with a hammer and a nail, and wondering what they are good for. 😃

As always, there is a balance. But experience has taught me to always dive in rather immediately, and then look at the documentation after a while, to see if I missed anything important, and then being able to understand how it relates to stuff.

There are these tools, py.test is also a fine example of those, which carry a lot of flexibility, and a lot of baggage with them. So naturally they will have solutions for problems I never encountered, and simply cannot understand by looking at their nails without seeing the hammer, nor the problem they are intended for. In these cases I tend to just ignore it, until I realize that I have a problem that is not well solved with the part of the tool set I used so far. Then I can revisit it, because now I see probably all the things necessary to understand how it helps me.

[kayhayen]

@riya-17 When you use Eclipse and with it PyDev, you are using an implementation of Python in Java for the JVM, called Jython. However, what you typically do is to configure an “interpreter”. The newly pushed PyDev Nuitka project to current develop branch refers to Python3.6, but you are very free in reality.

But what this does is to run “python” which is the same you use on the command line. Note that Nuitka will only work with CPython and not Jython, or IronPython, but is tied to CPython due to its use of the Python C/API.

So differences would only happen if you ended up using different binaries somehow.