yang library packaging

We need a library that is capable of loading YANG data from JSON and validating that against a YANG model, supplying the default values, etc. We’ve determined that yangson has performance issues which are WONTFIX by upstream, so we need something else. It looks that pyang doesn’t have yang data validation out-of-box, which means that we need to use something else.

There’s a C library for working with YANG, and there are Python bindings for its previous version. The C API is rather complex, there’s a bunch of rules about lifetime management of various structs, which are neatly abstracted away by a C++ wrapper. Ideally, I would like to wrap the C++ bindings with pybind11 because that’s a technology stack that I’m familiar with.

In Python, there are ways to distribute C or C++ extensions in a precompiled form. The most popular one is currently the manylinux approach which involves, essentially, taking an ancient Linux distribution (so as to not depend on “anything modern”, using very old versions of all libraries instead), build the extension on this old system, and distribute the resulting binaries. The goal/hope here is to achieve compatibility with “everything”. There are scripts like cibuildwheel which make it easy to integrate automatic building into CI/CD, including automatic pushing of resulting binaries for variety of architectures. Unfortunately, because libyang-cpp actively uses the modern C++20 standard, building currently requires a new enough GCC or clang (upstream is building on GCC 11, clang 12, etc). This is very tricky to do on ancient Linux distros because it is not “just the compiler”, it’s about the runtime libraries, the rest of the toolchain, etc.

One possible approach would be to build the libyang and libyang-cpp statically. That would involve “baking in” the libpcre2 and parts of the glibc, and based upon my preliminary checks, that “sounds possible”. A downside of this approach is that we would be bundling stuff, and that one cannot benefit from the existing scripting/CI infrastructure for automatic wheel building for various architectures. As a practical result, it would be hard to build an extension for “Linux/ARM”, for example, or even for Windows or for a Mac. As a result, GNPy would be hard to install on these platforms because you would essentially need a modern C++ compiler.

Another approach is to try to refresh the older Python bindings for the C library, and wrap everything again with CFFI. This means that we would have to duplicate all the work which has been already done for the C++ bindings once again for Python. The upside is that we won’t need to reinvent the build scripts, and it looks that someone is already trying to do this. We could still bundle the libyang.so.2 in the resulting wheel, and perhaps libpcre2 as well.

[ojnas]

Maybe another track that could be explored is to try and improve the performance of yangson. By a couple of simple optimizations I was able to shorten validation time on my computer from 32 seconds to 6 seconds for the Sweden OpenROADM topology and from 560 seconds to 32 seconds for a larger 50 ROADM node topology. This is maybe still too slow but it doesn’t seem impossible to find further optimizations. Of course, I can’t be sure that the optimizations does not have any unwanted side effects, even though they pass the upstream tests. If you want to try it for yourself you can clone my fork here:

https://github.com/ojnas/yangson

[AndreaDAmico]

Would this be an issue in Polito @AndreaDAmico ?

@AndreaDAmico , on what system are you developing GNPy?

I use Linux. A couple of colleagues of mine use GNPy in Windows.