[Feature Request] Lazy variable/dependency evaluation

Hello,

lets say your project includes building a lot of libraries, and the dependency graph of these libraries is given by this acyclic graph. You might solve this by writing

libEvar = library('mylibE', 'fileE.cpp', link_with : [])
libDvar = library('mylibD', 'fileD.cpp', link_with : [libEvar])
libCvar = library('mylibC', 'fileC.cpp', link_with : [libDvar, libEvar])
libBvar = library('mylibB', 'fileB.cpp', link_with : [libDvar])
libAvar = library('mylibA', 'fileA.cpp', link_with : [libBvar, libCvar, libEvar])

This works, but there is one major problem: It breaks once you change the order of those lines. If all of these lines are in the same meson.build file, then putting them into the correct order is easy. But if these lines are scattered across multiple meson.build files in nested subdirectories, then it might no longer be possible (Imagine if libEvar and libCvar are in a subdirectory of libDvar’s directory).

One solution would be to write this:

libs = []
# The order of the following 5 lines is irrelevant
libs += [['mylibA', 'fileA.cpp', ['mylibB', 'mylibC', 'mylibE']]]
libs += [['mylibB', 'fileB.cpp', ['mylibD']]]
libs += [['mylibC', 'fileC.cpp', ['mylibD', 'mylibE']]]
libs += [['mylibD', 'fileD.cpp', ['mylibE']]]
libs += [['mylibE', 'fileE.cpp', []]]

lookup = {}
foreach step : [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0]
        foreach lib : libs
            flag = true
            deps = []
            foreach dep : lib[2]
                if dep not in lookup
                     flag = false
                else
                     deps += lookup[dep]
                endif
            endforeach
            assert(step == 1 or flag, 'error building the dependency graph')
            if flag and lib[0] not in lookup
                lookup += {lib[0] : library(lib[0], lib[1], link_with : deps)}
            endif   
        endforeach
endforeach

This basically does what I want/need, but it feels like very bad style. Meson should have some functionality to build these acyclic graphs itself.

This lazy function would also solve similar problems that arise if you try to use an executable to generate sources, before building that executable.

I thought of using subprojects instead of subdirs, but that would result in sandbox violations if some libraries share source files with other libraries.

[mensinda]

Ok, this proposal could work, however, there are some problems.

my_array.lazy( libDvar = library('mylibD', 'fileD.cpp', link_with : [libEvar])  )

If the AST parsing to find all undefined variable references, is too complicated one could also write

Yes, this is way too complicated, since we would have to do a special case in the parser (not even the interpreter) for one object + method call combination…

Secondly, I don’t know the official 100% correct stance regarding eval(), however, I am 99.9% certain that any form of eval() will never be accepted.

What could work is a new object type for defining dependency graphs for user-defined objects (maybe even as a new experimental module). Essentially something like this could work:

dag = import('graph')


// Syntax:
// dag.insert('<provides>', depends: ['<optional>', '<list>', '<of>', '<deps>'], data: <opaque object>}
dag.insert('mylibA', depends: ['mylibB', 'mylibC', 'mylibE'], data: {'src': ['fileA.cpp']})
dag.insert('mylibB', depends: ['mylibD'],                     data: {'src': ['fileB.cpp']})
dag.insert('mylibC', depends: ['mylibD', 'mylibE'],           data: {'src': ['fileC.cpp']})
dag.insert('mylibD', depends: ['mylibE'],                     data: {'src': ['fileD.cpp']})
dag.insert('mylibE', data: {'src': ['fileE.cpp']})

dag.insert('mylibX', depends: ['mylibY'], data: {'src': ['fileX.cpp']})
dag.insert('mylibY', depends: ['mylibZ'], data: {'src': ['fileY.cpp']})
dag.insert('mylibZ', depends: ['mylibX'], data: {'src': ['fileZ.cpp']}) // Generates a cycle error

foreach i : dag.as_list()
  // Your code that just adds the libs
  // Guaranteed to have all dependencies resolved
endforeach

Also, thanks for telling me about graphlib, however, we only recently bumped our min. python version requirement to 3.6, so requiring 3.9 features is a few years away 😃

[Volker-Weissmann]

(I am honestly not trying to stir up old, long-dormant debates or make any sort of point here at all. It’s just that, at the end of this whole saga we — meaning, anyone reading this for posterity — were left with a bit of a cliffhanger, and I’m genuinely curious to hear how it ends.)

So, @Volker-Weissmann What was the outcome of your followup testing (if any)? Did you ever work out whether your builds truly required a DAG to properly define? Did you adopt one of the other approaches discussed here?

Sorry for the delay, sickness + overtime delayed me. In about a month or so, I will link a nicely written article about my solution. Hint: It involves 700 lines of graph theory rust.