Support Clusters for Internal Structure?

I don’t know if this is the same as #19 but I thought it might be useful to explain my use case.

Given a large codebase, some 350k LOC, (dis)organized into piles of packages, sometimes preventing a graph from being generated, since the dot file is too complex, I guess? I ended up trying to use max-bacon=1 and tried excluding some paths in the codebase, however that ends up removing a significant amount of nodes imported into the excluded files.

I saw the new cluster feature, and wondered if this might be a viable option to use clustering for internal packages as well - so that I can “collapse” significant chunks of the codebase if there are more than N nodes inside that package.

Does that make sense?

[thebjorn]

If I’m misusing the term,

No, you’re using it 100% correctly.

Pydeps started as a tool for helping us move from a repo/package containing all our code, to many smaller repos/packages (it’s an improvement, but brings its own set of challenges). I don’t think there is any easy solution to representing relationships in a 300+ Kloc code base in 2d., but manually reducing the graph can still give you useful insights…

I usually follow a workflow that looks something like this (using django-cms - ~50KLOC) as an example: pydeps cms --cluster (had to scale it down due to size):

then I look at the top and the bottom of the graph and remove nodes with high in/out degrees (since they’re significantly constraining the graph display algorithm that dot uses), e.g. a first pass gives: pydeps cms -Tpng --show-dot --cluster -x django* sekizai* south* cms.test*

…and repeating until I come to a graph that’s readable and interesting: pydeps cms --show-dot --cluster -x django* sekizai* south* cms.test* cms.exceptions* cms.constants* cms.publisher.query* cms.utils.conf* cms.utils.i18n* cms.admin.pageadmin* cms.api* -xx cms.admin

We were having increasing issues with developers unintentionally introducing circular imports, so we look for arrows going the “wrong” way (i.e. up). During the extract-to-individual projects phase we’d use nodes that clustered together in the graph as an indication that they should probably move as a unit.

[NeilGirdhar]

This would be really useful for me. In order to prevent circular imports, and to keep a logical program flow, it’s nice to ensure that internal packages import each other as a DAG. This is despite only requiring module imports to form a DAG. So, I’d like a way to collect modules into their respective packages and to make a graph of their dependencies.

@thebjorn Your last graph looks really nice and close to what I’d like to have, I think.