Removing implicit dependencies and generating package.json for each project

After using NX, I have a few suggestions for medium-large projects adopting NX. Collectively, they will solve the following issues: #1169, #1777, #602 and probably some others as well.

Affected command and implicit dependencies on workspace.json, nx.json, and package.json

From #1169:

If we introduce a new library every time that we add a new component, our implicit dependencies (angular.json, nx.json, and tsconfig.json) will be updated every time, and therefore all apps are considered affected.

This is a real problem that is affecting medium to large projects. One of the primary motivations to moving to a mono-repo (and hence move to using nx) is to figure out what we need to build and deploy.

I agree with having separate libs (as opposed to doing file-level dependency analysis to figure out what to build), but ultimately any changes to the more freqently changing files – specifically the angular.json/workspace.json, nx.json, and package.json – reduces the utility of the “affected” commands.

angular/workspace.json and nx.json

The primary issue here is adding new projects. But sometimes updating existing projects also leads to the same issue.

Proposed solution

1. Have a project.json in every app/lib folder, and this project.json would contain just the parts from angular.json/workspace.json + nx.json that are relevant to that project.

2. An explicit command called nx update-projects (which is also implicity run before any nx command), generate the root angular.json/workspace.json file + nx.json file from all the projects involved (recursive find of all project.json in the apps and libs folders **)

3. Have a top-level property called “generated” in angular.json/workspace.json and nx.json, which would indicate if those files are generated, and subsequently ignore them if they are during affected calculations.

---

** To know the names of all folders that contain apps and libs (even though today in some places in the code it is hardcoded), we can look at the root package.json for an “nx” property (e.g. nx: {appsFolder: "apps", libsFolder: "libs"}).

---

Root package.json

The primary issue here is adding a new external dependency to be used by one of the projects. We now have to build and deploy everything because affected command says so.

Proposed solution

The proposed solution below actually solves two problems: (1) affected command only shows those projects that are affected by the root package.json, (2) a package.json is generated in the build output of all projects, which helps further Dockerization / deployment (See #1777).

1. During the scan of source code to build a dependency graph for the affected command, nx only checks for imports/requires of mono-repo’s npmScope. Instead, have it maintain a dependency graph of all imports, including external dependencies.

2. Maintain a subset of dependencies/devDependencies/peer/optional per project in the project’s own nxdeps.json file (just like the cache in the output folder). This file would be updated after a call to nx update-projects. The purpose of this nxdeps.json file will be clear in (3b-ii) below.

3. Now, the affected command can be divided into two steps:

   3a) Like before, using the base and head (or affectedFiles), figure out which files were changed in which projects, and then check the dependency graph for all inter-project dependencies (i.e. any that starts with npmScope in the dependency graph).

   3b-i) If base and head were supplied, look at the two versions of package.json and check which dependencies in the dependencies/devDependencies/peer/optional have changed. Compare these with all non-project dependencies in the dependency graph.

   3b-ii) If affectedFiles was supplied instead, then lookup the current values of each non-project import from the dependency graph in the root package.json file, and compare these values (additions, removals, changes in version) with those in nxdeps.json for each project. ***

4. For the build output, generate a package.json for each project using:

   4a-i) Any package.json in the project-root of each project as the base (if it exists – some users already have it).

   4a-ii) If package.json does not exists in each project, then generate one using all the fields in the pacakge.json in the root of the monorepo … but do not use the dependencies/devDep/peer/optional. Use the project name as the name field.

   4b) Now using the dependency graph, as well as the externalDependencies configuration option, figure out which dependencies are really external, look up their version value in the root package.json.

   4c) Use (4a) and (4b) above to create a package.json in the build output folder.

   4d) Also, copy over the yarn.lock and package-lock.json in the root to the output folder if they exist.

---

*** If no nxdeps.json was supplied, but affectedFiles was used, give a warning to the user.

---

Update existing nx.json and angular/workspace.json

Finally, remove the following files from nx.json’s implicit dependencies: package.json, workspace.json/angular.json, nx.json.

---

Importing publishable libraries into others

The generation of package.json for each project using the above techniques presents an opportunity to hoist individual projects and libraries to publish them to npm/registry. That is, imagine if you could also specify externalLibraries in addition to externalDependencies (or simply are able to detect which libraries are publishable). Then you exclude bundling these libraries during a build, and instead keep their @npmScope/library-name imports intact in the output (i.e. whitelist these imports in nodeExternals).

[dzcpy]

Switched to lerna at last. nx brings more trouble than convinience in my projects…

[jorgekleeen]

I know is not directly related to this, but something I find difficult right now is to know after a while which dependencies in monorepo’s package.json are being used in each specific project. What is the recommended way to do make this kind of analysis?

I find this information relevant when you want to remove an old legacy library, and then you want to know which dependencies were specific to that project, so you can get rid of those dependencies.