💬 RFC: Task runner contract (pluggable task runners)

Following up on #1826 to make a more generic issue to foster further discussion.

Specifically this comment: https://github.com/projen/projen/issues/1826#issuecomment-1211720662

We need to make sure the contract for task runners is right. Then we can implement a default choice for the batteries-included projects based on that.

---

There are several issues at the moment with projen’s task implementation:

• Extra params cannot be included (E.g. projen test -- my.spec.ts). This is because the projen task can include many forked tasks and thereby it is not possible to infer which sub-process needs these extra parameters.
• It sets up it’s own opinion and hirerachy of tasks calling (e.g. see build task). Which makes it difficult to set up other task runners and optimize the behaviour. For example, turborepo can also run task hierarchies, but in an optimized way (using cache).

There are many competing task runners emerging. Notable ones:

• Turborepo
• Wireit: #1826
• Lage

---

As a user, I want to be able to swap a task runner for a custom one. Ideally, it would be another component or package that can easily plug into the projen project instance.

Should be as simple as:

new typescript.TypeScriptProject({
  // ... projen options

  taskRunner: new CustomTaskRunner({
    // optional task runner options
  }),
})

Where CustomTaskRunner implements an interface:

class CustomTaskRunner implements IProjenTaskRunner {
  // ...
}

---

Question: What should this interface look like?

interface IProjenTaskRunner {
  // ❓❓❓
}

[mrgrain]

I’ve been looking at this the last couple days. Here are my (unsorted) thoughts:

• Obviously #356 , but I have a fix for this open #2246
• It’s TaskRuntime that we need to swap out. The TaskModel will likely stay in projen, otherwise cross runtime compatibility will be a mess.
• Runtimes might extend the TaskModel with extra features. Using any of them will make a project less portable. That’s okay as a user, probably not a good idea as a provider of a project
• We should have a separate package projen-run or pjx that contains the basic default task runtime. Advantage: Projen can be ejected from a project and tasks still work.
• Replace spawn with a dependency model (dependsOn). A simple runtime can execute tasks in order, a more complex one can parallelize and cache.
• We will likely need a definition for input and output files, otherwise any caching attempts (wireit, nx) will be futile. However low priority.
• Tasks should be able to define more complex (positional) arguments and options
• DependsOn should be able to define calling a dependency with specific options, including passing options through from the parent. test-task --file {testFile} --dependsOn--> jest-task {file}
• The argument parsing might have to stay with projen in the first iteration if we want the projen test shortcuts. I’m not also not sure if it should be delegate to the task runtime at all, or maybe that could be something swap-able as well (but what’s the point).
• Builtin tasks need to be more flexible. Especially the should be parameterized without using env vars. Currently they use a silly workaround to define env vars in the task that are then read from the builtin.
• Also 3rd party “builtin” tasks should be a thing
• “Hidden” tasks that fully exists in the task model, but are not added to package.json and not available via projen hidden-task. TaskRuntimes might decide to still expose them to the user directly e.g. wireit hidden-task. Advantage: Could be used so that EslintTask would exists, even though it’s only ever called from projen test. Or to hide “CI” variations of a task from the user. E.g. test:ci would be somehow called in the CI but cannot be used by the user.

[moltar]

Fair enough!

I think having the option to intercept args is a good one.

That way you can pass thru args, if you want, or you could amend them to your liking, for example, and then pass to the underlying tasks.

E.g. maybe, passing -u (to jest), but then some other testing framework has a different flag for the same concept (updating snapshots), which you could then overload.