Architecture surface refactor and new node/ top-level entry point

We’re headlining Fraggle Rock in the Lighthouse 10.0 release, asking folks to try out a new API, so this is the best opportunity to make substantial changes to the Legacy and Fraggle-Rock-preview APIs for understandability and easy of use.

Our current architecture is something like:

┌──────────┬────────────┬───────────┐
│    CLI   │  DEVTOOLS  │    LR     │
├──────────┴────────────┴───────────┤
│               CORE                │
└───────────────────────────────────┘

(feel free to imagine a better diagram 😃

There’s also an implicit client in the top level: node users enter into core/ directly and that exacerbates the main issue, which is how things are divided up. eg. chrome-launching is in cli/, so is an annoying detail to handle yourself if using the node module, while -GA is in core, which can cause issues with bundling since the other clients don’t have a disk to load from or save to. You also end up with having to anticipate how these layers interact making things more complex and requiring support for core to possibly connect to a browser rather than simply having a Page passed in or not.

Proposal

Instead, we make a place for those decisions while simultaneously making cli/ and core/ simpler and the node experience more featureful.

I’ve played with a few refactors, but I’m very partial to @patrickhulce’s suggestion in https://github.com/GoogleChrome/lighthouse/pull/12393#issuecomment-825219129, which is basically make a new top-level node/ directory that handles stuff currently in cli/ that isn’t specific to a CLI, and stuff currently in core/ that (as a general rule of thumb) isn’t needed for the DevTools and LR bundles:

 ┌──────────┐
 │   CLI    │
 ├──────────┼────────────┬───────────┐
 │   NODE   │  DEVTOOLS  │    LR     │
 ├──────────┴────────────┴───────────┤
 │               CORE                │
 └───────────────────────────────────┘

New cli/

• parse cli flags
• custom exit codes where needed

node/

• launch Chrome
• -GA
• save reports to disk
• Sentry(?)
• node entry point (new package.json main)
• public types (#1773)

Some of core/ and node/ would be repeating each other (like a lot of re-exporting what’s in fraggle-rock/api.js, wherever that ends up), but it wouldn’t be that large and it would allow for a well documented and curated main export from lighthouse, free from e.g. the extras that the different clients need out of core/.

API

As a rough starting point (more or less what we have now):

function generateConfig(configJson?: LH.Config.Json, flags?: LH.Flags): Config

function lighthouse(url?: string, flags?: LH.Flags, configJSON?: LH.Config.Json, page?: LH.Puppeteer.Page): Promise<LH.RunnerResult | undefined>

/** @deprecated */
function legacyNavigation(url?: string, flags?: LH.Flags, configJSON?: LH.Config.Json, userConnection?: Connection): Promise<LH.RunnerResult | undefined>

function startFlow(page: LH.Puppeteer.Page, options?: {name?: string, config?: LH.Config.Json, configContext?: LH.Config.FRContext}): Promise<UserFlow>

function navigation(requestor: LH.NavigationRequestor | undefined, options: {page?: LH.Puppeteer.Page, config?: LH.Config.Json, configContext?: LH.Config.FRContext}): Promise<LH.RunnerResult|undefined>

function snapshot(options: {page?: LH.Puppeteer.Page, config?: LH.Config.Json, configContext?: LH.Config.FRContext}): Promise<LH.RunnerResult|undefined>

function startTimespan(options: {page?: LH.Puppeteer.Page, config?: LH.Config.Json, configContext?: LH.Config.FRContext}): Promise<{endTimespan: () => Promise<LH.RunnerResult|undefined>}>

The biggest opportunity for improvement is probably a pass or two rationalizing configJson, flags, and configContext before we foist them on the world 😃

There’s also a lot of legacy re-exports, that I think it would be helpful to revisit. e.g. lighthouse.Audit and lighthouse.Gatherer

• The new base-gatherer has no imports of its own, just types. Maybe we should switch just to type enforcement rather than asking custom gatherers to extend a class that does nothing?
• Similarly, core Audit stuff is just types, can it also switch to just an interface and all helper functions move off of Audit and into a module of audit utilities?

Public types

We’re at the point that putting together public types could be a useful exercise for improving Lighthouse’s API. Through transitive types we currently expose more or less the entire Lighthouse universe through core/index.js, and the best places to cut the type graph seem also to be some of the best places to cut our own dependency graph (e.g. base-gatherer.js and audit.js).

If we get serious about plugins again, we could publish a separate types package, something like @types/lighthouse-plugins/@types/lighthouse-deep-cuts or whatever, allowing deep requires into select parts of lighthouse to be typed for use in plugins.

[paulirish]

We also constructed an export surface for 10.0, but since paul’s computer broke I gotta go off memory:

Thanks! I edited your comment to group things a lil bit too.

auditFlowArtifacts,

This seems exposed only for us. I think we shouldnt expose as public api.

getAuditList,

Looks like this goes back to https://github.com/GoogleChrome/lighthouse/pull/367 Supporting it doesn’t seem hard, but i don’t think this CLI flag is that important… ?

traceCategories,

This goes back to https://github.com/GoogleChrome/lighthouse/pull/453 and https://github.com/GoogleChrome/lighthouse/issues/420 which showed the usecase. Is also probably something that should be available for us, but not public.

export {NetworkRecords} from ‘./computed/network-records.js’;

We exposed this for plugins, too. https://github.com/GoogleChrome/lighthouse/blob/main/docs/plugins.md#using-network-requests

• Do we need generateConfig/generateLegacyConfig? The config objects that we generate seem to be mostly internal structures and users should always provide a config json to Lighthouse apis.

Hmm. It doesn’t feel like we do. I think @benschwarz is doing some advanced programmatic use, like this but… with -G/-A stuff, too? Ben, the 10.0 apis will definitely change things up on you… so let’s sync on that. (But seeing as those configJson is supported at the entrypoint of the FR methods… i suspect generateConfig aint needed in a 10.x world)

[benschwarz]

@adamraine We use generateConfig fairly extensively, I suspect plenty of other folks using Lighthouse downstream also do too. I’d be happy to share our rationale for using it privately to help build out a use case, if it’s helpful 😄