Lazy Loading Engines Attack Plan!

Come one, come all! It’s time to begin the process of extending ember-engines to support lazy loading! This document should describe every single step to get us from beginning to end, provide points of contact, and help you identify places where you can help us deliver it sooner. This top-post will be continuously updated.

Background

As ember-engines exist right now you should consider them to be an isolation primitive. They allow you to provide guarantees as to the boundaries of your engine in its interaction with the host application and no more. This provides approximately zero benefit at runtime. However, this isolation primitive was built with the goal of being able to leverage that runtime isolation guarantee at build time to do clever things and allow us to asynchronously load engines. And if you’re interested in that, you’re in the right place!

Effort

Most of these changes will be made inside of the ember-engines addon. Our goal is to land most of the functionality there first to iron out how it is going to work. As we identify pieces that we can migrate into Ember and Ember CLI we will do so behind feature flags.

Participating

We want you to be able to dive in on any of this and are continuously updating this document with details on how to do particular pieces. Please feel free to claim portions of the effort by leaving a comment on this issue!

This effort is being organized by the Engine Lazy Loading Strike Team: Dan Gebhardt (@dgeb), Miguel Madero (@miguelmadero), Nathan Hammond (@nathanhammond), Robert Jackson (@rwjblue), and Trent Willis (@trentmwillis). All of us will be able to review and provide guidance across most tasks. If you have detailed questions in specific areas try to track down the people that we’ve assigned to be responsible for particular sections first.

Issue Changelog

• 2016-07-13 10:11AM PDT - Upstream to Ember Core section added to track relevant issues to land prior to the 2.8 beta cutover.
• 2016-07-12 9:10PM PDT - Added clarity about the loader.js approach.
• No edits yet required! Substantive changes will appear here.

---

Build - @nathanhammond, @rwjblue

Ember CLI is responsible for doing all sorts of things at build time. This is area is concerned with what gets spit out after running ember build in an Ember application which includes lazy-loading engines. Note: engines may not have circular dependencies.

Goals

• Stick to familiar patterns from Ember.
• Enable caching of engine assets.
• Enable engines to be (mostly) separate deployable bundles.
  • The host application will still have to be the entry point for an engine rebuild.
  • The host application’s index.html will contain the asset manifests for all engines using a meta config module.
  • The host application’s vendor.js will be updated if the engine’s routes.js file changes.

Approach

All occurrences of {engine} below are the name of the engine according to the name property in the module export of the index.js file inside of the engine.

Asset Manifest

• Constraint: The engine must enumerate its assets in order for them to be loadable after fingerprinting.
• Input: Broccoli tree after broccoli-asset-rev.
• Output: Asset manifest. See below section.
• Delivery: Inserted into a meta tag config inside of the host application’s index.html

Router

• Constraint: {engine}/routes.js must be present at boot of the host application as well as anything it imports.
• Input: {engine}/routes.js
• Output: AMD modules in the engine namespace: {engine}/routes.
• Delivery: Bundled into the host application’s vendor.js. Presence inside of the host application’s vendor.js is considered undefined behavior and may change in the future.

Engine Code

• Constraint: Must be individually addressable by the asset loading service.
• Input: {engine}/addon/*
• Output: dist/assets/{engine}.(js|css)
• Delivery: Asset loading service when attempting to route to engine.

Engine Vendor

• Constraint: Must be individually addressable by the asset loading service.
• Input: {engine}/vendor/*
• Output: dist/assets/{engine}-vendor.(js|css)
• Delivery: Asset loading service when attempting to route to engine.

Engine Static Assets

• Constraint: Must follow current patterns of how addons place public assets.
• Input: {engine}/public/*
• Output: dist/{engine}/*
• Delivery: Assets in public have no default behavior and must be manually loaded by the engine code.

Tasks

• Make the ember-engines EngineAddon object handle the production of the correct assets. [PR] [Owner: @nathanhammond]
• Extract this section into documentation for the EnginesAddon build process.

---

Asset Manifest - @trentmwillis

It is possible that engines can be included into an application multiple times at different mount points. We should support the ability to do so. This is an easy area to participate in.

Goals

• Allow engine assets to be fingerprinted for caching reasons.
• Support multiple inclusion of same engine without duplicate backing engine load.

Approach

This is non-normative and provided as a sketch for the beginning of an implementation. It assuredly has gaps. We probably want to have individual named manifests for each engine. We will need a way to map route path microsyntax to the engine’s moduleName which can come as a separate meta config item.

Sketch global engine config definition appearing at something like config/engines:

{
  engines: {
    'name-specified-by-mount': 'engine-name-from-package-json',
    'other-engine-mount': 'neat-engine',
    'duplicate-engine-mount': 'neat-engine',
    'other-engine-mount.nested-engine': 'foo-engine'
  }
}

Sketch engine manifest appearing at something like config/engines/neat-engine:

{
  'neat-engine': {
    app: 'neat-engine-2b00042f7481c7b056c4b410d28f33cf.js',
    appStyles: '...',
    vendor: '...',
    vendorStyles: '...'
  }
}

Tasks

• Specify the asset manifest in more detail. [Owner: @trentmwillis]
  • Land the associated RFC.
• Generate the manifest from build output. [PR] [Owner: @trentmwillis]
• Land multi meta module PR on Ember CLI.
• Insert the manifest into index.html as a meta tag from the ember-asset-loader addon using contentFor. [PR] [Owner: @trentmwillis]

---

Asset Loading Service - @trentmwillis

This is a service that should be implemented which receives an Asset Manifest and returns a promise which resolves once all of the assets from that manifest have finished loading. This is an easy area to participate in.

Goals

• Simple API.
• Prevent duplicate requests for the same engine.
• Supports whatever the Asset Manifest is from above.
• Identify engines by route path microsyntax from the host application root.

Approach

This is non-normative and provided as a sketch for the beginning of an implementation. It assuredly has gaps.

{
  lookup: {},
  load(bundle) {

    var loaded = new Promise(function(resolve, reject) {
      // Load the assets.
    });

    this.lookup[bundle.name] = loaded;

    loaded.then((response) => {
      this._populateRegistry(response);
      this.lookup[bundle.name] = true;
    })

    // Used to block the 
    return loaded;
  },
  _populateRegistry() {
    // TODO: Populate the loader.js aliases.
  }
}

Tasks

• Design the service’s public API. [Owner: @trentmwillis]
  • Land the associated RFC.
• Implement the service. [Addon] [Owner: @trentmwillis]
• Register this service during the boot of Ember itself through an instance-initializer. [[PR]((https://github.com/trentmwillis/ember-asset-loader/pull/10)] [Owner: @trentmwillis]
• loader.js cache miss should probably be reset on each engine load.

The following may not work, but should be explored as an isolation guarantee:

• After script and style tags mount the engine modules which are registered will be duplicated inside of loader.js alias for the modules at the engine’s mount point.
• The asset loading service rewrites the loading definition of all modules by getting a reference to the function and the module arguments and doing a new define for each of them.
• Duplicate engines will identify that modules are already loaded for neat-engine and will simply insert a duplicate module inside of loader.js at the appropriate mount point.

---

Runtime Routing - @trentmwillis, @nathanhammond

Support for async loading of route handlers. The route files themselves won’t be present at time of transition, so routing will have different outcomes dependent upon host application or crossing engine boundaries.

Goals

• Minimize changes to existing routing behavior.
• Pause transition at engine boundaries while loading assets from the asset service.

Approach

This has been pretty-well-distilled into tasks in that there is little that still needs design work.

Tasks

• Optional - Deprecate default query params inside of Ember.
• Document route-serializer usage in ember-engines and how it differs from behavior outside of engines. [PR] [Owner: @trentmwillis]
• Document inaccessibleByURL behavior in ember-engines. [Owner: @trentmwillis]
• Document change in loading behavior for failed transitions when crossing an engine boundary. [PR] [Owner: @trentmwillis]
• Support simultaneous loading of engines if you cross multiple engine boundaries. Works by virtue of getHandler being called synchronously. No additional work needed. [Owner: @trentmwillis]
• Land async getHandler in Router.js and merge into Ember.js. [Owner: @trentmwillis]
• Address loading substates for this.mount. [PR] [Owner: @trentmwillis].

---

Community Breakage - @miguelmadero

It’s possible that some of these changes will break tooling that exists in the Ember community, such as the Ember Inspector. It is hard to anticipate what these may be. Plan to review behaviors in Ember Inspector and LOG_RESOLVER.

• Identify any issues with early resolution of routes in the ember-inspector.
• Identify any issues with LOG_RESOLVER eager resolution.

---

Routeless Lazy Engines - @dgeb, @rwjblue

Routeless lazy engines are engines which are mounted using the {{mount}} helper. They will be discovered at build time while walking the dependency tree and packaged identically to above. They can be thought of as “lazy components” which also bundle all of their own dependencies.

Goals

• Route transition will become dependent upon template content?.
• Support async loading at the moment the mount keyword is encountered in the template?

Approach

How this will function is poorly defined as template rendering is currently a synchronous behavior. The “easiest” solution likely requires us to know of {{mount}} usage in a template during the transition lifecycle.

Tasks

• ???

---

Upstream to Ember Core

• Partial rollback of route-serializers RFC.
• Upstream code into Ember prior to 2.8 beta.
  • Update ember-resolver addon to use resolveEngine. [PR]
  • Move mount keyword upstream to ember.js

---

Non-Goals

These are things we’re not aiming to address at this time.

• CSS is order dependent and pollutes global scope. Write your CSS inside of your engines very carefully.
• Arbitrary segmentation of applications. We’ll circle back to this just before landing to make sure we didn’t paint ourselves into a corner.

[mike183]

Thanks @nathanhammond, I’ve messaged you on slack to avoid creating too much noise for everyone else.

@peStoney - I think you had a similar use case to mine, you may also find the previous comment useful.

[nathanhammond]

Strike Team Meeting Notes: 2016-07-21

The strike team got together and had a conversation about the next steps and how we’re progressing. These are the notes from that conversation. No timeline estimates yet, but the scope of work is better and better understood. Please review the new RFCs and give us feedback!

Asset Manifest RFC

• Link
• Serialize to assetmanifest.json every time. No reason not to have it.
  • Aside: primary config into separate JSON file using same technique.
• Last step to optionally insert into index.html
• Move default out of the environment config.
  • Overrides only go there.
  • May apply to non-engine loading things.

Asset Loading Service RFC

• Link
• Multiple manifests should definitely be supported.
• Open-ended, experiment in some separate addon.
  • Nest it into the ember-engines addon?
• May not land in Ember itself, possibly default Ember CLI blueprint.

Build Time

• Would return the addon bundle.
• Would need to fix the merge of all addons down together into an addon tree.
  • Need to change that behavior.