Proposed Keystone Package Architecture

Proposed Keystone Package Architecture

Now that we’re approaching the big 3.0 (or should I say 0.3.0), I think it would be a good idea to revive the Plugin Architecture issue once again. This is just an initial first draft of my ideas on the topic. Please feel free to comment and criticize at will. I don’t presume that my ideas are the best solution for this issue. However, I have already successfully implemented a subset of these in some of my client’s apps and, with their permission, I’m sharing them here.

Before we get to the technical stuff, some folks have suggested to me that I should use terms such as modules or packages, as opposed to plugins. I know its just a matter of semantics, but at some point we should probably agree on what terminology to use. I prefer not to use the term modules to avoid confusion with npm modules. So, for the sake of this discussion, I will refer to them as packages. We can always revisit the semantics issue at a later time.

Package Categories

The way I envision Keystone’s package architecture, practically every Keystone feature will be encapsulated in a package. Packages, will be divided into four (4) major categories:

Category	Description	Examples
System	There will be at least 2 system packages (maybe more): the Keystone engine and the Keystone Admin-UI	Keystone, admin-ui
Core	Will include many of Keystone’s core functionalities	core fields types, basic authentication
Contributed	User contributed packages providing functionality not provided by Keystone core packages	storage, third-party authentication, custom admin-ui themes
Custom	Application specific functionality	anything else we can think of 😃

Not all of these package categories will be available on the initial version. Check out the next section (the 3 Phased Approach) to see how I propose to phase these into the project.

3 Phased Approach

Even if done correctly, this project will be a massive undertaking, and will require contributions and feedback from the entire Keystone community. Given the size and complexity of the project, not to mention the potentially breaking changes it may introduce, we need to carefully plan its implementation.

The following three (3) phased approach, in my opinion, will help us mitigate these issues.

Phase	Description	Impact
I	In this phase the project will basically be an add-on to Keystone. If detected, Keystone will initialize the package architecture, which will in turn discover and load any packages that are present, within a prescribed directory structure. Only contributed and custom/local packages will be available in this phase.	Low
II	Here is when we convert some of Keystone’s functionality into core packages.	Medium
III	Finally, we should convert the Keystone engine and the Admin-UI into system packages.	High

Phase I should be a relatively easy to implement and should not introduce any breaking changes, given that the architecture itself will be an optional feature.

Phase II, should not include any breaking changes either. If designed and implemented correctly, core packages should work seamlessly with existing applications. This phase will, however, force developers to use/understand the package architecture. Hopefully, they would have already done so during Phase I.

Phase III is a whole different ballgame and will definitely break existing apps. Keystone configuration and bootstrapping will likely undergo dramatic changes. We can, of course, mitigate this with proper documentation and dissemination of the breaking changes, but it’s something we definitely need to consider.

Proposed Feature Set

Our package architecture should including (at minimum) the following feature set:

• Ability to publish packages
• Ability to install published packages
• Ability to create local/custom packages
• Package generator (initial scaffolding)
• Package auto-discovery
• Dependency injection
• Third-party library management (added per @cameronjroe’s suggestion)
• Client-side asset aggregation (for css and javascript)
• LESS/SASS/STYLUS support (added Stylus per @morenoh149’s suggestion)
• Custom routes/route controllers
• Custom views
• Custom models
• Package mount path should be configurable (though each package should have a “default”)

Not all of these features will be available on the initial version. We need to come up with a road map specifying a target version for each feature.

Can anyone think of any features I may have missed?

Package Publishing/Installation

Publishing and installing packages is one of the most important features of our package architecture. It is imperative that package publication and installation be as easy as humanly possible. There are a number of viable options we can pursue to accomplish this.

1. Use existing tools (npm/yeoman)
2. Use of Git submodules
3. Create a CLI tool for package management

The npm/yeoman alternative seems like the most logical candidate to me right now. It’s simple and straight forward, and we’re all quite familiar with these tools. That said, as Keystone matures, I would like to see a Keystone CLI tool to handle these and other rapid development functions. A Keystone CLI, though, is way beyond the scope of this document, and is something, the viability of which, we should address at a later time.

Third-Party Library Dependency Management

Thanks to @cameronjroe’s for bringing up this issue.

Third party library dependency installation and deconfliction is a reality of any system with community developed packages. Fortunately, we already have excellent tools available to help us with this task. We can use both npm and bower for this purpose.

Just like with the package publishing/installation subject above, we can tackle this by implementing a well-defined workflow that employs these tools (npm/bower) or we can create our own CLI tool which would take care of this workflow for us.

The choice between a manual workflow versus a custom CLI tools warrants further discussion.

Proposed Directory Structure

The following is what I consider should be the basic directory structure for the project. It is self-explanatory and will make package discovery very simple to code.

packages
--- system/
    --- keystone/
    --- admin-ui/
--- core/
    --- field-types/
    --- admin-ui-theme/ (... or maybe just "theme/")
    --- auth/
--- contributed/
    ...
--- custom/
    ...

Each package, in turn, will have the following proposed structure:

sample-package/
--- public/ (optional, will be served statically by Express)
    --- assets/
        --- css/ (can optionally be aggregated and minimized)
        --- js/ (can optionally be aggregated and minimized)
        --- img/
--- server/ (optional)
    --- fields/ (will contain custom React fields)
    --- models/ (will contain Mongoose models that will automatically be loaded)
    --- routes/ (will contain Express routes that will automatically be loaded)
    --- controllers/ (route route controllers and middleware)
--- index.js (required, will contain package registration, initialization code, and dependency requirements)

This file structure, I believe, is flexible enough to create almost any type of package (with both front-end and backend features), from a simple React field type, to an entire authentication subsystem.

Package Generation

Just like for package publication/installation there are a number of viable alternatives for this task. Since we already have a Yeoman generator for keystone apps, I figured the simplest solution would be to extend it with sub generators for different types of packages. keystone:fieldtype and keystone:theme are two that immediately come to mind, but I’m sure we can think of more.

What ever we use, we need to ensure that, in addition to the basic scaffolding for the package, it also include basic tests to help/encourage package developers with unit-testing.

Package Dependency Injection

When I started to tackle the dependency injection issue in my own projects I started developing my own DI library, when I came upon an npm module called dependable. It’s an open source DI library developed by the folks at i.TV, in which every package takes the form of a container. Containers can be anything (a string, an object, a function, etc.) and can specify any other containers as its dependencies. I found it simple and elegant, and have been using it for several months now without incident.

Here’s an example of how to use dependable.

var dependable = require('dependable');

var containerA = dependable.container();
containerA.register('packageA', function () { // <-- define packageA
    var package = {};

    .... (some initialization code)

    package.doThis = function doThis() {
        ...
        return 'I did this';
    };

    package.doThat = function doThat() {
        ...
        return 'I did that';
    };

    return package;
});

var containerB = dependable.container();
containerB.register('packageB', { 'something': 'Some data ...' }); // <-- define packageB

var containerC = dependable.container();
containerC.register('packageC', function (packageA, packageB) { // <-- inject packageA and packageB
    var package = {};

    package.doSomethingElse = function() {
        console.log(packageA.doThis()); // <-- use injected packageA 
        console.log(packageA.doThat()); 
        console.log(packageB.something); // <-- use injected packageB
    };

    return package;
});

It’s that simple. Those of you who have used Angular and other frameworks with DI will find this very familiar.

I believe dependable is an excellent fit for this project. However, if anyone has any other viable alternatives I am open to suggestions.

Client-Side Asset Aggregation

To deal with package asset aggregation on the client side I’ve been using an npm module called node-assetmanager.

Asset aggregation and minification is definitely a feature we need to have available for this project. I’ve only used node-assetmanager in a single project, so I can’t say for certain it is the ultimate solution for us. While I have yet to encounter any negative issues with node-assetmanager, I have by no means covered the wide number of use cases we will likely face.

So far, this is the best module I’ve found for this, but alternative suggestions are more than welcome. 😃

Keystone Events

The package architecture will not directly handle anything related to Keystone events. However, the Keystone object will be available as an injectable dependency, so package developers can choose to use it to listening for/hook into Keystone events. So long as the Keystone event system is well documented this should not be an issue.

Conclusion

That’s the gist of it! I hope I didn’t miss anything. Like I mentioned earlier, my intention is just to restart the discussion from issue #185, so this is by no means a comprehensive outline of the project.

A package architecture for Keystone will open a world of possibilities. The ease with which the community would be able to contribute new functionality makes this project not just attractive, but a logical next step for Keystone.

@JedWatson, is this more or less a direction in which you want to take Keystone?

Once we agree on a general architecture, the next steps will be to come up with a roadmap and a well-defined API, followed by a prototype. I know we’re not there yet, so let’s discuss!

[wmertens]

Right now this is the longest-ago-updated issue. @JohnnyEstilles is there anything you want to change about the issue given 0.4? Is everything still valid?

What would be the first steps (issues) to get there?

[JedWatson]

@wmertens @LorbusChris +1k

This is absolutely the direction we’re going in. I actually have some thoughts that, down the track, would allow the Admin UI (or other components) to be used independently of the back-end service, which would then open up all sorts of possibilities for alternate databases, architectures, etc.

The move to React is really opening up a lot of possibilities and right now we’re focusing on what’s in front of our noses (rebuild existing functionality, port everything to a REST API w/ SPA Admin UI, rework Lists so custom Field types can be used, and port the UI to use Elemental) but once this is complete we’re going to really open the flood gates on modularity and customisation 😀

I’ll be doing more detailed write-ups as soon as we cross those milestones.