[website, contributing] Add documentation explaining "easy" local development flow for extending and modifying lore

How do you setup a local dev environment to work on lore’s itself?

What kind of process is necessary for submitting pull requests?

I wanted to experiment with some ideas (injecting reducers in a way similar to how middleware is injected) that required modifying lore core. I forked the repo, cloned locally, and then tried to replace the lore packages in a lore generated project with references of the type

//package.json
'lore', 'file:../../lore/packages/lore'

this didn’t work for me so I tried yarn(npm) link and I couldn’t get that to work.

Additionally I notice that lore package.json is designed for a built project with main point to ‘lib’ and not ‘src’ and I tried changing the package.json for all the lore packages to use ‘src’ as main but that didn’t work.

How do you setup a local dev environment for lore itself?

[jchansen]

@timcase great question, and I regret that it never occurred to me to document a “suggested development flow” before. It took a bit of trial and error before I found a flow was acceptable to me, and I’m sorry you experienced some of that “error”. I’ll absolutely take this conversation and convert it into a formal contributing guide.

What follows may be more information than you want/need (it became epicly long), but I’ll error on the side of being thorough, and providing some supplemental information that may be helpful to know. I’ll start with a direct answer how I setup a local dev environment, and then elaborate on why I go that route.

TLDR; My Local Dev Environment / Setup

1. Clone this repo (lore/lore)
2. Run npm install from the root. Lerna will make sure all packages have their dependencies installed and cross-link them.
3. Open the Lore application you want to use as a sandbox/development environment (anything created with the lore new <app name> command)
4. Do NOT modify the package.json file. Instead, open index.js and change the references there. For example, the lore package on my laptop is located at /Users/jchansen/lore/lore/packages/lore. So if I was experimenting with changes to the core engine, I would modify the line var lore = require('lore') to be var lore = require('/Users/jchansen/lore/lore/packages/lore/src'). If you were experimenting with modifications to a hook, the process would be the same.
5. The approach described above will cause webpack to consume and watch the src directory of the target package, which means any changes you make to the code, will cause your application to automatically rebuild and deploy your application to the browser.
6. While this strategy may seem a bit strange (and is absolutely not what I would expect to hear as a recommendation) it’s the only way I know of to achieve that goal. All other approaches I tried either had serious issues, or required that I rebuild a library after each change.

Full Explanation Behind the Setup / Steps in Greater Detail

Fork the Repo / Install Dependencies / Lerna

First step is to fork the repo. Then, you need to run npm install from the root of the project. This installs lerna and babel from the devDependencies. Once npm install is finished, npm executes a postinstall step that causes lerna to “bootstrap” the project.

Lerna Bootstrapping Process

During this process lerna examines each package the /packages directory and installs their dependencies with one key modification: if the dependency is another package in the /packages folder, it creates a symlink to it instead of installing it from npm.

Conceptually, this may seem similar to linking packages using the 'lore': 'file:../packages/lore syntax, but there’s some differences worth calling out.

For whatever reason, npm does not behave consistently across all install strategies. For example, the file:../packages/lore syntax does not execute the same hooks that would be triggered when installing the package from npm, which provides extremely unreliable indicators as to whether the package will work as expected when you publish it (because the install process differs). I tried the file: approach in the pre-lerna days, and it caused a lot of confusion and frustration. There’s an issue somewhere that confirms and documents npm’s inconsistent behavior, but I don’t have the link at the moment.

The other thing about the file:../packages/lore syntax is that it doesn’t maintain a link to the original code. Or at least I think that’s true…it’s been a while since I used it. But from what I remember, it duplicated my package into the project I “linked” it to, which meant changes I made to (say) packages/lore were not reflected in the project I tried to link it to using the file:../packages/lore syntax. It basically copied the code from packages/lore into the node_modules folder of the application.

The result of that meant that any time I changed the code in packages/lore, I then had to re-run npm install inside the application I was building to consume it. Except that because the version hadn’t changed inside the package.json file for packages/lore, npm didn’t re-build the code, because it didn’t think it needed to. So to force it, I had to delete the node_modules/lore folder in the application FIRST, and THEN re-run npm install to get the updated code. Rinse and repeat. It made for a horrible development flow : (

That’s mostly background and context, but the key takeaway is this: lerna provides a genuine symlink between packages, which means you never have to “rebuild” any of the projects in packages. Or at least that’s true under normal circumstances.

Lib and Src directories

Many of the package.json files within the packages folder reference a main entry point of lib/index.js as opposed to src/index.js. This so the packages can be free to use ES6 style code, but still compile down to ES5 (at least until browsers are able to understand import statements natively).

Obviously changing src files will not magically rebuild the lib directory, so in many cases, changing src files requires running npm install inside the package you changed. For example, if I make a modifcation to packages/lore-hook-redux/src, I will need to run npm install inside packages/lore-hook-redux to regenerate the packages/lore-hook-redux/lib directory used by consuming applications.

An alternate approach, which can make for a smoother development flow (though more risky for publishing) is to modify the main entry of the package.json file to point to src/index.js instead of lib/index.js. Of course, you can always do that temporarily, and then change if back before publishing.

Why install babel at the root?

Lore currently has something like 40 packages, many of which require babel for transpiling. Babel is an fairly time intensive library to install, and allowing each packages to require it as a direct dependency created an absurdly long install time, often in excess of 10 minutes.

To solve this, a decision was made to install babel at the root of the project, and then have all the packages reference that installation using relative paths. You can see an example of this in the build script for lore-hook-redux.

At the time of this writing however, I believe lerna has an integrated a solution that can automatically hoist common dependencies into the root directly, which would allow me to achieve the same effect but without the relative paths.

My Development Environment / Flow

I find the easiest way to work on lore is from within a real or example project. For example, when establishing the normalization behavior for the upcoming v0.12 release, I actually created the normalization example project first, and then used it as the platform to write and test all the modifications required for lore to get that behavior working.

The trick is to ignore the package.json file and focus on the index.js file instead.

There is nothing (I’m aware of) that you can do to the package.json that will create a seamless experience between the code in the lore repo and the code in your application, so I gave up trying. So instead, I simply modify the import/require calls in index.js, and circumvent the node_modules folder completely.

To elaborate, let’s take a look at a typical index.js file:

var lore = require('lore');

// Allows you to access your lore app globally
window.lore = lore;

// Summon the app!
lore.summon({
  hooks: {
    // ... 
    connect: require('lore-hook-connect'),
    // ...
  }
});

And let’s say I’m working on a modification to the core engine, to allow users to control how the Root component is created and mounted (another v0.12 feature). The lore package on my laptop is located at /Users/jchansen/lore/lore/packages/lore, and I’ll be making modifications to the src files in that package.

To integrate this modified code in my application, the simplest way is to update the require call for lore to reference that src folder directly, like this:

var lore = require('/Users/jchansen/lore/lore/packages/lore/src');

// Allows you to access your lore app globally
window.lore = lore;

// Summon the app!
lore.summon({
  hooks: {
    // ... 
    connect: require('lore-hook-connect'),
    // ...
  }
});

Once that’s done, Webpack will automatically consume and watch the files in /Users/jchansen/lore/lore/packages/lore/src, and any changes to those files will cause Webpack to automatically rebuild and reload the application.

That behavior is what I want out of a development flow, and that approach is the only way I know how to obtain it.

Overriding Hooks

A quick note about overriding hooks. One approach is certainly to emulate the approach above, and make modifications directly to the respective package in package/lore-hook-*.

But the approach I’ve started following more recently is actually copying the hook into the consuming application and making my modifications there. Then, once I’m sure I like the changes, I copy them back into lore and submit a PR.

One reason for that approach is that it allows the web application to function without a copy of the lore repo, meaning I can publish it and someone else can download it and try it out. Or sometimes I’m just experimenting with an idea, and simply don’t want to commit the changes to a branch in lore, so I just keep the experiment in the application it was designed for.

To illustrate, let’s say I need to override the blueprints in lore-hook-connect (which I did), and I realize the hook doesn’t expose a way to that (which it didn’t). My process currently looks like this:

// create a /hooks directory in the application
mkdir hooks

// copy the lore-hook-connect src directory to /hooks/lore-hook-connect
cp -r node_modules/lore-hook-connect/src hooks/lore-hook-connect

Now that I have a copy of the hook, I update the index.js file to use my projects version of the hook, like this:

var lore = require('lore');

// Allows you to access your lore app globally
window.lore = lore;

// Summon the app!
lore.summon({
  hooks: {
    // ... 
    connect: require('./hooks/lore-hook-connect'),
    // ...
  }
});

And then I modify that code until I’m happy with my changes.

When I implemented normalization, I ended up modifying ~5 hooks in addition to lore’s core along the way. I used this approach for the entire process, and found it pretty enjoyable. You can see proof of that in the beta.1 tag for the normalization example (the link shows you the hooks directory showing the overrides).

Master no longer has the hooks folder, because I removed the custom hooks once I merged the changes through PRs.

[timcase]

Sweet, you just saved me again, so I did what you described and adding src to the end of the package paths in index.js and now I see that my changes to src files are getting hot reloaded by webpack. I’m good to go now with hacking on lore core. This is a pretty good setup, I’m happy with it. It took reading your explanations to get this setup but honestly it’s not that bad once you understand how everything fits together. I like hacking on the hooks locally and then moving them into the source repo later on. I’m also glad I didn’t have to install watchman (there was a notion in my head that webpack should handle it too, but I didn’t offhand know how). Thanks again. Now back to building validation with redux_forms…