How to test lazy Engines

WARNING: Long post ahead with lots of details

Current State of Things

Lazy Engines pose an interesting problem with testing. Since web application test suites generally don’t start a unique browser instance (or even tab instance) for each test, any assets loaded during one test are likely to persist to all subsequent tests. Additionally, since the test modules are defined eagerly, all imports for those test most be present at the beginning of loading the test suite. This has led to the current recommendation that all lazy assets be loaded preemptively in testing environments, using the preloadAssets helper provided by ember-asset-loader.

This approach leaves much to be desired as at no point do you actually test the lazy loading behavior. This is especially important when working with an application with multiple lazy Engines as it is impossible to identify user flow-dependent failures, such as one area accidentally depending on a lazily loaded asset, during your automated testing.

Previously Discussed Solutions

One theoretical approach to fix the above problem is to run each test as part of a unique context, through either multiple browser instances or even iframes. This however is likely to incur a significant performance penalty when used with large test suites as larger applications are likely to have a high initial start up cost. In some test attempts, this has been prohibitively expensive even with distributed/parallelized test suites.

Proposed Solution

Given the above, it seems likely that the best approach lies somewhere between “load all the things upfront” and “reset the world for every test”. To that end, an approach similar to one @rwjblue has brought up seems to make sense. That is, at the start of a test suite, we should capture a list of all loaded modules, then, after each test, we can evict any modules that are not part of that initial list and unload <script> and <link> elements that were introduced.

One potential concern with this approach is that it won’t properly account for modules that alter global modules or leave some residual state. This is a problem, but I do not think it is one for us to try and solve as these issues are likely to surface even with preloading of all assets. We can detect new globals by using QUnit.config.noglobals, but the other class of issues will likely be highly specific to the application under test.

Given the above, I think the “module reset” solution is likely to yield the best results.

cc/ @rwjblue @nathanhammond

[mydea]

Sure, I opened #341 !

[trentmwillis]

@mydea can you please open a new issue for your specific problem?