Karma test run fails frequently with 404s for bundled script files

I’m submitting a bug report

• Library Version: aurelia-cli@0.30.1 karma@1.7.0 karma-chrome-launcher@2.2.0 karma-cli@1.0.1 karma-jasmine@1.1.0 karma-typescript-preprocessor@0.3.1

• Operating System: Windows 10

• Node Version: 6.10.1

• NPM Version: 3.10.10

• Browser: Chrome 60

• Language: TypeScript 2.2.2

Current behavior: We are experiencing major stability issues with the Karma web server when running tests using au test --watch.

Sometimes the tests run correctly for a test run, but very often the task fails with a whole slew of 404 warnings for bundled files, such as the following:

28 07 2017 12:22:01.601:INFO [launcher]: Launching browser Chrome with unlimited concurrency
28 07 2017 12:22:01.613:INFO [launcher]: Starting browser Chrome
28 07 2017 12:22:03.266:INFO [Chrome 60.0.3112 (Windows 10 0.0.0)]: Connected on socket VRL8rnxw6pOi57IrAAAA with id 187727
28 07 2017 12:22:03.676:WARN [web-server]: 404: /base/src/app.js
28 07 2017 12:22:03.678:WARN [web-server]: 404: /base/src/services/security/SecurityService.js
28 07 2017 12:22:03.678:WARN [web-server]: 404: /base/src/services/resources/ResourceService.js
28 07 2017 12:22:03.679:WARN [web-server]: 404: /base/src/model/EventNames.js
28 07 2017 12:22:03.679:WARN [web-server]: 404: /base/src/navigation/AuthorizeStep.js
28 07 2017 12:22:03.682:WARN [web-server]: 404: /base/src/model/observers/EntityGraphObserver.js
28 07 2017 12:22:03.683:WARN [web-server]: 404: /base/src/model/common-data/BankAccount.js
28 07 2017 12:22:03.684:WARN [web-server]: 404: /base/src/model/ObjectState.js
28 07 2017 12:22:03.684:WARN [web-server]: 404: /base/src/model/common-data/Address.js
Chrome 60.0.3112 (Windows 10 0.0.0) ERROR
  Uncaught Error: Script error for "src/app"
  http://requirejs.org/docs/errors.html#scripterror
  at wwwroot/scripts/vendor-bundle.js:4389

The exact files that result in 404s varies a lot as well, but they are always coming from app-bundle.js. The tests may run successfully to begin with, but if the underlying app code is touched then the next test run often breaks. Basically, it’s flaky as hell - on some of our developers’ boxes, they can barely get past this issue at all, so our developer workflow is broken in this area.

Here is our Karma configuration (very close to the base CLI file):

"use strict";
const path = require("path");
const project = require("./aurelia_project/aurelia.json");
const tsconfig = require("./tsconfig.json");

let testSrc = [
    { pattern: project.unitTestRunner.source, included: false },
    "test/aurelia-karma.js",
    "node_modules/jasmine-jquery/lib/jasmine-jquery.js",
    { pattern: "test/locales/*.json", watched: true, served: true, included: false },
    { pattern: "wwwroot/img/**/*", watched: false, served: true, included: false }  
];

let output = project.platform.output;
let appSrc = project.build.bundles.map(x => path.join(output, x.name));
let entryIndex = appSrc.indexOf(path.join(output, project.build.loader.configTarget));
let entryBundle = appSrc.splice(entryIndex, 1)[0];
let files = [entryBundle].concat(testSrc).concat(appSrc);

module.exports = function (config) {
    config.set({
        basePath: "",
        frameworks: [project.testFramework.id],
        files: files,
        proxies: {
            "/img/": "/base/wwwroot/img/"
        },
        exclude: [],
        preprocessors: {
            [project.unitTestRunner.source]: [project.transpiler.id]            
        },
        typescriptPreprocessor: {
            typescript: require("typescript"),
            options: tsconfig.compilerOptions
        },
        reporters: ["progress"],
        port: 9876,
        colors: true,
        logLevel: config.LOG_INFO,
        autoWatch: true,
        browsers: ["Chrome"],
        singleRun: false,
        // client.args must be a array of string.
        // Leave 'aurelia-root', project.paths.root in this order so we can find
        // the root of the aurelia project.
        client: {
            args: ["aurelia-root", project.paths.root]
        }
    });
};

We haven’t made any changes to /test/aurelia-karma.js, or to the au test task, they are identical to current CLI versions for 0.30.1.

We really need some help in this area.

Expected/desired behavior: Running tests should be stable for the Karma configuration, and never result in intermittent 404s to files that are bundled and loaded using RequireJS.

[JeroenVinke]

I may have a lead. When I delay execution of the test files it seems to resolve the issue. Try and replace this line in aurelia-karma.js with setTimeout(() => require(allTestFiles, window.__karma__.start), 2000).

If you can confirm that this helps I can look for a more decent solution. Let me know

[JeroenVinke]

Thanks @sam-piper and @WizzApp. I’ve looked at this issue and found that if I use a minimal scss file (like one scss file with body { background-color: red}) I don’t get this issue. It might not be related to SASS, but instead it might have to do with the size of the app-bundle. Weird thing is that when I refresh Karma in the browser the tests run again. I haven’t been able to get to the bottom of it yet, hopefully I can tell more in the next couple of days