Improve build times

Here’s the timeline of a recent build, which took in total 1h30:

This points to two low hanging fruits which can reduce the cascading effect:

• We could start the macOSUniversal as soon as macOS is done building, right before it starts running tests
• We could sign in parallel to running tests

I am confident that we can get a build down to 1h if we address these two points. Creating an action item list:

• Split macOS into macOS and macOSTests
• Create a long-running Sign job which, just like Publish, uses DevOps API to fetch unsigned assets and produce signed assets in parallel

---

How to create chart

const AdmZip = require('adm-zip');
const open = require('open');

function main(zipPath) {
  const zip = new AdmZip(zipPath);
  const entries = zip.getEntries()
    .filter(e => !e.isDirectory)
    .filter(e => !/\//.test(e.entryName) && /\.txt$/i.test(e.entryName))

  const tasks = [];

  for (const entry of entries) {
    const startRx = /^(?<timestamp>\S+) ##\[section\]Starting: (?<task>\w+)\s+\S+ ##\[section\]Starting: Initialize job$/mgi;
    const finishRx = /^\S+ ##\[section\]Finishing: Finalize Job\s+(?<timestamp>\S+) ##\[section\]Finishing: (?<task>\w+)$/mgi;

    const contents = entry.getData().toString('utf-8');
    let match = startRx.exec(contents);

    if (!match) {
      continue;
    }

    const start = Date.parse(match.groups['timestamp'].trim());
    const task = match.groups['task'].trim();


    match = finishRx.exec(contents);

    if (!match || match.groups['task'].trim() !== task) {
      continue;
    }

    const end = Date.parse(match.groups['timestamp'].trim());

    tasks.push({ task, start, end });
  }

  const min = Math.min(...tasks.map(t => t.start));

  for (const task of tasks) {
    task.start -= min;
    task.end -= min;
  }

  tasks.sort((a, b) => a.start - b.start);

  const max = Math.max(...tasks.map(t => t.end));
  const scale = 800 / max;

  const svg = `<svg width="840" height="500" xmlns="http://www.w3.org/2000/svg">
    <pattern id="grid" width="100" height="100" patternUnits="userSpaceOnUse">
      <path d="M 100 0 L 0 0 0 100" fill="none" stroke="gray" stroke-width="1"/>
    </pattern>
    <rect width="100%" height="100%" fill="url(#grid)"/>
    ${tasks.map((t, i) => `
      <rect x="${20 + t.start * scale}" y="${20 + i * 20}" width="${(t.end - t.start) * scale}" height="16" style="fill:rgb(0,0,255)" />
      <text x="${22 + t.start * scale}" y="${32 + i * 20}" font-family="Verdana" font-size="12" fill="white">${t.task}</text>
    `).join('')}
  </svg>`;

  const uri = `data:image/svg+xml,${encodeURIComponent(svg)}`;
  open(uri, { app: { name: open.apps.edge } });
}

main(process.argv[2])

Place this in a main.js file, and run npm i adm-zip open in the same folder. Then, download the logs ZIP from a build and point the script to it:

$ node main.js LOGS.ZIP

[joaomoreno]

In a 1-1, @TylerLeonhardt just had a brilliant realisation. Splitting build and test apart is costly because test needs pretty much the same amount of overhead as build. The bulk of build is actually the overhead, as built itself takes like 3 mins out of a total of 22 mins. So extracting test would take a significant effort just to shave off 3 minutes altogether. It was at this point that I saw a 💡 above @TylerLeonhardt’s head and he said well, let’s just clone the macOS job into a macOSTest job… the former doesn’t test and the latter doesn’t publish. In essence, we now have two parallel jobs which both do exactly the same overhead but after that initial part, one builds and publishes and the other tests. The build one will finish early, starting all other jobs which depend on it. This way we can shave off 19 minutes, with minimal effort. This is happening in https://github.com/microsoft/vscode/tree/joao/build-parallel-darwin.

[joaomoreno]

We’ve talked about that in the past, and each has attempted that once, so no promises there. I want to focus on the thing that is causing the waterfall for now: macOS.