[Question] Is there way to define testing timeout for after running a webserver?

Hi, thanks for this great project in advance. I’m curious is there any way to define a timeout for after running a webserver. I found that a test running time has been included in a webserver build time so depending on build environments, it seems that a test could have a small and large time difference.

Let’s assume a timeout of the config is 50sec, and the test time takes 45sec. If the build time is less than 5sec on my local, the test would be passed. However, if an env that is much slower than my local(exceed build time more than 5sec) would be not.

I know the webServer.timeout is also useful but if I must run testing without timeout or any constraint of a webserver then it seems not proper one. To ensure testing time regardless of fluctuating build time, how about adding like timeoutAfterRunningWebServer for this? (the name could be weird but just for reference)

(Just for my quick workaround is to set a long timeout as possible but not sure that is the best solution… By any chance, is there good way than mine, or I’ve missed any info regarding this from the official doc?)

// Build time 

// test.spec.js
test('It takes 45sec for testing', ({ page }) => {
  await page.waitForTimeout(45000);
});

// config.js
const config: PlaywrightTestConfig = {
  webServer: {
    command: 'npm run start',
    port: 3000,
    reuseExistingServer: !process.env.CI,
  },
  timeout: 50000
};

// Proposal - config.js
const config: PlaywrightTestConfig = {
  webServer: {
    command: 'npm run start',
    port: 3000,
    reuseExistingServer: !process.env.CI,
  },
  timeoutAfterRunningWebServer: 50000, // Not include build time
};

[rwoll]

I’ve found it seems the starting time of a webServer doesn’t impact to the test-level timeout

If you have:

import type { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  timeout: 1 * 1000,
  webServer: {
    command: "bash -c 'sleep 5 && python3 -m http.server 8888'",
    port: 8888,
  },
};

export default config;

your tests will still run. The test-level timeout is 1 second, and although it takes 5 seconds for the port to be available, we don’t count that port check against the individual test timeout, so the test still has 1 second to run.

As the results, I’m not sure which one is the best…is it valid to use the url option still instead of the port?

If you’re dealing with a web server that does HTTP things (like webpack-dev-server), I recommend the URL option over port. Many dev servers will open a port, before actually being fully setup and serving HTTP requests, so using a URL option in this case is safer, otherwise, since the port is open, PW will move on and try your tests (and possibly timeout in there).

---

Here’s an example that illustrates this subtle distinction:

Consider this simple server that:

1. immediately starts listening on port 8888
2. only serves 404s while it boots up for the first 5 seconds.
3. after booting up, it will start serving 200s and your actual application.

const http = require('http');

// Create an empty server without any routes.
let dynamicRouteHandler = (req, res) => { res.statusCode = 404; res.end(); };
const server = http.createServer((req, res) => { dynamicRouteHandler(req, res) });

// Start listening on port 8888.
server.listen(8888, () => {
    console.log(`Server is running on http://localhost:8888`);
});

// Simulate taking 5 seconds to install an actual route handler to serve our HTML!
console.log('installing routes…');
setTimeout(() => {
    dynamicRouteHandler = (req, res) => {
        res.write(`<p>hello</p>`);
        res.end();
    };
    console.log('Routes installed!');
},5_000);

Your test looks like:

import test, { expect } from "@playwright/test";

test('works', async ({ page }) => {
  console.log('running actual test case');
  await expect(page.locator('p')).toHaveText('hello');
});

If your config uses port:

import type { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  timeout: 1000, // 1 second
  webServer: {
    command: "node ./server.js",
    port: 8888,
  },
};

export default config;

your test will fail, because the port check immediately passed, and Playwright went to run your test while we were in phase (2).

To fix this, you can specify url, so we wait until the server is serving the actual content before running the test:

import type { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  timeout: 1000, // 1 second
  webServer: {
    command: "node ./server.js",
    url: 'http://localhost:8888/',
  },
};

export default config;

Closing this issue since it sounds like you got things working, but feel free to open a new one if you need additional help!

Thanks! 🎉

[togo26]

If you’re dealing with a web server that does HTTP things (like webpack-dev-server), I recommend the URL option over port. Many dev servers will open a port, before actually being fully setup and serving HTTP requests, so using a URL option in this case is safer, otherwise, since the port is open, PW will move on and try your tests (and possibly timeout in there).

@rwoll Thanks for the great explanation and the example!👍