Filip/playwright
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

docs(test runner): configuration docs (#6801)

Browse source
This commit is contained in:
Dmitry Gozman 2021-05-30 15:14:44 -07:00 committed by GitHub
parent f7e720568d
commit 709a4cbe89
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 346 additions and 196 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
docs/src/test-advanced.md
View file

@ -7,6 +7,49 @@ title: "Advanced Configuration"
<br/>
## Project configuration
- `metadata: any` - Any JSON-serializable metadata that will be put directly to the test report.
- `name: string` - Project name, useful when defining multiple [test projects](#projects).
- `outputDir: string` - Output directory for files created during the test run.
- `repeatEach: number` - The number of times to repeat each test, useful for debugging flaky tests. Overridden by `--repeat-each` command line option.
- `retries: number` - The maximum number of retry attempts given to failed tests. Overridden by `--retries` command line option.
- `screenshot: 'off' | 'on' | 'only-on-failure'` - Whether to capture a screenshot after each test, off by default.
- `off` - Do not capture screenshots.
- `on` - Capture screenshot after each test.
- `only-on-failure` - Capture screenshot after each test failure.
- `snapshotDir: string` - [Snapshots](#snapshots) directory. Overridden by `--snapshot-dir` command line option.
- `testDir: string` - Directory that will be recursively scanned for test files.
- `testIgnore: string | RegExp | (string | RegExp)[]` - Files matching one of these patterns are not considered test files.
- `testMatch: string | RegExp | (string | RegExp)[]` - Only the files matching one of these patterns are considered test files.
- `timeout: number` - Timeout for each test in milliseconds. Overridden by `--timeout` command line option.
- `video: 'off' | 'on' | 'retain-on-failure' | 'retry-with-video'` - Whether to record video for each test, off by default.
- `off` - Do not record video.
- `on` - Record video for each test.
- `retain-on-failure` - Record video for each test, but remove all videos from successful test runs.
- `retry-with-video` - Record video only when retrying a test.
## Test run options
These options would be typically different between local development and CI operation:
- `forbidOnly: boolean` - Whether to exit with an error if any tests are marked as `test.only`. Useful on CI. Overridden by `--forbid-only` command line option.
- `globalSetup: string` - Path to the global setup file. This file will be required and run before all the tests. It must export a single function.
- `globalTeardown: string` - Path to the global teardown file. This file will be required and run after all the tests. It must export a single function.
- `globalTimeout: number` - Total timeout in milliseconds for the whole test run. Overridden by `--global-timeout` command line option.
- `grep: RegExp | RegExp[]` - Patterns to filter tests based on their title. Overridden by `--grep` command line option.
- `maxFailures: number` - The maximum number of test failures for this test run. After reaching this number, testing will stop and exit with an error. Setting to zero (default) disables this behavior. Overridden by `--max-failures` and `-x` command line options.
- `preserveOutput: 'always' | 'never' | 'failures-only'` - Whether to preserve test output in the `outputDir`:
- `'always'` - preserve output for all tests;
- `'never'` - do not preserve output for any tests;
- `'failures-only'` - only preserve output for failed tests.
- `projects: Project[]` - Multiple [projects](#projects) configuration.
- `reporter: 'list' | 'line' | 'dot' | 'json' | 'junit'` - The reporter to use. See [reporters](#reporters) for details.
- `quiet: boolean` - Whether to suppress stdout and stderr from the tests. Overridden by `--quiet` command line option.
- `shard: { total: number, current: number } | null` - [Shard](#shards) information. Overridden by `--shard` command line option.
- `updateSnapshots: boolean` - Whether to update expected snapshots with the actual results produced by the test run. Overridden by `--update-snapshots` command line option.
- `workers: number` - The maximum number of concurrent worker processes to use for parallelizing tests. Overridden by `--workers` command line option.
## Projects
Playwright Test supports running multiple test projects at the same time. This is useful for running the same tests in multiple configurations. For example, consider running tests against multiple versions of the database.

10
docs/src/test-cli.md
View file

@ -14,6 +14,8 @@ All the options are available in the [configuration file](./test-configuration.m
- `--headed`: Run tests in headed browsers. Useful for debugging.
- `--browser`: Run test in a specific browser. Available options are `"chromium"`, `"firefox"`, `"webkit"` or `"all"` to run tests in all three browsers at the same time.
- `-c <file>` or `--config <file>`: Configuration file. If not passed, defaults to `playwright.config.ts` or `playwright.config.js` in the current directory.
- `-c <dir>` or `--config <dir>`: Directory with the tests to run without configuration file.
@ -22,7 +24,7 @@ All the options are available in the [configuration file](./test-configuration.m
- `-g <grep>` or `--grep <grep>`: Only run tests matching this regular expression. For example, this will run `'should add to cart'` when passed `-g="add to cart"`.
- `--global-timeout <milliseconds>`: Total timeout for the whole test run. By default, there is no global timeout.
- `--global-timeout <number>`: Total timeout for the whole test run in milliseconds. By default, there is no global timeout.
- `--list`: List all the tests, but do not run them.
@ -30,6 +32,8 @@ All the options are available in the [configuration file](./test-configuration.m
- `--output <dir>`: Directory for artifacts produced by tests, defaults to `test-results`.
- `--project <name>`: Only run tests from one of the specified [projects](./test-advanced.md#projects). Defaults to running all projects defined in the configuration file.
- `--quiet`: Whether to suppress stdout and stderr from the tests.
- `--repeat-each <N>`: Run each test `N` times, defaults to one.
@ -40,10 +44,8 @@ All the options are available in the [configuration file](./test-configuration.m
- `--shard <shard>`: [Shard](./test-parallel.md#shards) tests and execute only selected shard, specified in the form `current/all`, 1-based, for example `3/5`.
- `--project <project...>`: Only run tests from one of the specified [projects](./test-advanced.md#projects). Defaults to running all projects defined in the configuration file.
- `--timeout <number>`: Maximum timeout in milliseconds for each test, defaults to 10 seconds.
- `--update-snapshots` or `-u`: Whether to update [snapshots](./test-snapshots.md) with actual results instead of comparing them. Use this when snapshot expectations have changed.
- `--workers <workers>` or `-j <workers>`: The maximum number of concurrent worker processes that run in [parallel](./test-parallel.md).
- `--workers <number>` or `-j <number>`: The maximum number of concurrent worker processes that run in [parallel](./test-parallel.md).

319
docs/src/test-configuration.md
View file

@ -7,124 +7,279 @@ title: "Configuration"
<br/>
## Configuration object
## Configure [Browser], [BrowserContext], videos and screenshots
Configuration file exports a single configuration object.
Playwright Tests supports browser and context options that you typically pass to [`method: BrowserType.launch`] and [`method: Browser.newContext`] methods, for example `headless`, `viewport` or `ignoreHTTPSErrors`. It also provides options to record video for the test or capture screenshot at the end.
You can modify browser launch options, context creation options and testing options either globally in the configuration file, or locally in the test file.
You can specify any options either locally in a test file, or globally in the configuration file.
See the full list of launch options in [`browserType.launch()`](https://playwright.dev/docs/api/class-browsertype#browsertypelaunchoptions) documentation.
- Browser options match [`method: BrowserType.launch`] method.
- Context options match [`method: Browser.newContext`] method.
- `screenshot` option - whether to capture a screenshot after each test, off by default. Screenshot will appear in the test output directory, typically `test-results`.
- `off` - Do not capture screenshots.
- `on` - Capture screenshot after each test.
- `only-on-failure` - Capture screenshot after each test failure.
- `video` options - whether to record video for each test, off by default. Video will appear in the test output directory, typically `test-results`.
- `off` - Do not record video.
- `on` - Record video for each test.
- `retain-on-failure` - Record video for each test, but remove all videos from successful test runs.
- `retry-with-video` - Record video only when retrying a test.
See the full list of context options in [`browser.newContext()`](https://playwright.dev/docs/api/class-browser#browsernewcontextoptions) documentation.
### Global configuration
Create `playwright.config.js` (or `playwright.config.ts`) to configure your tests, and specify options in the `use` section. Playwright Test will automatically pick it up.
```js
// pwtest.config.ts
module.exports = {
use: {
// Browser options
headless: false,
slowMo: 50,
// Context options
viewport: { width: 1280, height: 720 },
ignoreHTTPSErrors: true,
},
};
```
```ts
import { PlaywrightTestConfig } from 'playwright/test';
const config: PlaywrightTestConfig = {
// 20 seconds per test.
timeout: 20000,
use: {
// Browser options
headless: false,
slowMo: 50,
// Forbid test.only on CI.
forbidOnly: !!process.env.CI,
// Two retries for each test.
retries: 2,
});
// Context options
viewport: { width: 1280, height: 720 },
ignoreHTTPSErrors: true,
},
};
export default config;
```
## Global configuration
If you put your configuration file in a different place, pass it with `--config` option.
You can specify different options for each browser using projects in the configuration file. Below is an example that changes some global testing options, and Chromium browser configuration.
```sh
npx playwright test --config=tests/my.config.js
```
### Local configuration
With `test.use()` you can override some options for a file or a `describe` block.
```js
// config.ts
import { PlaywrightTestConfig } from "playwright/test";
// example.spec.js
const { test, expect } = require('playwright/test');
// Run tests in this file with portrait-like viewport.
test.use({ viewport: { width: 600, height: 900 } });
test('my portrait test', async ({ page }) => {
// ...
});
test.describe('headed block', () => {
// Run tests in this describe block in headed mode.
test.use({ headless: false });
test('my headed portrait test', async ({ page }) => {
// ...
});
});
```
```ts
// example.spec.ts
import { test, expect } from 'playwright/test';
// Run tests in this file with portrait-like viewport.
test.use({ viewport: { width: 600, height: 900 } });
test('my portrait test', async ({ page }) => {
// ...
});
test.describe('headed block', () => {
// Run tests in this describe block in headed mode.
test.use({ headless: false });
test('my headed portrait test', async ({ page }) => {
// ...
});
});
```
## Testing options
In addition to configuring [Browser] or [BrowserContext], videos or screenshots, Playwright Test has many options configuring how your tests are run. Below are the most common ones, see [advanced configuration](./test-advanced.md) for the full list.
- `forbidOnly`: Whether to exit with an error if any tests are marked as `test.only`. Useful on CI.
- `globalSetup`: Path to the global setup file. This file will be required and run before all the tests. It must export a single function.
- `globalTeardown`: Path to the global teardown file. This file will be required and run after all the tests. It must export a single function.
- `retries`: The maximum number of retry attempts per test.
- `testDir`: Directory with the test files.
- `testIgnore`: Glob patterns or regular expressions that should be ignored when looking for the test files. For example, `'**/test-assets'`.
- `testMatch`: Glob patterns or regular expressions that match test files. For example, `'**/todo-tests/*.spec.ts'`. By default, Playwright Test runs `.*(test|spec)\.(js|ts|mjs)` files.
- `timeout`: Time in milliseconds given to each test.
- `workers`: The maximum number of concurrent worker processes to use for parallelizing tests.
You can specify these options in the configuration file.
```js
// playwright.config.js
module.exports = {
// Look for test files in the "tests" directory, relative to this configuration file
testDir: 'tests',
// Each test is given 30 seconds
timeout: 30000,
// Forbid test.only on CI
forbidOnly: !!process.env.CI,
// Two retries for each test
retries: 2,
// Limit the number of workers on CI, use default locally
workers: process.env.CI ? 2 : undefined,
use: {
// Configure browser and context here
},
};
```
```ts
// playwright.config.ts
import { PlaywrightTestConfig } from 'playwright/test';
const config: PlaywrightTestConfig = {
// Each test is given 90 seconds.
timeout: 90000,
// Failing tests will be retried at most two times.
// Look for test files in the "tests" directory, relative to this configuration file
testDir: 'tests',
// Each test is given 30 seconds
timeout: 30000,
// Forbid test.only on CI
forbidOnly: !!process.env.CI,
// Two retries for each test
retries: 2,
// Limit the number of workers on CI, use default locally
workers: process.env.CI ? 2 : undefined,
use: {
// Configure browser and context here
},
};
export default config;
```
## Different options for each browser
To specify different options per browser, for example command line arguments for Chromium, create multiple projects in your configuration file. Below is an example that runs all tests in three browsers, with different options.
```js
// playwright.config.js
module.exports = {
// Put any shared options on the top level.
use: {
headless: true,
},
projects: [
{
name: 'chromium',
name: 'Chromium',
use: {
// Configure the browser to use.
browserName: 'chromium',
// Launch options
headless: false,
slowMo: 50,
// Context options
viewport: { width: 800, height: 600 },
ignoreHTTPSErrors: true,
// Testing options
video: 'retain-on-failure',
// Any Chromium-specific options.
viewport: { width: 600, height: 800 },
},
},
{
name: 'Firefox',
use: { browserName: 'firefox' },
},
{
name: 'WebKit',
use: { browserName: 'webkit' },
},
],
};
```
```ts
// playwright.config.ts
import { PlaywrightTestConfig } from 'playwright/test';
const config: PlaywrightTestConfig = {
// Put any shared options on the top level.
use: {
headless: true,
},
projects: [
{
name: 'Chromium',
use: {
// Configure the browser to use.
browserName: 'chromium',
// Any Chromium-specific options.
viewport: { width: 600, height: 800 },
},
},
{
name: 'Firefox',
use: { browserName: 'firefox' },
},
{
name: 'WebKit',
use: { browserName: 'webkit' },
},
],
};
export default config;
```
## Local configuration
Playwright Test will run all projects by default, but you can use `--project` command line option to run a single one.
With `test.use()` you can override some options for a file, or a `describe` block.
```sh
$ npx playwright test
```js
// my.spec.ts
import { test, expect } from "playwright/test";
Running 3 tests using 3 workers
// Run tests in this file with portrait-like viewport.
test.use({ viewport: { width: 600, height: 900 } });
test('my test', async ({ page }) => {
// Test code goes here.
});
✓ example.spec.ts:3:1 › [Chromium] should work (2s)
✓ example.spec.ts:3:1 › [Firefox] should work (2s)
✓ example.spec.ts:3:1 › [WebKit] should work (2s)
```
## Test Options
```sh
$ npx playwright test --project=webkit
- `metadata: any` - Any JSON-serializable metadata that will be put directly to the test report.
- `name: string` - Project name, useful when defining multiple [test projects](#projects).
- `outputDir: string` - Output directory for files created during the test run.
- `repeatEach: number` - The number of times to repeat each test, useful for debugging flaky tests. Overridden by `--repeat-each` command line option.
- `retries: number` - The maximum number of retry attempts given to failed tests. Overridden by `--retries` command line option.
- `screenshot: 'off' | 'on' | 'only-on-failure'` - Whether to capture a screenshot after each test, off by default.
- `off` - Do not capture screenshots.
- `on` - Capture screenshot after each test.
- `only-on-failure` - Capture screenshot after each test failure.
- `snapshotDir: string` - [Snapshots](#snapshots) directory. Overridden by `--snapshot-dir` command line option.
- `testDir: string` - Directory that will be recursively scanned for test files.
- `testIgnore: string | RegExp | (string | RegExp)[]` - Files matching one of these patterns are not considered test files.
- `testMatch: string | RegExp | (string | RegExp)[]` - Only the files matching one of these patterns are considered test files.
- `timeout: number` - Timeout for each test in milliseconds. Overridden by `--timeout` command line option.
- `video: 'off' | 'on' | 'retain-on-failure' | 'retry-with-video'` - Whether to record video for each test, off by default.
- `off` - Do not record video.
- `on` - Record video for each test.
- `retain-on-failure` - Record video for each test, but remove all videos from successful test runs.
- `retry-with-video` - Record video only when retrying a test.
Running 1 test using 1 worker
## Test run options
✓ example.spec.ts:3:1 › [WebKit] should work (2s)
```
These options would be typically different between local development and CI operation:
There are many more things you can do with projects:
- Run a subset of test by specifying different `testDir` for each project.
- Run the same test in multiple configurations, for example with desktop Chromium and emulating Chrome for Android.
- Running "core" tests without retries to ensure stability of the core functionality, and use `retries` for other tests.
- And much more!
- `forbidOnly: boolean` - Whether to exit with an error if any tests are marked as `test.only`. Useful on CI. Overridden by `--forbid-only` command line option.
- `globalSetup: string` - Path to the global setup file. This file will be required and run before all the tests. It must export a single function.
- `globalTeardown: string` - Path to the global teardown file. This file will be required and run after all the tests. It must export a single function.
- `globalTimeout: number` - Total timeout in milliseconds for the whole test run. Overridden by `--global-timeout` command line option.
- `grep: RegExp | RegExp[]` - Patterns to filter tests based on their title. Overridden by `--grep` command line option.
- `maxFailures: number` - The maximum number of test failures for this test run. After reaching this number, testing will stop and exit with an error. Setting to zero (default) disables this behavior. Overridden by `--max-failures` and `-x` command line options.
- `preserveOutput: 'always' | 'never' | 'failures-only'` - Whether to preserve test output in the `outputDir`:
- `'always'` - preserve output for all tests;
- `'never'` - do not preserve output for any tests;
- `'failures-only'` - only preserve output for failed tests.
- `projects: Project[]` - Multiple [projects](#projects) configuration.
- `reporter: 'list' | 'line' | 'dot' | 'json' | 'junit'` - The reporter to use. See [reporters](#reporters) for details.
- `quiet: boolean` - Whether to suppress stdout and stderr from the tests. Overridden by `--quiet` command line option.
- `shard: { total: number, current: number } | null` - [Shard](#shards) information. Overridden by `--shard` command line option.
- `updateSnapshots: boolean` - Whether to update expected snapshots with the actual results produced by the test run. Overridden by `--update-snapshots` command line option.
- `workers: number` - The maximum number of concurrent worker processes to use for parallelizing tests. Overridden by `--workers` command line option.
:::note
`--browser` command line option is not compatible with projects. Specify `browserName` in each project instead.
:::

94
docs/src/test-fixtures.md
View file

@ -14,38 +14,36 @@ Playwright Test is based on the concept of the test fixtures. Test fixtures are
- Playwright Test can efficiently retry the flaky failures, instead of re-running the whole suite.
- You can group tests based on their meaning, instead of their common setup.
Here is how typical test environment setup differs between traditional test style and the fixture-based one:
Here is how typical test environment setup differs between traditional test style and the fixture-based one. We assume a `TodoPage` class that helps interacting with a "todo list" page of the web app.
### Without fixtures
```js
// example.spec.js
// todo.spec.js
const { test } = require('playwright/test');
const { TodoPage } = require('./todo-page');
describe('database', () => {
let table;
describe('todo tests', () => {
let todoPage;
beforeEach(async ()=> {
table = await createTable();
beforeEach(async ({ page }) => {
todoPage = new TodoPage(page);
await todoPage.goto();
await todoPage.addToDo('item1');
await todoPage.addToDo('item2');
});
afterEach(async () => {
await dropTable(table);
await todoPage.removeAll();
});
test('create user', () => {
table.insert();
test('should add an item', async () => {
await todoPage.addToDo('my item');
// ...
});
test('update user', () => {
table.insert();
table.update();
// ...
});
test('delete user', () => {
table.insert();
table.delete();
test('should remove an item', async () => {
await todoPage.remove('item1');
// ...
});
});
@ -54,32 +52,29 @@ describe('database', () => {
### With fixtures
```js
// example.spec.js
// todo.spec.js
const base = require('playwright/test');
const { TodoPage } = require('./todo-page');
// Extend basic test by providing a "table" fixture.
// Extend basic test by providing a "todoPage" fixture.
const test = base.test.extend({
table: async ({}, use) => {
const table = await createTable();
await use(table);
await dropTable(table);
todoPage: async ({ page }, use) => {
const todoPage = new TodoPage(page);
await todoPage.goto();
await todoPage.addToDo('item1');
await todoPage.addToDo('item2');
await use(todoPage);
await todoPage.removeAll();
},
});
test('create user', ({ table }) => {
table.insert();
test('should add an item', async ({ todoPage }) => {
await todoPage.addToDo('my item');
// ...
});
test('update user', ({ table }) => {
table.insert();
table.update();
// ...
});
test('delete user', ({ table }) => {
table.insert();
table.delete();
test('should remove an item', async ({ todoPage }) => {
await todoPage.remove('item1');
// ...
});
```
@ -87,30 +82,27 @@ test('delete user', ({ table }) => {
```ts
// example.spec.ts
import { test as base } from 'playwright/test';
import { TodoPage } from './todo-page';
// Extend basic test by providing a "table" fixture.
const test = base.extend<{ table: Table }>({
table: async ({}, use) => {
const table = await createTable();
await use(table);
await dropTable(table);
const test = base.extend<{ todoPage: TodoPage }>({
todoPage: async ({ page }, use) => {
const todoPage = new TodoPage(page);
await todoPage.goto();
await todoPage.addToDo('item1');
await todoPage.addToDo('item2');
await use(todoPage);
await todoPage.removeAll();
},
});
test('create user', ({ table }) => {
table.insert();
test('should add an item', async ({ todoPage }) => {
await todoPage.addToDo('my item');
// ...
});
test('update user', ({ table }) => {
table.insert();
table.update();
// ...
});
test('delete user', ({ table }) => {
table.insert();
table.delete();
test('should remove an item', async ({ todoPage }) => {
await todoPage.remove('item1');
// ...
});
```

76
docs/src/test-intro.md
View file

@ -212,41 +212,20 @@ test.describe('feature foo', () => {
So far, we've looked at the zero-config operation of Playwright Test. For a real world application, it is likely that you would want to use a config.
Create `playwright.config.js` (or `playwright.config.ts`) to configure your tests. You can specify browser launch options, run tests in multiple browsers and much more with the config. Here is an example configuration that runs every test in Chromium, Firefox and WebKit.
Create `playwright.config.js` (or `playwright.config.ts`) to configure your tests. You can specify browser launch options, run tests in multiple browsers and much more with the config. Here is an example configuration that runs every test in Chromium, Firefox and WebKit. Look for more options in the [configuration section](./test-configuration.md).
```js
module.exports = {
timeout: 30000, // Each test is given 30 seconds.
// Each test is given 30 seconds.
timeout: 30000,
// A project per browser, each running all the tests.
projects: [
{
name: 'chromium',
use: {
browserName: 'chromium',
headless: true,
viewport: { width: 1280, height: 720 },
},
},
use: {
// Run browsers in the headless mode.
headless: true,
{
name: 'webkit',
use: {
browserName: 'webkit',
headless: true,
viewport: { width: 1280, height: 720 },
},
},
{
name: 'firefox',
use: {
browserName: 'firefox',
headless: true,
viewport: { width: 1280, height: 720 },
},
}
],
// Specify the viewport size.
viewport: { width: 1280, height: 720 },
},
};
```
@ -254,37 +233,16 @@ module.exports = {
import { PlaywrightTestConfig } from 'playwright/test';
const config: PlaywrightTestConfig = {
timeout: 30000, // Each test is given 30 seconds.
// Each test is given 30 seconds.
timeout: 30000,
// A project per browser, each running all the tests.
projects: [
{
name: 'chromium',
use: {
browserName: 'chromium',
headless: true,
viewport: { width: 1280, height: 720 },
},
},
use: {
// Run browsers in the headless mode.
headless: true,
{
name: 'webkit',
use: {
browserName: 'webkit',
headless: true,
viewport: { width: 1280, height: 720 },
},
},
{
name: 'firefox',
use: {
browserName: 'firefox',
headless: true,
viewport: { width: 1280, height: 720 },
},
}
],
// Specify the viewport size.
viewport: { width: 1280, height: 720 },
},
};
export default config;
```