Filip/playwright
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

docs: update test runner docs (#6795)

Browse source
This commit is contained in:
Dmitry Gozman 2021-05-29 08:58:17 -07:00 committed by GitHub
parent 7f0d817afd
commit f7e720568d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 207 additions and 33 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

31
docs/src/test-annotations.md
View file

@ -6,23 +6,38 @@ title: "Annotations"
Sadly, tests do not always pass. Playwright Test supports test annotations to deal with failures, flakiness and tests that are not yet ready.
```js
// example.spec.ts
import { test } from 'playwright/test';
// example.spec.js
const { test, expect } = require('playwright/test');
test('basic', async ({ table }) => {
test.skip(version == 'v2', 'This test crashes the database in v2, better not run it.');
test('some feature', async ({ page, browserName }) => {
test.skip(browserName !== 'webkit', 'This feature is iOS-only');
// Test goes here.
});
test('can insert multiple rows', async ({ table }) => {
test.fail('Broken test, but we should fix it!');
test('another feature', async ({ page }) => {
test.fail(true, 'Broken, need to fix!');
// Test goes here.
});
```
Annotations may be conditional, in which case they only apply when the condition is truthy. Annotations may depend on test arguments. There could be multiple annotations on the same test, possibly in different configurations.
```ts
// example.spec.ts
import { test, expect } from 'playwright/test';
Possible annotations include:
test('some feature', async ({ page, browserName }) => {
test.skip(browserName !== 'webkit', 'This feature is iOS-only');
// Test goes here.
});
test('broken feature', async ({ page }) => {
test.fail();
// Test goes here.
});
```
Annotations apply when the condition is truthy, or always when no condition is passed, and may include a description. Annotations may depend on test fixtures. There could be multiple annotations on the same test, possibly in different configurations.
Available annotations:
- `skip` marks the test as irrelevant. Playwright Test does not run such a test. Use this annotation when the test is not applicable in some configuration.
- `fail` marks the test as failing. Playwright Test will run this test and ensure it does indeed fail. If the test does not fail, Playwright Test will complain.
- `fixme` marks the test as failing. Playwright Test will not run this test, as opposite to the `fail` annotation. Use `fixme` when running the test is slow or crashy.

53
docs/src/test-cli.md
View file

@ -10,21 +10,40 @@ npx playwright test --help
Arguments passed to `npx playwright test` are treated as a filter for test files. For example, `npx playwright test my-spec` will only run tests from files with `my-spec` in the name.
All the options are available in the [configuration file](#writing-a-configuration-file). However, selected options can be passed to a command line and take a priority over the configuration file:
- `--config <file>` or `-c <file>`: Configuration file. Defaults to `pwtest.config.ts` or `pwtest.config.js` in the current directory.
- `--forbid-only`: Whether to disallow `test.only` exclusive tests. Useful on CI. Overrides `config.forbidOnly` option from the configuration file.
- `--grep <grep>` or `-g <grep>`: Only run tests matching this regular expression, for example `/my.*test/i` or `my-test`. Overrides `config.grep` option from the configuration file.
- `--global-timeout <number>`: Total timeout in milliseconds for the whole test run. By default, there is no global timeout. Overrides `config.globalTimeout` option from the configuration file.
- `--help`: Display help.
All the options are available in the [configuration file](./test-configuration.md). However, selected options can be passed to a command line and take a priority over the configuration file.
- `--headed`: Run tests in headed browsers. Useful for debugging.
- `-c <file>` or `--config <file>`: Configuration file. If not passed, defaults to `playwright.config.ts` or `playwright.config.js` in the current directory.
- `-c <dir>` or `--config <dir>`: Directory with the tests to run without configuration file.
- `--forbid-only`: Whether to disallow `test.only`. Useful on CI.
- `-g <grep>` or `--grep <grep>`: Only run tests matching this regular expression. For example, this will run `'should add to cart'` when passed `-g="add to cart"`.
- `--global-timeout <milliseconds>`: Total timeout for the whole test run. By default, there is no global timeout.
- `--list`: List all the tests, but do not run them.
- `--max-failures <N>` or `-x`: Stop after the first `N` test failures. Passing `-x` stops after the first failure. Overrides `config.maxFailures` option from the configuration file.
- `--output <dir>`: Directory for artifacts produced by tests, defaults to `test-results`. Overrides `config.outputDir` option from the configuration file.
- `--quiet`: Whether to suppress stdout and stderr from the tests. Overrides `config.quiet` option from the configuration file.
- `--repeat-each <number>`: Specifies how many times to run each test. Defaults to one. Overrides `config.repeatEach` option from the configuration file.
- `--reporter <reporter>`. Specify reporter to use, comma-separated, can be some combination of `dot`, `json`, `junit`, `line`, `list` and `null`. See [reporters](#reporters) for more information.
- `--retries <number>`: The maximum number of retries for each [flaky test](#flaky-tests), defaults to zero (no retries). Overrides `config.retries` option from the configuration file.
- `--shard <shard>`: [Shard](#shards) tests and execute only selected shard, specified in the form `current/all`, 1-based, for example `3/5`. Overrides `config.shard` option from the configuration file.
- `--project <project...>`: Only run tests from one of the specified [projects](#projects). Defaults to running all projects defined in the configuration file.
- `--timeout <number>`: Maximum timeout in milliseconds for each test, defaults to 10 seconds. Overrides `config.timeout` option from the configuration file.
- `--update-snapshots` or `-u`: Whether to update snapshots with actual results instead of comparing them. Use this when snapshot expectations have changed. Overrides `config.updateSnapshots` option from the configuration file.
- `--workers <workers>` or `-j <workers>`: The maximum number of concurrent worker processes. Overrides `config.workers` option from the configuration file.
- `--max-failures <N>` or `-x`: Stop after the first `N` test failures. Passing `-x` stops after the first failure.
- `--output <dir>`: Directory for artifacts produced by tests, defaults to `test-results`.
- `--quiet`: Whether to suppress stdout and stderr from the tests.
- `--repeat-each <N>`: Run each test `N` times, defaults to one.
- `--reporter <reporter>`: Choose a reporter: minimalist `dot`, concise `line` or detailed `list`. See [reporters](./test-reporters.md) for more information.
- `--retries <number>`: The maximum number of [retries](./test-retries.md) for flaky tests, defaults to zero (no retries).
- `--shard <shard>`: [Shard](./test-parallel.md#shards) tests and execute only selected shard, specified in the form `current/all`, 1-based, for example `3/5`.
- `--project <project...>`: Only run tests from one of the specified [projects](./test-advanced.md#projects). Defaults to running all projects defined in the configuration file.
- `--timeout <number>`: Maximum timeout in milliseconds for each test, defaults to 10 seconds.
- `--update-snapshots` or `-u`: Whether to update [snapshots](./test-snapshots.md) with actual results instead of comparing them. Use this when snapshot expectations have changed.
- `--workers <workers>` or `-j <workers>`: The maximum number of concurrent worker processes that run in [parallel](./test-parallel.md).

18
docs/src/test-examples.md
View file

@ -9,12 +9,24 @@ title: "Examples"
## Multiple pages
The default `context` argument is a [BrowserContext][browser-context]. Browser contexts are isolated execution environments that can host multiple pages. See [multi-page scenarios](./multi-pages.md) for more examples.
The default `context` argument is a [BrowserContext]. Browser contexts are isolated execution environments that can host multiple pages. See [multi-page scenarios](./multi-pages.md) for more examples.
```js
import { test } from "playwright/test";
// example.spec.js
const { test } = require('playwright/test');
test("tests on multiple web pages", async ({ context }) => {
test('tests on multiple web pages', async ({ context }) => {
const pageFoo = await context.newPage();
const pageBar = await context.newPage();
// Test function
});
```
```ts
// example.spec.ts
import { test } from 'playwright/test';
test('tests on multiple web pages', async ({ context }) => {
const pageFoo = await context.newPage();
const pageBar = await context.newPage();
// Test function

134
docs/src/test-fixtures.md
View file

@ -19,7 +19,7 @@ Here is how typical test environment setup differs between traditional test styl
### Without fixtures
```js
// example.spec.ts
// example.spec.js
describe('database', () => {
let table;
@ -54,6 +54,37 @@ describe('database', () => {
### With fixtures
```js
// example.spec.js
const base = require('playwright/test');
// Extend basic test by providing a "table" fixture.
const test = base.test.extend({
table: async ({}, use) => {
const table = await createTable();
await use(table);
await dropTable(table);
},
});
test('create user', ({ table }) => {
table.insert();
// ...
});
test('update user', ({ table }) => {
table.insert();
table.update();
// ...
});
test('delete user', ({ table }) => {
table.insert();
table.delete();
// ...
});
```
```ts
// example.spec.ts
import { test as base } from 'playwright/test';
@ -93,6 +124,19 @@ There are two types of fixtures: `test` and `worker`. Test fixtures are set up f
Test fixtures are set up for each test. Consider the following test file:
```js
// hello.spec.js
const test = require('./hello');
test('hello', ({ hello }) => {
test.expect(hello).toBe('Hello');
});
test('hello world', ({ helloWorld }) => {
test.expect(helloWorld).toBe('Hello, world!');
});
```
```ts
// hello.spec.ts
import test from './hello';
@ -110,8 +154,31 @@ It uses fixtures `hello` and `helloWorld` that are set up by the framework for e
Here is how test fixtures are declared and defined. Fixtures can use other fixtures - note how `helloWorld` uses `hello`.
```js
// hello.js
const base = require('playwright/test');
// Extend base test with fixtures "hello" and "helloWorld".
// This new "test" can be used in multiple test files, and each of them will get the fixtures.
module.exports = base.test.extend({
// This fixture is a constant, so we can just provide the value.
hello: 'Hello',
// This fixture has some complex logic and is defined with a function.
helloWorld: async ({ hello }, use) => {
// Set up the fixture.
const value = hello + ', world!';
// Use the fixture value in the test.
await use(value);
// Clean up the fixture. Nothing to cleanup in this example.
},
});
```
```ts
// hello.ts
import { test as base } from 'playwright/test';
import base from 'playwright/test';
// Define test fixtures "hello" and "helloWorld".
type TestFixtures = {
@ -148,6 +215,22 @@ Playwright Test uses worker processes to run test files. You can specify the max
Here is how the test looks:
```js
// express.spec.js
const test = require('./express-test');
const fetch = require('node-fetch');
test('fetch 1', async ({ port }) => {
const result = await fetch(`http://localhost:${port}/1`);
test.expect(await result.text()).toBe('Hello World 1!');
});
test('fetch 2', async ({ port }) => {
const result = await fetch(`http://localhost:${port}/2`);
test.expect(await result.text()).toBe('Hello World 2!');
});
```
```ts
// express.spec.ts
import test from './express-test';
import fetch from 'node-fetch';
@ -165,6 +248,51 @@ test('fetch 2', async ({ port }) => {
And here is how fixtures are declared and defined:
```js
// express-test.js
const base = require('playwright/test');
const express = require('express');
// Define "port" and "express" worker fixtures.
module.exports = base.test.extend({
// We pass a tuple to specify fixtures options.
// In this case, we mark this fixture as worker-scoped.
port: [ async ({}, use, workerInfo) => {
// "port" fixture uses a unique value of the worker process index.
await use(3000 + workerInfo.workerIndex);
}, { scope: 'worker' } ],
// "express" fixture starts automatically for every worker - we pass "auto" for that.
express: [ async ({ port }, use) => {
// Setup express app.
const app = express();
app.get('/1', (req, res) => {
res.send('Hello World 1!')
});
app.get('/2', (req, res) => {
res.send('Hello World 2!')
});
// Start the server.
let server;
console.log('Starting server...');
await new Promise(f => {
server = app.listen(port, f);
});
console.log('Server ready');
// Use the server in the tests.
await use(server);
// Cleanup.
console.log('Stopping server...');
await new Promise(f => server.close(f));
console.log('Server stopped');
}, { scope: 'worker', auto: true } ],
});
```
```ts
// express-test.ts
import { test as base } from 'playwright/test';
import express from 'express';
@ -179,7 +307,7 @@ type ExpressWorkerFixtures = {
// Note that we did not provide an test-scoped fixtures, so we pass {}.
const test = base.extend<{}, ExpressWorkerFixtures>({
// We pass a tuple to with the fixture function and options.
// We pass a tuple to specify fixtures options.
// In this case, we mark this fixture as worker-scoped.
port: [ async ({}, use, workerInfo) => {
// "port" fixture uses a unique value of the worker process index.

4
docs/src/test-intro.md
View file

@ -100,8 +100,8 @@ Here is a list of the pre-defined fixtures that you are likely to use most of th
|Fixture |Type |Description |
|:----------|:----------------|:--------------------------------|
|page |[Page] |Isolated page for this test run. |
|context |[BrowserContext] |Isolated context for this test run. The `page` fixture belongs to this context as well. Learn how to [configure context](#modify-options) below. |
|browser |[Browser] |Browsers are shared across tests to optimize resources. Learn how to [configure browser](#modify-options) below. |
|context |[BrowserContext] |Isolated context for this test run. The `page` fixture belongs to this context as well. Learn how to [configure context](./test-configuration.md). |
|browser |[Browser] |Browsers are shared across tests to optimize resources. Learn how to [configure browser](./test-configuration.md). |
|browserName|[string] |The name of the browser currently running the test. Either `chromium`, `firefox` or `webkit`.|
## Test and assertion features