By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.
This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.
This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
Currently, running `npx playwright test` in a subfolder of your project
will result in a `test-results` folder created in your `cwd`.
This is unexpected; instead, we should always resolve all paths
against `rootDir` - directory that contains config.
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
- Line reporter now shows stats in addition to the test name:
```log
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.
- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.
- Updated tests to the golden style.
This patch adds support to multiple diffs. These are possible
due to soft assertions.
Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
* `fonts: "ready"`
* `animations: "disabled"`
* `size: "css"`
New default is based on the very (unscientific) output of repeated a few
times:
```
$ for i in `seq 5 13`; do time npm run ctest -- --reporter=line --workers=$i 'page/*' 'selector*' 'channels*' 'resource-timing*'; done
```
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)"
commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot (#12539)"
commit a3dff45974.
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.
The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.
For example, given the following config:
```js
// playwright.config.ts
module.exports = {
projects: [
{ name: 'Mobile Safari' },
{ name: 'Desktop Chrome' },
],
};
```
And the following test structure:
```
smoke-tests/
└── basic.spec.ts
```
Will result in the following screenshots folder structure by default:
```
__screenshots__/
└── darwin/
├── Mobile Safari/
│ └── smoke-tests/
│ └── basic.spec.ts/
│ └── screenshot-expectation.png
└── Desktop Chrome/
└── smoke-tests/
└── basic.spec.ts/
└── screenshot-expectation.png
```
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.
Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"
This reverts commit 73dee69558.
* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"
This reverts commit d744a87aee.
* Revert "chore(test runner): run hooks/modifiers as a part of the test (#12329)"
This reverts commit 47045ba48d.
It's a straightforward change to support new, common, keyboard commands
Note that I've tested this locally with Chrome on my Mac but it seems that CI doesn't want to pass Chrome tests - it's running on ubuntu though. Does this mean that I should introduce per-platform editing commands? At the moment there is only a single [`macEditingCommands`](0ed33522c5/packages/playwright-core/src/server/macEditingCommands.ts) file.
References https://github.com/microsoft/playwright/issues/12000
Co-authored-by: Andrey Lushnikov <aslushnikov@gmail.com>
chore(test runner): run hooks/modifiers as a part of the test
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
This uses `Module._resolveFilename` to intercept module resolution and
check `tsconfig.paths` similarly to pirates usage ot `Module._compile`.
Previously, we resolved during compilation that required reproducible
resolution due to caching. Now we can resolve as we go and support
all `tsconfig.paths`.
- `stdout.isTTY` controls whether list reporter updates lines or just adds them;
- `env.CI` is used in a few places to affect the defaults:
- whether to open interactive html;
- default reporter dot/line;
- default terminal reporter added to non-terminal reporters;
- `env.PWTEST_SKIP_TEST_OUTPUT` is removed;
- `env.PW_TEST_DEBUG_REPORTERS` is introduced specifically for tests.
This reverts commit e5c9d1e39f.
Reason for revert: turns out this fix results in a 5-second delay
when starting tests in docker, with `test-results` folder being
a non-removable mount.
The reason for the delay is the `maxBusyTries` option that we
supply by default to rimraf when trying to remove the folder.
While this option might come handy when removing temporary
browser profile folder, it doesn't serve us well in this particular
usecase.
References #12106
This way we control the timeout error message from the runner,
so that later on we can differentiate between test timeout, fixture
timeout and hook timeout.
This patch prepares for the `toHaveScreenshot` implementation
by splitting common parts from `toMatchSnapshot`.
Drive-by: fix default extension generation from `.bin` to `.dat`
for unknown buffers.
This patch adds additional options to `toMatchSnapshot` method:
- `pixelCount` - acceptable number of pixels that differ to still
consider images equal. Unset by default.
- `pixelRatio` - acceptable ratio of all image pixels (from 0 to 1) that differ to still
consider images equal. Unset by default.
Fixes#12167, #10219
In experimental ESM mode a child process is forked in order to run the tests. Currently the exit code of this child process is not propagated to the exit code of the parent process, which means that the process exits with a status code of `0` even if some of the tests failed.
This makes it difficult to use Playwright in CI in experimental mode, as the CI pipeline as a whole will pass despite the test failures.
This change addresses this by propagating the exit code in the case where it is non-zero.
This changes PlaywrigtServer to serve connections like `ws://localhost:3333/?browser=chromium`:
- launches the browser;
- talks `browserType.connect`-style protocol over websocket;
- compatible with `connectOptions` fixture.
```js
await playwright.chromium.connect({ wsEndpoint: 'ws://localhost:3333/?browser=chrome' });
```
The new (as of 1.18) `async testInfo.attach(…)` API handles this
gracefully (and is part of the reason for the new API's existence).
However, for the foreseeable future, it's still possible to manually
push onto the attachments array where we can't validate the contents
until it's too late, so this change ensures more graceful handling in
that case.
Fixes#11565
This option stops all kinds of CSS animations while doing screenshot:
- CSS animations
- CSS transitions
- Web Animations
Animations get different treatment depending on animation duration:
- finite animations are fast-forwarded to its end, issuing the
`transitionend` event.
- Infinite animations are resetted to its beginning, and then
resumed after the screenshot.
References #9938, fixes#11912
Soft expects will still fail the test, but will not abort it's execution. As a consequence of this:
- `TestResult` now might have multiple errors, which is reflected with a new `testResult.erros: TestError[]` field.
- `TestInfo` now might have multiple errors as well, which is reflected with a new `testInfo.errors: TestError[]` field.
Fixes#7819
This introduces `locator('div', { has: locator })` syntax that matches elements containing other elements.
Can be used together with `hasText`.
Internally, has selector engine takes an inner selector escaped with double-quotes:
`div >> has="li >> span >> text=Foo" >> span`.
When element that is being dragged stays under the mouse,
it prevents the hit target check on drop from working,
because drop target is overlayed by the dragged element.
To workaround this, we perform a one-time hit target check
before moving for the drop, as we used to.
These variables aren't CLI-only anymore, so pick some more general
names for them.
Note: all language ports would need to follow-up with the rename
after the next roll.
Fixes#11450
This reworks DeadlineRunner to use exception to signal timeout. This way,
we'll be able to run fixtures against a shared deadline vs their own
deadline and still get an easy control-flow timeout handling.
In several of the Playwright APIs, falsey values were not handled correctly. This changeset adds tests (and some fixes):
- route.continue: If options.postData was the empty string, the continue failed to override the post data.
- page.post (application/json with options.data: false|''|0|null): Raw falsey values were getting dropped (i.e. you can't do the equivalent of curl --header application/json … -d 'false'). This has been fixed with most values across all browsers, but an additional fix is needed for 'null' which the channel serializer treats extra specially.
- testInfo.attach: This didn't get reported as an error when options.path was the empty string, but should have been.
#11413 (and its fix#11414) inspired this search as they are the same
class of bug.
- Use file path, not content to calculate the attachment hash.
- Always cleanup fixture from the list on teardown, to avoid reporting
teardown error multiple times: from the test, and from the cleanup.
We do not have a timeout for any other close method, such as
browserContext.close or browser.close, and hitting default
30 seconds is very realistic with large Electron apps.
Previously, reporter would look for a stack frame directly in the test file.
Often times, that is not a top stack frame, especially when the test uses
some helper functions.
This changes error snippets and locations to use the top frame. When top
frame does not match the test file, we additionally show the location
to avoid confusion:
```
1) a.spec.ts:7:7 › foobar ========================================================================
Error: oh my
at helper.ts:5
3 |
4 | export function ohMy() {
> 5 | throw new Error('oh my');
| ^
6 | }
7 |
at ohMy (.../reporter-base-should-print-codeframe-from-a-helper/helper.ts:5:15)
at .../reporter-base-should-print-codeframe-from-a-helper/a.spec.ts:8:9
at FixtureRunner.resolveParametersAndRunHookOrTest (.../src/fixtures.ts:281:12)
```
In case of self-hosted github runners, it's much easier to checkout
under `playwright-internal` folder name instead of a default
`playwright` name. This confuses our stack collection.
This patch makes it generic.
This changes previous layout shift attempt (see #9546)
to account for more valid usecases:
- On the first event that is intercepted we enforce the hit target. This
is similar to the current mode that checks hit target before the action,
but is better timed.
- On subsequent events we assume that everything is fine. This covers more
scenarios like react rerender, glass pane on mousedown, detach on mouseup.
This check is enabled by default, with `process.env.PLAYWRIGHT_NO_LAYOUT_SHIFT_CHECK`
to opt out.
Textual snapshot diffs were previously broken in the HTML Report. The strikethrough'd text extended beyond the intended region.
HTML Report Before:
<img width="693" alt="Screen Shot 2021-12-27 at 4 43 35 PM" src="https://user-images.githubusercontent.com/11915034/147518750-a60f9002-6eed-48a1-a412-20fabd076fa6.png">
HTML Report After:
<img width="206" alt="Screen Shot 2021-12-27 at 4 48 37 PM" src="https://user-images.githubusercontent.com/11915034/147518762-19a4c8f9-ccc3-4a3c-a962-5a42edc6fc5d.png">
This now matches what's expected and shown in the terminal (which has always been correct):
<img width="1384" alt="Screen Shot 2021-12-27 at 4 36 29 PM" src="https://user-images.githubusercontent.com/11915034/147518799-f538259e-5a45-4d6f-916c-a12ccb620c5b.png">
NB: This MR is a workaround, but not a root cause fix. It works, but I never fully got to the root cause so a bug upstream may be required. It's unclear whether it's (1) in [`colors`](https://www.npmjs.com/package/colors), (2) in [`ansi-to-html`](https://www.npmjs.com/package/ansi-to-html), or (3) Playwright's use of the two. Since the terminal output is correct, I suspect it is in `ansi-to-html`. For example:
```js
const colors = require("colors");
const Convert = require('ansi-to-html');
const convert = new Convert();
// original (strike incorrectly wraps everything in the HTML)
console.log(convert.toHtml(colors.strikethrough("crossed out") + ' ' + colors.red("red")))
// prints: <strike>crossed out <span style="color:#A00">red<span style="color:#FFF"></span></span></strike>
// workaround
console.log(convert.toHtml(colors.reset(colors.strikethrough("crossed out")) + ' ' + colors.red("red")))
// prints: <strike>crossed out</strike> <span style="color:#A00">red<span style="color:#FFF"></span></span>
```
Fixes#11116
## Before
* When docker wasn't running: crashed with a TypeError
* When any image without `RepoTags` was present: crashed with TypeError
* Typo in recommended command ("playwight")
* No indication of `PW_TEST_IMAGE` env var in error output
<details>
<summary>Docker not running</summary>
```
Using config at ~redacted/playwright.config.ts
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused
TypeError: Cannot read property 'find' of null
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
test.ts:9:1 › some redacted description
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused
TypeError: Cannot read property 'find' of null
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js
```
</details>
<details>
<summary>Image without repo tags</summary>
```
Using config at ~redacted/playwright.config.ts
TypeError: Cannot read property 'includes' of undefined
at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
at Array.find (<anonymous>)
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
some-test.ts › some description
TypeError: Cannot read property 'includes' of undefined
at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
at Array.find (<anonymous>)
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element (26ms)
1) some-test.ts › some description =====
'Grid agent creation failed'
1 failed
some-test.ts › some description ======
```
</details>
## After
* Helpful error message if docker isn't running
* Doesn't crash when local-only images in list
* No typo in `playwright install docker-image` command
* When other playwright images are found they are listed and `PW_TEST_IMAGE` is mentioned
### After: Docker not running
```
Using config at ~redacted/playwright.config.ts
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images ║
║ Please ensure docker daemon is running. ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
test.ts:9:1 › description
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images ║
║ Please ensure docker daemon is running. ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ test.ts:9:1 › description
1) test.ts:9:1 › description =====
'Grid agent creation failed'
1 failed
test.ts:9:1 › description ======
```
## After: No exact match found
```
Using config at ~redacted/playwright.config.ts
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image. ║
║ ║
║ Available images: ║
║ - mcr.microsoft.com/playwright:v1.16.3 ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal ║
║ ║
║ Use available images via PWTEST_IMAGE_NAME environment variable: ║
║ PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║ ║
║ Alternatively, please pull docker image with the following command: ║
║ ║
║ npx playwright install docker-image ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image. ║
║ ║
║ Available images: ║
║ - mcr.microsoft.com/playwright:v1.16.3 ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal ║
║ ║
║ Use available images via PWTEST_IMAGE_NAME environment variable: ║
║ PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║ ║
║ Alternatively, please pull docker image with the following command: ║
║ ║
║ npx playwright install docker-image ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ test.ts:9:1 › description
1) test.ts:9:1 › description =====
'Grid agent creation failed'
1 failed
test.ts:9:1 › description ======
```
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
When configuring a proxy, Chromium requires a magic tokens to get some
local network requests to go through the proxy. This has tripped up a
few users, so we make the behavior default to the expected: proxy
everything including the local requests. This matches the other vendors
as well.
NB: This can be disabled via
`PLAYWRIGHT_DISABLE_FORCED_CHROMIUM_PROXIED_LOOPBACK=1`
Supercedes: #8345Fixes: #10631
