This will result in the following message:
```
root@docker-desktop:/playwright# node a.mjs
node:internal/process/esm_loader:94
internalBinding('errors').triggerUncaughtException(
^
browserType.launch: Executable doesn't exist at /ms-playwright/chromium-1021/chrome-linux/chrome
╔══════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just updated to 1.26.0. ║
║ Please update docker image as well. ║
║ - current: mcr.microsoft.com/playwright:v1.23.0-focal ║
║ - required: mcr.microsoft.com/playwright:v1.26.0-focal ║
║ ║
║ <3 Playwright Team ║
╚══════════════════════════════════════════════════════════════════════╝
at file:///playwright/a.mjs:3:43 {
name: 'Error'
}
```
Fixes#15483
We lack `documentId` when doing a reload over browser protocols, so
`reload()` waits for the next navigation to finish. Sometimes, the page
might issue a same-document navigation while reload is in progress,
which confuses the reload command.
To fix the issue, just ignore same-document navigations for reload,
because it is always a new document.
This aligns all retargeting in a single place, so that we
can pass around "retarget strategy" to make sure we retarget
as expected in every single action.
After protocol fixes in all browsers, we can now scroll and click display:contents elements.
The only problem is that `elementsFromPoint()` misbehaves in Chromium and Firefox, so we
need a workaround. Hopefully, it will be fixed upstream - shadow dom spec folks think
"it becomes a real compatibility concern".
This needs Chromium 105 roll.
Also, exlude certain errors from triggering this message:
- `no tests found`
- `duplicate test titles are not allowed`
- `--forbid-only found a focused test`
- `Timed out waiting 3600s for the entire test run`
Previously, when some iframe started/finished a new navigation, we
could have removed and then re-addded load/domcontentloaded on the
main frame.
Drive-by: unflake wheel test in Firefox.
In the following example, `locale` inside the `describe`
would be reverted to the default value:
```js
test.use({ locale: 'en-GB' });
test.decsribe(() => {
test.use({ locale: undefined });
});
```
It could be that iframe was blocking some event, like load or networkidle,
and we never updated the lifecycle after the iframe was detached. This
lead to goto and other navigation commands to never resolve.
A small change in the API requires `shortCircuit: true` when returning the
transformed source.
Another API change removes third argument from the chained resolve/load,
but passing three instead of two still works.
As of July 14, 2022, Safari Technoly Preview returns the following:
```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15
```
Fixes#15474.
Notes:
* page-level requests that are also handled by a SW's fetch handler, should not be interceptable at the page-level
* `Network.requestWillBeSent` does not provide enough metadata for Playwright to fire the `request` event at that time, so it does it as soon as it gets to the end of the request lifecycle
Line reporter now shows stats in addition to the test name:
```
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
[21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
- Two lines per test: title and location.
- Align project labels.
- Add trace badge that opens trace viewer.
- Add video and image diff badges that show scrolled test result view.
fix(test runner): more firendly test duration
Reported test duration now does not include time spent in
`beforeAll`, `afterAll` and fixtures that have a separate timeout.
This is to avoid different reported test execution time,
depending on the test execution order.
Also reverted screenshot test changes from 4de14e7 as the breaking change was reverted upstream in Revert [251706@main] getComputedStyle(img).height returns string of a rounded int not a float WebKit/WebKit#1902
Fixes#8206.
Since #8206 is a long-awaited (~ 1 year old), popular (~ 45 reactions, frequently requested in community channels, etc.), this PR aims to unblock folks.
Notably, we do not innovate on the `webServer` API, despite knowing we're not in love with it. We'll save the innovation for either Plugins or a new `LaunchConfigs` option. (We haven't yet arrived at a Plugin API we like, and instead of launching a new option guessing what the "better" launchConfig API would be, let's wait and see how folks use this new Array-variant of `webServer` which—despite its name—can be used for non-Web Server launches!
When running without tty, line reporter outputs a line for each
percent of the tests, thus limiting the output to ~100 lines.
In addition, reporters now support PLAYWRIGHT_LIVE_TERMINAL
env variable to force tty mode.
* `clip` option is always passed from the client code
* with this change, we can no longer capture screenshot of a blinking
caret; the browser-side API doesn't have this capability.
Useful to set some options without coming up with an artificial name.
```js
test.describe(() => {
test.use({ colorScheme: 'dark' });
test('my test', () => {});
})
```
Currently, it is possible to run a function, e.g. a second `beforeEach` hook,
that will receive `null` for the fixture that has already failed before.
This PR skips running any user function that uses a fixture that has already
failed, just like if the fixture would be initialized again and failing for
the second time.
The following options now work across languages:
- `recordHar`
- `serviceWorkers`
In addition, object properties are now sorted alphabetically.
Drive-by: fixed `--target` help message to include all available targets.
Resolves#14255.
PNGs should be preferred as they are deterministic, but increasing this
limit should be fine for JPEG users. (Looking through jpeg-js source code, the actual memory allocation is based on the size of the image—so unless a user is hitting this limit already—this should not impact the memory consumption of Playwright.
Currently, if `text.fixme()` or `test.skip()` is used within a test, we
add a `fixme` or `skip` annotation. However, if the wrapper style is
used:
```
test.fixme('should work', () => {…})
```
the annotations were missing. This change adds annotations for the
above.
These annotations are important for reporting purposes and knowing
exactly what flavor of "skipped" was used.
Fixes#15239.
- Never use open shadow root for highlight. This messes up
our selectors that accidentally match internal preview elements.
- Remove failing electron test that we do not care about.
- Skip `channels.spec.ts` in non-default mode.
Previously, we only validated/converted on the way to the server,
but not from the server.
Validating both ways catches issues earlier, and allows us to
perform automatic conversions, for example only converting
buffers to base64 when sending over wire.
Some of the current timeout error messages are confusing, because they do not suggest that the issue is most likely a slow test. This PR updates timeout messages as follows:
- Test timeout of 30000ms exceeded.
- Test timeout of 30000ms exceeded while setting up "browser".
- Test timeout of 30000ms exceeded while tearing down "context".
- Test timeout of 30000ms exceeded while setting up "playwright configuration".
- Test timeout of 30000ms exceeded while running "beforeEach" hook.
- Test timeout of 30000ms exceeded while running "afterEach" hook.
- "beforeAll" hook timeout of 30000ms exceeded.
- "afterAll" hook timeout of 30000ms exceeded.
- Worker teardown timeout of 30000ms exceeded.
- "skip" modifier timeout of 30000ms exceeded.
- Fixture "myCustomFixture" timeout of 5000ms exceeded.
Potential fixes to avoid startup races:
- Wait for "juggler listening" message.
- Make sure `transport.onclose` is called when connecting to the transport after the actual pipe closure.
- Do not resolve raw headers upon `loadingFinished`, since they may still come later in
`responseReceivedExtraInfo`.
- Introduce separate promises for `encodedBodySize`, `transferSize` and `responseHeadersSize`.
- Make sure we resolve each of them either with data available
from the browser, or a fallback calculation.
- Set raw response headers for redirects on WebKit.
- Do not stall on cached responses in Chromium, they have erroneously set `hasExtraInfo` flag.
- Use `transferSize` that is available in Firefox protocol.
Previously, screenshot was taken after hooks and fixtures teardown.
However, hooks can easily modify the state of the page, and
screenshot would not reflect the moment of failure.
Instead, we take screenshots immediately after the test function
finishes with an error.
These mean "I don't want to specify this fixture/option"
instead of "I want the value of undefined", aligned with how TypeScript works.
We already do similar things in the config.
fix(network): make allHeaders wait until all header are available
Before, calling `allHeaders()` from `page.on('request')` would yield
provisional headers instead.
With these changes:
- In Firefox, all headers are available immediately.
- In Chromium, all headers are available upon requestWillBeSentExtraInfo.
- In WebKit, all headers are available upon responseReceived.
- In all browsers, intercepted requests use "provisional" headers
as all headers, since there is no network stack to change the headers.
Drive-by: migrated Chromium to `hasExtraInfo` flags that simplifies
the logic quite a bit.
When target element is inside a non-main frame, there could be an
overlay in some of the parent frames that intercepts pointer events.
However, we never detected this case.
Fixes#14904.
This is done to make looking at the raw contents of the report
friendlier when using a file browser. However, it should be noted, the
public API of the HTML Reporter makes no guarantees of its contents
structure/layout/naming-conventions.
Currently, `loadstate` and `load` are two separate events in the protocol,
and are fired in this order. As a result, `waitForLoadState()` sometimes
resolves before the `'load'` event is fired, which is unexpected.
Also fixes a flaky test that assumed `load` event comes after `domcontentloaded`
for the empty page, which is not always a case in Chromium.
- `har` option is now an object `{ path, fallback }`.
- Allows falling back to `abort()`, `continue()` or throwing.
- Matches based on url + method.
- Follows redirects in the HAR file.
- Nice error/stack when throwing.
- Tests.
Adds cross-browser support for easily allowing/blocking Service Workers via a Context option.
Includes plumbing for Playwright Test's `use`.
Resolves#14522.
Relates #1090.
Supercedes #14321.
`npx playwright open --save-har=file.har` produces a HAR file that
can be later inspected or used for mocks.
`--save-har-glob="**/*.{png,woff2}"` option can be used to
produce a smaller har that only contains entries matching the glob.
Language override behavior changed upstream in WebKit/WebKit@039ebd9
New logic is closer to the actual behavior of WebKit on macOS, meaning that when the user changes system language the actual locale changes according to some weird OS rules:
ru-RU => navigator.language === 'ru'
fr-CH => navigator.language === 'fr-FR'
es-MX => navigator.language === 'es-MX'
Our locale emulation is aligned with that, so setting locale to fr-CH will result in fr-FR etc.
fix: sanitize URLs with vbscript:
The vbscript: protocols can be used to run scripts in much the same way as the javascript: protocol. This PR adds in validation for those aforementioned protocols in snapshotterInjected.ts and snapshotRenderer.ts.
This restores the old hit target check, in addition to the new
hit target interceptor.
This way, we got some coverage for iframes and other quirky cases,
but keep the bullet-proof hit target check in place.
If the tests are in an explicitly configured testDir (either at the global config level or per project) .gitignore filters are not applied.
Fixes#14381
Fixes focus and blur management when `page.locator(…).focus()` and `page.locator(…).type(…)` are used which was regressed by 7a5b070 (#13510).
#13510 relied on an implicit assumption that this (conditional) [`blur`](7a5b070e95/packages/playwright-core/src/server/injected/injectedScript.ts (L672)) call would always be followed by a call that resulted in a newly focused element via this [`focus`](7a5b070e95/packages/playwright-core/src/server/injected/injectedScript.ts (L674)) call.
However, some elements are [not focusable](https://html.spec.whatwg.org/multipage/interaction.html#focusable-area), so we were blurring incorrectly, and losing focus that we should have maintained.
Two regression tests were added that pass on the commit prior to 7a5b070e95 (and match manual testing/expectations):
* `page.locator(…).focus()`: _keeps focus on element when attempting to focus a non-focusable element_
* `page.locator(…).type(…)`: _should type repeatedly in input in shadow dom_
Additionally, a third test (_should type repeatedly in input in shadow dom_) was added to check the invariant from #13510 that states:
> This affects [contenteditable] elements, but not input elements.
and allows us to introduce the targeted fix (contenteditble check before blur) without breaking FF again.
And _should type repeatedly in contenteditable in shadow dom with nested elements_ was added to ensure the above fix works with nest contenteditble detection.
Fixes#14254.
There are a few issues this covers:
- Some fixtures like `page` and `context` are not allowed in `beforeAll`
hooks, so using them in automatic fixture makes it throw.
- Running automatic fixture solely for `afterAll` is unexpected.
This currently happens when `afterAll` is run for cleanup after
fixture timeout/throw.
For built-in playwright fixture, we keep `'all-hooks-included'` auto mode.
Added a doc explaining the execution order.
References #14037.
We used to query and save all non-zero `scrollTop`/`scrollLeft` to restore the page perfectly. However, this became very slow with Chromium v100 regression (see issue).
This change only queries and stores `scrollTop`/`scrollLeft` for elements that are ancestors of the action target. The resulting snapshot does not perfectly recreate the original screen, but should be good enough to inspect the action.
fix(chromium): work around about:blank issue on Chromium
We don't receive the `loaderId` which translates to `newDocumentId`,
so we expect the same-document navigation. Instead, we can wait
for any new-document navigation as a workaround, only for `about:blank`.
This also reverts commit f0f65fa247.
This patch:
- adds a hard dependency from `experimental-ct-*` packages to the
same-version of `@playwright/test`
- aligns `experimental-ct-*` package versions with main package
version
- starts publishing experimental CT packages together with other
packages
We should not fast-return if someone tries to run
`npx playwright install-deps` on Debian.
Instead, we should warn users, fallback to Ubuntu 20.04 dependencies,
and proceed.
Note: all toHaveScreenshot tests still use `__screenshots__` directory
for their expectations. One more test was added to make sure that
by default, `toHaveScreenshot` uses old snapshots.
Currently, when I choose "Java" in the sources list and then
click on the page to generate the "click" action, sources reset
to "JavaScript". This is very inconvenient.
This changes the logic to only forcefully change files if either
old or new file is a user file, not a generated one.
Use cases considered:
- run `codegen`, click around, choose different language, click more;
- run script with inspector, pause, click "Record" and record an action;
- same as above, but then continue and see that user source is revealed.
This fixes a few issues:
- strict mode was producing false negatives if multiple query paths
lead to the same element being picked;
- in some cases the number of intermediate items in the list was
exponential and crashed quickly.
What changed:
- `visible` engine is a real engine now;
- `capture` selectors are transformed to `has=` selectors for
easier implementation;
- chained querying switched from a list to a set to avoid
exponential size.
This allows us to use the full retry logic of rimraf in the `onexit` handler.
Note this is already covered by failing on Windows test
`should remove temp dir on process.exit`.
We skip waiting for "visible" state that enforces non-zero size.
Other invisible conditions like "display:none" fail during the
actual "scrolling" step and will retry.
Prior to this change, we were pre-maturely stopping a worker (since it
was deemed redundant under a race condition), and then we immediately
created a new worker with the same hash to finish off the test run. The
worker creation is expensive, so this slowed down the overall test run
time.
See the following for logs of the old code illustrating the extra stops and starts: https://gist.github.com/rwoll/1c592ed9e8f9169274fa972674de6703
feat: support experimental doc entries
- Params/options/members are marked as experimental in the docs.
- `experimental.d.ts` is generated that contains all types and
includes experimental features.
- `experimental.d.ts` is references in our tests so that we
can test experimental features.
- `fonts` option is restored as experimental.
- Always show a fixture that was running during timeout.
- Give custom titles to built-in fixtures.
- Specify setup/teardown fixture phase in the message.
- Split connect vs launch browser fixtures for better naming.
Example timeout message:
```log
Timeout of 2000ms exceeded while running fixture "built-in playwright configuration" teardown.
```
Reason: turns out Debian Buster requires just one source list to
install `ttf-ubuntu-font-family` font.
All other dependencies are satisfied.
Fixes#13530
Firefox has a bug: calling `node.focus()` does make the node focused,
but some internal "current contenteditable node" is not changed.
Blurring the previous one and focusing the new one helps.
To make launch/connect not affect the test timeout:
- `browser` fixture is declared with `timeout: 0`.
- Default timeouts are set to 30sec for launch and 3min for connect.
This was introduced in #7500 to fight `ERROR: The process "4436" not found.`
messages when killing a process that did already exit.
Since then, we no longer inherit stdout/stderr, so the error message
should not appear anymore.
This is a speculative fix to leftover tmp directories.
When users issues SIGINT twice, we enter `gracefullyClose()`
twice, and shortcut the second time. It turns out, we do
not wait for directories removal.
Note: it is unknown how often we reach this codepath in practice.
Previously, we were guessing whether something is optional from the
comment (looking for stuff like "defaults to" or "optional").
To remove this error-prone technique, every optional property or
argument (but not option) is explicitly marked with "?".
As a result, fix documentation issues:
- Removed `TestStep.data` that is not a thing.
- Added `TestConfig.name` documentation.
- Refined a lot of optional types in the documentation.
- Aligned `test.fail()` and `test.slow()` docs with `test.skip()`.
- Strict set of docs vs types inconsistensies in the generator
A few changes:
- `Matchers<R, T>` now carries both return and argument type.
- Based on the argument type, we apply playwright-specific Page/Locator matchers.
- Return type is usually void, unless wrapped with `expect.resolves`,
`expect.rejects` or `expect.poll()`.
- To preserve compatibility with any extended types in the wild,
argument type is optional.
fix: wait for worker to be ready before initializing
The test dispatcher waits until a worker is ready before sending the
`init` message. This guarantees that the worker process is listening for the
message. Otherwise, the worker process might miss the `init` message and
the subsequent `run` message would crash the worker.
If you typo'd an `expect` property, you got a cryptic error message:
```
Uncaught TypeError: Cannot create proxy with a non-object as target or handler
```
Now we get this nice friendly message:
```
1) a.spec.ts:6:9 › explodes ======================================================================
Error: expect: Property 'toBeLessThen' not found.
Did you mean 'toBeLessThan'?
See https://playwright.dev/docs/test-assertions for available options and documentation.
5 | const { test } = pwt;
6 | test('explodes', () => {
> 7 | expect.soft(1).toBeLessThen();
| ^
8 | });
9 |
```
Fixes#13218
- This supports `role=button[name=Hello]` similarly to CSS selectors.
- Does not change `_react` or `_vue` behavior that insist on quoting the string.
- Uses CSS notion of "identifier" characters.
This patch will check if browser channel is already installed.
If it is, it'll abort installation with the following error:
```
aslushnikov:~/prog/playwright$ npx playwright install msedge
Failed to install browsers
Error:
╔═════════════════════════════════════════════════════════════════╗
║ ATTENTION: "msedge" is already installed on the system! ║
║ ║
║ "msedge" installation is not hermetic; installing newer version ║
║ requires *removal* of a current installation first. ║
║ ║
║ To *uninstall* current version and re-install latest "msedge": ║
║ ║
║ - Close all running instances of "msedge", if any ║
║ - Use "--force" to install browser: ║
║ ║
║ npx playwright install --force msedge ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════════════════════════════╝
```
To re-install browser channel, use `--force`.
Fixes https://github.com/microsoft/playwright/issues/13061
It turns out that "non stalling evaluate" can stall in Chromium
in some weird conditions, like `document.open` after some weird
`iframe.src` value.
We now only hide highlight in those frames where we did install
highlight in the first place.
- Added docs to `selectors.md`.
- `[pressed]` and `[checked]` do not match `"mixed"` states.
- Disallow `[name]` shorthand without a value.
- Renamed `includeHidden` to `include-hidden`.
Previously, any unpaired quote in the text selector "escaped"
everything till the end of the selector string, and so any
subsequent chained selectors, including ">>" separator were ignored.
An example of misbehaving selector: `text=19" >> nth=1`.
Now, when text selector contains a non-leading quote, selector parser
does not assume it should escape ">>" separator and correctly
tokenizes all selectors from the chain.
Note that this behavior is a workaround for the fact that our
text selectors is somewhat poorly defined in this area. That said,
this workaround seems to be safe enough. It still does not work for
unpaired leading quotes like this: `text="19 >> nth=1`.
This has two values:
- `"hide"` to hide input caret for taking screenshot
- `"initial"` to keep caret behavior unchanged
Defaults to `"hide"`.
Fixes#12643
Previously, we always formed groups consisting of a single test.
Now, we group tests that share `beforeAll`/`afterAll` hooks into
`config.workers` equally-sized groups.
This introduces `role=button[name="Click me"][pressed]` attribute-style
role selector. It is only available under `env.PLAYWRIGHT_EXPERIMENTAL_FEATURES`.
Supported attributes:
- `role` is required, for example `role=button`;
- `name` is accessible name, supports matching operators and regular expressions:
`role=button[name=/Click(me)?/]`;
- `checked` boolean/mixed, for example `role=checkbox[checked=false]`;
- `selected` boolean, for example `role=option[selected]`;
- `expanded` boolean, for example `role=button[expanded=true]`;
- `disabled` boolean, for example `role=button[disabled]`;
- `level` number, for example `role=heading[level=3]`;
- `pressed` boolean/mixed, for example `role=button[pressed="mixed"]`;
- `includeHidden` - by default, only non-hidden elements are considered.
Passing `role=button[includeHidden]` matches hidden elements as well.
This patch prints a friendly instructions in case Docker image version
mismatches Playwright version and there are missing browser
dependencies.
With this patch, Playwright will yield the following error:
```
root@f0774d2b2097:~# node a.mjs
node:internal/process/promises:279
triggerUncaughtException(err, true /* fromPromise */);
^
browserType.launch:
╔════════════════════════════════════════════════════════════════════════════════════════════╗
║ Host system is missing dependencies to run browsers. ║
║ This is most likely due to docker image version not matching Playwright version: ║
║ - Playwright: 1.22.0 ║
║ - Docker: 1.21.0 ║
║ ║
║ Either: ║
║ - (recommended) use docker image "mcr.microsoft.com/playwright:v1.22.0-focal" ║
║ - (alternative 1) run the following command inside docker to install missing dependencies: ║
║ ║
║ npx playwright install-deps ║
║ ║
║ - (alternative 2) use Aptitude inside docker: ║
║ ║
║ apt-get install libgbm1 ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════════════════════╝
at file:///root/a.mjs:3:10 {
name: 'Error'
}```
Fixes#12796
Co-authored-by: Dmitry Gozman <dgozman@gmail.com>
This patch:
- starts using directory of `package.json` to resolve default
output directory path
- starts using either `package.json` directory or configuration
directory to resolve all relative paths
References #12970
This patch:
- Adds 3 new host platform types:
* `generic-linux` and `generic-linux-arm64` for the unsupported
linux distributions
* `<unknown>` for non-supported OS versions
- Prints a warning when downloading Ubuntu browser builds on
unsupported Linux distribution
- Makes sure launch doctor prints all missing shared libraries
on unknown Linux distributions
- Also prints an `apt` command as an alternative to Playwright CLI
dependency installation.
Supports inline regex in addition to string: `_react=BookItem[author = /Ann?a/i]`.
This is similar to `text=` selector, but applies to `_react` and `_vue`
selectors. In the future, will also apply to `role=` selector.
Resolves#11318.
* Adds `TestConfig.attachments` public API. (We opted to not implement an analog to the async `TestInfo.attach(…)` API.)
* Adds `TestConfig.attachments` to common reporters.
* Dogfoods some git and CI-info inference to generate useful atttachments
* Updates HTML Reporter to include a side bar to present a pre-defined set of attachments (a.k.a git/commit context sidebar)
Here's what it looks like:
<img width="1738" alt="Screen Shot 2022-03-21 at 3 23 28 PM" src="https://user-images.githubusercontent.com/11915034/159373291-8b937d30-fba3-472a-853a-766018f6b3e2.png">
See `tests/playwright-test/reporter-html.spec.ts` for an example of usage (for dogfood-ing only). In the future, if this becomes user-facing, there the Global Setup bit would likely become unnecessary (as would interaction with attachments array); there would likely just be a nice top-level config and/or CLI flag to enable collecting of info.
Previously, we preserved input/textarea values by providing
`value` attribute or text child. This produces DOM that does not
actually match the original page.
This change starts using special attributes to modify values
directly when rendering.
Same treatment is also applied to options in `select` and
`checked` property of checkboxes and radio buttons.
This patch aligns the strategies that are used to generate new
screnshot expectations and to compare screenshot expectations against
baseline.
With this patch, `toHaveScreenshot` will:
- when generating a new expectation: will wait for 2 consecutive
screenshots to match and accept the last one as expectation.
- when given an expectation:
* will compare first screenshot against expectation. If matches,
resolve successfully
* if first screenshot doesn't match, then wait for 2 consecutive
screenshots to match and then compare last screenshot with the
expectation.
An example of a new detailed call log:
```
1) a.spec.ts:3:1 › should work ===================================================================
Error: Screenshot comparison failed:
20000 pixels (ratio 0.03 of all image pixels) are different
Call log:
- expect.toHaveScreenshot with timeout 5000ms
- verifying given screenshot expectation
- fast-path: checking first screenshot to match expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts in all frames are loaded
- fast-path failed: first screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
- waiting for 2 consecutive screenshots to match
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts in all frames are loaded
- 2 consecutive screenshots matched
- final screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
- 20000 pixels (ratio 0.03 of all image pixels) are different
Expected: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-expected.png
Received: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-actual.png
Diff: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-diff.png
3 | test('should work', async ({ page }) => {
4 | await page.goto('file:///Users/andreylushnikov/prog/playwright/tests/assets/rotate-z.html');
> 5 | await expect(page).toHaveScreenshot();
| ^
6 | });
7 |
```
By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.
This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.
This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
Currently, running `npx playwright test` in a subfolder of your project
will result in a `test-results` folder created in your `cwd`.
This is unexpected; instead, we should always resolve all paths
against `rootDir` - directory that contains config.
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
- Line reporter now shows stats in addition to the test name:
```log
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.
- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.
- Updated tests to the golden style.
This patch adds support to multiple diffs. These are possible
due to soft assertions.
Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
* `fonts: "ready"`
* `animations: "disabled"`
* `size: "css"`
New default is based on the very (unscientific) output of repeated a few
times:
```
$ for i in `seq 5 13`; do time npm run ctest -- --reporter=line --workers=$i 'page/*' 'selector*' 'channels*' 'resource-timing*'; done
```
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)"
commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot (#12539)"
commit a3dff45974.
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.
The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.
For example, given the following config:
```js
// playwright.config.ts
module.exports = {
projects: [
{ name: 'Mobile Safari' },
{ name: 'Desktop Chrome' },
],
};
```
And the following test structure:
```
smoke-tests/
└── basic.spec.ts
```
Will result in the following screenshots folder structure by default:
```
__screenshots__/
└── darwin/
├── Mobile Safari/
│ └── smoke-tests/
│ └── basic.spec.ts/
│ └── screenshot-expectation.png
└── Desktop Chrome/
└── smoke-tests/
└── basic.spec.ts/
└── screenshot-expectation.png
```
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.
Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"
This reverts commit 73dee69558.
* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"
This reverts commit d744a87aee.
* Revert "chore(test runner): run hooks/modifiers as a part of the test (#12329)"
This reverts commit 47045ba48d.
It's a straightforward change to support new, common, keyboard commands
Note that I've tested this locally with Chrome on my Mac but it seems that CI doesn't want to pass Chrome tests - it's running on ubuntu though. Does this mean that I should introduce per-platform editing commands? At the moment there is only a single [`macEditingCommands`](0ed33522c5/packages/playwright-core/src/server/macEditingCommands.ts) file.
References https://github.com/microsoft/playwright/issues/12000
Co-authored-by: Andrey Lushnikov <aslushnikov@gmail.com>
chore(test runner): run hooks/modifiers as a part of the test
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
This uses `Module._resolveFilename` to intercept module resolution and
check `tsconfig.paths` similarly to pirates usage ot `Module._compile`.
Previously, we resolved during compilation that required reproducible
resolution due to caching. Now we can resolve as we go and support
all `tsconfig.paths`.
- `stdout.isTTY` controls whether list reporter updates lines or just adds them;
- `env.CI` is used in a few places to affect the defaults:
- whether to open interactive html;
- default reporter dot/line;
- default terminal reporter added to non-terminal reporters;
- `env.PWTEST_SKIP_TEST_OUTPUT` is removed;
- `env.PW_TEST_DEBUG_REPORTERS` is introduced specifically for tests.
This reverts commit e5c9d1e39f.
Reason for revert: turns out this fix results in a 5-second delay
when starting tests in docker, with `test-results` folder being
a non-removable mount.
The reason for the delay is the `maxBusyTries` option that we
supply by default to rimraf when trying to remove the folder.
While this option might come handy when removing temporary
browser profile folder, it doesn't serve us well in this particular
usecase.
References #12106
This way we control the timeout error message from the runner,
so that later on we can differentiate between test timeout, fixture
timeout and hook timeout.
This patch prepares for the `toHaveScreenshot` implementation
by splitting common parts from `toMatchSnapshot`.
Drive-by: fix default extension generation from `.bin` to `.dat`
for unknown buffers.
This patch adds additional options to `toMatchSnapshot` method:
- `pixelCount` - acceptable number of pixels that differ to still
consider images equal. Unset by default.
- `pixelRatio` - acceptable ratio of all image pixels (from 0 to 1) that differ to still
consider images equal. Unset by default.
Fixes#12167, #10219
In experimental ESM mode a child process is forked in order to run the tests. Currently the exit code of this child process is not propagated to the exit code of the parent process, which means that the process exits with a status code of `0` even if some of the tests failed.
This makes it difficult to use Playwright in CI in experimental mode, as the CI pipeline as a whole will pass despite the test failures.
This change addresses this by propagating the exit code in the case where it is non-zero.
This changes PlaywrigtServer to serve connections like `ws://localhost:3333/?browser=chromium`:
- launches the browser;
- talks `browserType.connect`-style protocol over websocket;
- compatible with `connectOptions` fixture.
```js
await playwright.chromium.connect({ wsEndpoint: 'ws://localhost:3333/?browser=chrome' });
```
The new (as of 1.18) `async testInfo.attach(…)` API handles this
gracefully (and is part of the reason for the new API's existence).
However, for the foreseeable future, it's still possible to manually
push onto the attachments array where we can't validate the contents
until it's too late, so this change ensures more graceful handling in
that case.
Fixes#11565
This option stops all kinds of CSS animations while doing screenshot:
- CSS animations
- CSS transitions
- Web Animations
Animations get different treatment depending on animation duration:
- finite animations are fast-forwarded to its end, issuing the
`transitionend` event.
- Infinite animations are resetted to its beginning, and then
resumed after the screenshot.
References #9938, fixes#11912
Soft expects will still fail the test, but will not abort it's execution. As a consequence of this:
- `TestResult` now might have multiple errors, which is reflected with a new `testResult.erros: TestError[]` field.
- `TestInfo` now might have multiple errors as well, which is reflected with a new `testInfo.errors: TestError[]` field.
Fixes#7819
This introduces `locator('div', { has: locator })` syntax that matches elements containing other elements.
Can be used together with `hasText`.
Internally, has selector engine takes an inner selector escaped with double-quotes:
`div >> has="li >> span >> text=Foo" >> span`.
When element that is being dragged stays under the mouse,
it prevents the hit target check on drop from working,
because drop target is overlayed by the dragged element.
To workaround this, we perform a one-time hit target check
before moving for the drop, as we used to.
These variables aren't CLI-only anymore, so pick some more general
names for them.
Note: all language ports would need to follow-up with the rename
after the next roll.
Fixes#11450
This reworks DeadlineRunner to use exception to signal timeout. This way,
we'll be able to run fixtures against a shared deadline vs their own
deadline and still get an easy control-flow timeout handling.
In several of the Playwright APIs, falsey values were not handled correctly. This changeset adds tests (and some fixes):
- route.continue: If options.postData was the empty string, the continue failed to override the post data.
- page.post (application/json with options.data: false|''|0|null): Raw falsey values were getting dropped (i.e. you can't do the equivalent of curl --header application/json … -d 'false'). This has been fixed with most values across all browsers, but an additional fix is needed for 'null' which the channel serializer treats extra specially.
- testInfo.attach: This didn't get reported as an error when options.path was the empty string, but should have been.
#11413 (and its fix#11414) inspired this search as they are the same
class of bug.
- Use file path, not content to calculate the attachment hash.
- Always cleanup fixture from the list on teardown, to avoid reporting
teardown error multiple times: from the test, and from the cleanup.
We do not have a timeout for any other close method, such as
browserContext.close or browser.close, and hitting default
30 seconds is very realistic with large Electron apps.
Previously, reporter would look for a stack frame directly in the test file.
Often times, that is not a top stack frame, especially when the test uses
some helper functions.
This changes error snippets and locations to use the top frame. When top
frame does not match the test file, we additionally show the location
to avoid confusion:
```
1) a.spec.ts:7:7 › foobar ========================================================================
Error: oh my
at helper.ts:5
3 |
4 | export function ohMy() {
> 5 | throw new Error('oh my');
| ^
6 | }
7 |
at ohMy (.../reporter-base-should-print-codeframe-from-a-helper/helper.ts:5:15)
at .../reporter-base-should-print-codeframe-from-a-helper/a.spec.ts:8:9
at FixtureRunner.resolveParametersAndRunHookOrTest (.../src/fixtures.ts:281:12)
```
In case of self-hosted github runners, it's much easier to checkout
under `playwright-internal` folder name instead of a default
`playwright` name. This confuses our stack collection.
This patch makes it generic.
This changes previous layout shift attempt (see #9546)
to account for more valid usecases:
- On the first event that is intercepted we enforce the hit target. This
is similar to the current mode that checks hit target before the action,
but is better timed.
- On subsequent events we assume that everything is fine. This covers more
scenarios like react rerender, glass pane on mousedown, detach on mouseup.
This check is enabled by default, with `process.env.PLAYWRIGHT_NO_LAYOUT_SHIFT_CHECK`
to opt out.
Textual snapshot diffs were previously broken in the HTML Report. The strikethrough'd text extended beyond the intended region.
HTML Report Before:
<img width="693" alt="Screen Shot 2021-12-27 at 4 43 35 PM" src="https://user-images.githubusercontent.com/11915034/147518750-a60f9002-6eed-48a1-a412-20fabd076fa6.png">
HTML Report After:
<img width="206" alt="Screen Shot 2021-12-27 at 4 48 37 PM" src="https://user-images.githubusercontent.com/11915034/147518762-19a4c8f9-ccc3-4a3c-a962-5a42edc6fc5d.png">
This now matches what's expected and shown in the terminal (which has always been correct):
<img width="1384" alt="Screen Shot 2021-12-27 at 4 36 29 PM" src="https://user-images.githubusercontent.com/11915034/147518799-f538259e-5a45-4d6f-916c-a12ccb620c5b.png">
NB: This MR is a workaround, but not a root cause fix. It works, but I never fully got to the root cause so a bug upstream may be required. It's unclear whether it's (1) in [`colors`](https://www.npmjs.com/package/colors), (2) in [`ansi-to-html`](https://www.npmjs.com/package/ansi-to-html), or (3) Playwright's use of the two. Since the terminal output is correct, I suspect it is in `ansi-to-html`. For example:
```js
const colors = require("colors");
const Convert = require('ansi-to-html');
const convert = new Convert();
// original (strike incorrectly wraps everything in the HTML)
console.log(convert.toHtml(colors.strikethrough("crossed out") + ' ' + colors.red("red")))
// prints: <strike>crossed out <span style="color:#A00">red<span style="color:#FFF"></span></span></strike>
// workaround
console.log(convert.toHtml(colors.reset(colors.strikethrough("crossed out")) + ' ' + colors.red("red")))
// prints: <strike>crossed out</strike> <span style="color:#A00">red<span style="color:#FFF"></span></span>
```
Fixes#11116
## Before
* When docker wasn't running: crashed with a TypeError
* When any image without `RepoTags` was present: crashed with TypeError
* Typo in recommended command ("playwight")
* No indication of `PW_TEST_IMAGE` env var in error output
<details>
<summary>Docker not running</summary>
```
Using config at ~redacted/playwright.config.ts
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused
TypeError: Cannot read property 'find' of null
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
test.ts:9:1 › some redacted description
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused
TypeError: Cannot read property 'find' of null
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js
```
</details>
<details>
<summary>Image without repo tags</summary>
```
Using config at ~redacted/playwright.config.ts
TypeError: Cannot read property 'includes' of undefined
at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
at Array.find (<anonymous>)
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
some-test.ts › some description
TypeError: Cannot read property 'includes' of undefined
at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
at Array.find (<anonymous>)
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element (26ms)
1) some-test.ts › some description =====
'Grid agent creation failed'
1 failed
some-test.ts › some description ======
```
</details>
## After
* Helpful error message if docker isn't running
* Doesn't crash when local-only images in list
* No typo in `playwright install docker-image` command
* When other playwright images are found they are listed and `PW_TEST_IMAGE` is mentioned
### After: Docker not running
```
Using config at ~redacted/playwright.config.ts
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images ║
║ Please ensure docker daemon is running. ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
test.ts:9:1 › description
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images ║
║ Please ensure docker daemon is running. ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ test.ts:9:1 › description
1) test.ts:9:1 › description =====
'Grid agent creation failed'
1 failed
test.ts:9:1 › description ======
```
## After: No exact match found
```
Using config at ~redacted/playwright.config.ts
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image. ║
║ ║
║ Available images: ║
║ - mcr.microsoft.com/playwright:v1.16.3 ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal ║
║ ║
║ Use available images via PWTEST_IMAGE_NAME environment variable: ║
║ PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║ ║
║ Alternatively, please pull docker image with the following command: ║
║ ║
║ npx playwright install docker-image ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image. ║
║ ║
║ Available images: ║
║ - mcr.microsoft.com/playwright:v1.16.3 ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal ║
║ ║
║ Use available images via PWTEST_IMAGE_NAME environment variable: ║
║ PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║ ║
║ Alternatively, please pull docker image with the following command: ║
║ ║
║ npx playwright install docker-image ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ test.ts:9:1 › description
1) test.ts:9:1 › description =====
'Grid agent creation failed'
1 failed
test.ts:9:1 › description ======
```
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
