Fixes#16456.
The extra span/name was coming from
077b8a9289/packages/html-reporter/src/links.tsx (L72)
(in conjunction with the line above it).
These two lines assume `path` and `body` are mutally exclusive.
Regression likely introduced by the PR that added the now removed line #10778.
This patch changes `expect(locator).toHaveAttribute()` so that the
`value` argument can be omitted. When done so, the method will
assert attribute existance.
Fixes#16517
Also, exlude certain errors from triggering this message:
- `no tests found`
- `duplicate test titles are not allowed`
- `--forbid-only found a focused test`
- `Timed out waiting 3600s for the entire test run`
In the following example, `locale` inside the `describe`
would be reverted to the default value:
```js
test.use({ locale: 'en-GB' });
test.decsribe(() => {
test.use({ locale: undefined });
});
```
A small change in the API requires `shortCircuit: true` when returning the
transformed source.
Another API change removes third argument from the chained resolve/load,
but passing three instead of two still works.
Line reporter now shows stats in addition to the test name:
```
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
[21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
- Two lines per test: title and location.
- Align project labels.
- Add trace badge that opens trace viewer.
- Add video and image diff badges that show scrolled test result view.
fix(test runner): more firendly test duration
Reported test duration now does not include time spent in
`beforeAll`, `afterAll` and fixtures that have a separate timeout.
This is to avoid different reported test execution time,
depending on the test execution order.
Fixes#8206.
Since #8206 is a long-awaited (~ 1 year old), popular (~ 45 reactions, frequently requested in community channels, etc.), this PR aims to unblock folks.
Notably, we do not innovate on the `webServer` API, despite knowing we're not in love with it. We'll save the innovation for either Plugins or a new `LaunchConfigs` option. (We haven't yet arrived at a Plugin API we like, and instead of launching a new option guessing what the "better" launchConfig API would be, let's wait and see how folks use this new Array-variant of `webServer` which—despite its name—can be used for non-Web Server launches!
When running without tty, line reporter outputs a line for each
percent of the tests, thus limiting the output to ~100 lines.
In addition, reporters now support PLAYWRIGHT_LIVE_TERMINAL
env variable to force tty mode.
Useful to set some options without coming up with an artificial name.
```js
test.describe(() => {
test.use({ colorScheme: 'dark' });
test('my test', () => {});
})
```
Currently, it is possible to run a function, e.g. a second `beforeEach` hook,
that will receive `null` for the fixture that has already failed before.
This PR skips running any user function that uses a fixture that has already
failed, just like if the fixture would be initialized again and failing for
the second time.
Currently, if `text.fixme()` or `test.skip()` is used within a test, we
add a `fixme` or `skip` annotation. However, if the wrapper style is
used:
```
test.fixme('should work', () => {…})
```
the annotations were missing. This change adds annotations for the
above.
These annotations are important for reporting purposes and knowing
exactly what flavor of "skipped" was used.
Fixes#15239.
Some of the current timeout error messages are confusing, because they do not suggest that the issue is most likely a slow test. This PR updates timeout messages as follows:
- Test timeout of 30000ms exceeded.
- Test timeout of 30000ms exceeded while setting up "browser".
- Test timeout of 30000ms exceeded while tearing down "context".
- Test timeout of 30000ms exceeded while setting up "playwright configuration".
- Test timeout of 30000ms exceeded while running "beforeEach" hook.
- Test timeout of 30000ms exceeded while running "afterEach" hook.
- "beforeAll" hook timeout of 30000ms exceeded.
- "afterAll" hook timeout of 30000ms exceeded.
- Worker teardown timeout of 30000ms exceeded.
- "skip" modifier timeout of 30000ms exceeded.
- Fixture "myCustomFixture" timeout of 5000ms exceeded.
Previously, screenshot was taken after hooks and fixtures teardown.
However, hooks can easily modify the state of the page, and
screenshot would not reflect the moment of failure.
Instead, we take screenshots immediately after the test function
finishes with an error.
These mean "I don't want to specify this fixture/option"
instead of "I want the value of undefined", aligned with how TypeScript works.
We already do similar things in the config.
Fixes#14904.
This is done to make looking at the raw contents of the report
friendlier when using a file browser. However, it should be noted, the
public API of the HTML Reporter makes no guarantees of its contents
structure/layout/naming-conventions.
Adds cross-browser support for easily allowing/blocking Service Workers via a Context option.
Includes plumbing for Playwright Test's `use`.
Resolves#14522.
Relates #1090.
Supercedes #14321.
If the tests are in an explicitly configured testDir (either at the global config level or per project) .gitignore filters are not applied.
Fixes#14381
There are a few issues this covers:
- Some fixtures like `page` and `context` are not allowed in `beforeAll`
hooks, so using them in automatic fixture makes it throw.
- Running automatic fixture solely for `afterAll` is unexpected.
This currently happens when `afterAll` is run for cleanup after
fixture timeout/throw.
For built-in playwright fixture, we keep `'all-hooks-included'` auto mode.
Added a doc explaining the execution order.
Note: all toHaveScreenshot tests still use `__screenshots__` directory
for their expectations. One more test was added to make sure that
by default, `toHaveScreenshot` uses old snapshots.
Prior to this change, we were pre-maturely stopping a worker (since it
was deemed redundant under a race condition), and then we immediately
created a new worker with the same hash to finish off the test run. The
worker creation is expensive, so this slowed down the overall test run
time.
See the following for logs of the old code illustrating the extra stops and starts: https://gist.github.com/rwoll/1c592ed9e8f9169274fa972674de6703
feat: support experimental doc entries
- Params/options/members are marked as experimental in the docs.
- `experimental.d.ts` is generated that contains all types and
includes experimental features.
- `experimental.d.ts` is references in our tests so that we
can test experimental features.
- `fonts` option is restored as experimental.
- Always show a fixture that was running during timeout.
- Give custom titles to built-in fixtures.
- Specify setup/teardown fixture phase in the message.
- Split connect vs launch browser fixtures for better naming.
Example timeout message:
```log
Timeout of 2000ms exceeded while running fixture "built-in playwright configuration" teardown.
```
To make launch/connect not affect the test timeout:
- `browser` fixture is declared with `timeout: 0`.
- Default timeouts are set to 30sec for launch and 3min for connect.
Previously, we were guessing whether something is optional from the
comment (looking for stuff like "defaults to" or "optional").
To remove this error-prone technique, every optional property or
argument (but not option) is explicitly marked with "?".
As a result, fix documentation issues:
- Removed `TestStep.data` that is not a thing.
- Added `TestConfig.name` documentation.
- Refined a lot of optional types in the documentation.
- Aligned `test.fail()` and `test.slow()` docs with `test.skip()`.
- Strict set of docs vs types inconsistensies in the generator
A few changes:
- `Matchers<R, T>` now carries both return and argument type.
- Based on the argument type, we apply playwright-specific Page/Locator matchers.
- Return type is usually void, unless wrapped with `expect.resolves`,
`expect.rejects` or `expect.poll()`.
- To preserve compatibility with any extended types in the wild,
argument type is optional.
fix: wait for worker to be ready before initializing
The test dispatcher waits until a worker is ready before sending the
`init` message. This guarantees that the worker process is listening for the
message. Otherwise, the worker process might miss the `init` message and
the subsequent `run` message would crash the worker.
If you typo'd an `expect` property, you got a cryptic error message:
```
Uncaught TypeError: Cannot create proxy with a non-object as target or handler
```
Now we get this nice friendly message:
```
1) a.spec.ts:6:9 › explodes ======================================================================
Error: expect: Property 'toBeLessThen' not found.
Did you mean 'toBeLessThan'?
See https://playwright.dev/docs/test-assertions for available options and documentation.
5 | const { test } = pwt;
6 | test('explodes', () => {
> 7 | expect.soft(1).toBeLessThen();
| ^
8 | });
9 |
```
Fixes#13218
This has two values:
- `"hide"` to hide input caret for taking screenshot
- `"initial"` to keep caret behavior unchanged
Defaults to `"hide"`.
Fixes#12643
Previously, we always formed groups consisting of a single test.
Now, we group tests that share `beforeAll`/`afterAll` hooks into
`config.workers` equally-sized groups.
This patch:
- starts using directory of `package.json` to resolve default
output directory path
- starts using either `package.json` directory or configuration
directory to resolve all relative paths
References #12970
Resolves#11318.
* Adds `TestConfig.attachments` public API. (We opted to not implement an analog to the async `TestInfo.attach(…)` API.)
* Adds `TestConfig.attachments` to common reporters.
* Dogfoods some git and CI-info inference to generate useful atttachments
* Updates HTML Reporter to include a side bar to present a pre-defined set of attachments (a.k.a git/commit context sidebar)
Here's what it looks like:
<img width="1738" alt="Screen Shot 2022-03-21 at 3 23 28 PM" src="https://user-images.githubusercontent.com/11915034/159373291-8b937d30-fba3-472a-853a-766018f6b3e2.png">
See `tests/playwright-test/reporter-html.spec.ts` for an example of usage (for dogfood-ing only). In the future, if this becomes user-facing, there the Global Setup bit would likely become unnecessary (as would interaction with attachments array); there would likely just be a nice top-level config and/or CLI flag to enable collecting of info.
By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.
This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.
This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
Currently, running `npx playwright test` in a subfolder of your project
will result in a `test-results` folder created in your `cwd`.
This is unexpected; instead, we should always resolve all paths
against `rootDir` - directory that contains config.
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
- Line reporter now shows stats in addition to the test name:
```log
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.
- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.
- Updated tests to the golden style.
This patch adds support to multiple diffs. These are possible
due to soft assertions.
Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
* `fonts: "ready"`
* `animations: "disabled"`
* `size: "css"`
New default is based on the very (unscientific) output of repeated a few
times:
```
$ for i in `seq 5 13`; do time npm run ctest -- --reporter=line --workers=$i 'page/*' 'selector*' 'channels*' 'resource-timing*'; done
```
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)"
commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot (#12539)"
commit a3dff45974.
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.
The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.
For example, given the following config:
```js
// playwright.config.ts
module.exports = {
projects: [
{ name: 'Mobile Safari' },
{ name: 'Desktop Chrome' },
],
};
```
And the following test structure:
```
smoke-tests/
└── basic.spec.ts
```
Will result in the following screenshots folder structure by default:
```
__screenshots__/
└── darwin/
├── Mobile Safari/
│ └── smoke-tests/
│ └── basic.spec.ts/
│ └── screenshot-expectation.png
└── Desktop Chrome/
└── smoke-tests/
└── basic.spec.ts/
└── screenshot-expectation.png
```
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.
Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"
This reverts commit 73dee69558.
* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"
This reverts commit d744a87aee.
* Revert "chore(test runner): run hooks/modifiers as a part of the test (#12329)"
This reverts commit 47045ba48d.
chore(test runner): run hooks/modifiers as a part of the test
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
This uses `Module._resolveFilename` to intercept module resolution and
check `tsconfig.paths` similarly to pirates usage ot `Module._compile`.
Previously, we resolved during compilation that required reproducible
resolution due to caching. Now we can resolve as we go and support
all `tsconfig.paths`.
- `stdout.isTTY` controls whether list reporter updates lines or just adds them;
- `env.CI` is used in a few places to affect the defaults:
- whether to open interactive html;
- default reporter dot/line;
- default terminal reporter added to non-terminal reporters;
- `env.PWTEST_SKIP_TEST_OUTPUT` is removed;
- `env.PW_TEST_DEBUG_REPORTERS` is introduced specifically for tests.
This reverts commit e5c9d1e39f.
Reason for revert: turns out this fix results in a 5-second delay
when starting tests in docker, with `test-results` folder being
a non-removable mount.
The reason for the delay is the `maxBusyTries` option that we
supply by default to rimraf when trying to remove the folder.
While this option might come handy when removing temporary
browser profile folder, it doesn't serve us well in this particular
usecase.
References #12106
