When two fixtures have different time slots, timeout in the first one
should not prevent the second one from tearing down.
Similarly, timeout in afterEach hook should not prevent fixture
teardowns.
This allows any time slot that has a legitimate timeout of zero to be
updated later on. See test for an example.
Previously, setting timeout to zero at any moment was considered a
"debug mode" and any subsequent timeouts were ignored.
Closes https://github.com/microsoft/playwright/issues/32070. We were
applying `additionalFileMatcher` not just to `filteredProjectSuites`,
but also to `projectSuites`. `projectSuites` is where we take dependency
projects from, though - so `--only-changed` led to empty dependency
projects, resulting in the reported bug.
The fix is to only apply `additionalFileMatcher` on
`filteredProjectSuites`.
Supercedes https://github.com/microsoft/playwright/pull/31915, closes
https://github.com/microsoft/playwright/issues/31811.
When TypeScript resolves a specifier via path mapping, it does not
interpret `package.json`. If path mapping resolves to a directory, it
only looks at the `index.js` file in that directory if it's in CommonJS
mode.
We need to mirror this in our `esmLoader.ts`.
---------
Co-authored-by: Dmitry Gozman <dgozman@gmail.com>
Introduce `--tsconfig` to specify a single config to be used for all
imported files, instead of looking up tsconfig for each file separately.
Fixes#12829.
This allows a dynamic import of a TS file to be processed by Babel.
For some reason, Playwright used to revert the CJS transforms. However,
ESM loader and transforms are always active, so CJS should be too.
When two attachments have the same content sha1, we used the first one's
name for the downloaded file, no matter which one the user clicked to
download. Now we pass the name explicitly.
References #31912.
Closes https://github.com/microsoft/playwright/issues/22211
Currently, when the server notifies the UI about changed files, the UI
determines what files to re-run based on an old test list. By listing
tests before that, we make sure that the test list is up-to-date, and
that added tests are included in the next run.
I've also removed the `listChanged` event as discussed in the team sync.
The event isn't used anywhere and fires in exactly the same cases where
`testFilesChanged` fired, so i've folded them into one another. This allowed simplifying `Watcher`.
- Update copy to clipboard button.
- Reveal test source in the Source tab instead of external editor.
- New button to reveal in the external editor in the Source tab.
- Move the Pick Locator button next to snapshot tabs.
Introduces an `--only-changed [base ref]` option.
`playwright test --only-changed` filters the test run to only run test
suites that have uncommitted changes.
`playwright test --only-changed=foo` runs only tests that were changed
since commit `foo`.
In pull request CI, this can be used to run changed tests first and fail
fast: `--only-changed=$GITHUB_BASE_REF`.
During local development, it can be used to quickly filter down to the
touched set of tests suites.
In some rare usecases, this can also help to cut down on CI usage for
pull requests. Tread with caution though.
File dependencies are taken into account to ensure that if you touched a
utility file, all relevant tests are still executed.
Closes https://github.com/microsoft/playwright/issues/15075
Broken out of https://github.com/microsoft/playwright/pull/31727 as per
@dgozman's
[request](https://github.com/microsoft/playwright/pull/31727#discussion_r1685793229).
The PR goal is to remove the `suite` argument from the Component
testing's Vite Plugin. `suite` is used to enrich Vite's dependency graph
with information about dependencies between test suites and helper
files. It essentially merges the Vite graph with the
`compilationCache.ts > fileDependencies` graph, and then writes the
result back into `compilationCache.ts > externalDependencies`.
By refactoring this to make the connection on the reading end in
`collectAffectedTestFiles`, we can drop the `suite` parameter.
We didn't yet have a test that depended on the dependency graph being
connected correctly between `fileDependencies` and
`externalDepedencies`, so I've [extended an existing
test](53a539938b)
to capture that.
Previously, only the "actual" attachment was created, pointing to the
file in `test-results`. Now, the "expected" attachment pointing to the
file in `__screenshots__` is also created. This will help any reporters
that would like to know the "expected" path, for example to do a manual
accept/decline of the baseline.
Fixes#30693.
... unless an array of file-system-friendly parts is provided.
Motivation: attachment name is used as a file system path when
downloading attachments, so we keep them fs-friendly.
References #30693.
When used in a terminal, the `list` reporter prints out information
about test steps to help debugging. In non-TTY environments like GitHub
Actions, currently it doesn't.
This PR changes that, so that in non-TTY environments you'll see the
"step end" messages appearing, but not the "step begin" messages. This
is a good middleground, because it helps the user understand test
progress, without being too verbose.
Closes https://github.com/microsoft/playwright/issues/31674
Retaining traces in the following scenarios:
- browser crash;
- manual `browser.close()`;
- implicit `browser.close()` from the `browser` fixture upon test end.
This does not affect the library, where `browser.close()` will not
retain the trace and will close the browser as fast as possible.
References #31541, #31535, #31537.
Historically, this plugin was important to translate dynamic imports
into require calls so that we can intercept them and transpile.
This is not needed anymore with ESM loader enabled by default, so we can
avoid this transformation and support dynamic imports of ESM-only
packages/files.
Fixes#17075, fixes#23255, fixes#31140, references #23662.
When merging blob reports test ids are patched to make sure there is no
collision when merging reports that might have overlapping test ids.
However, even if you were merging reports that had no overlapping ids,
all test ids will be modified, which is an undesirable side effect.
This PR only modify test ids when the same test id has already been used
in a previous blob report.
----
This change is also part of
https://github.com/microsoft/playwright/pull/30962
The spirit of this change is reverting #23153. Since that time, we have
moved tracing and `artifactsDir` lifetime into the test runner, so the
reason for revert is mitigated.
Fixes#30287, fixes#30718, fixes#30959.
This includes two major changes:
- reuse `SerializedFS` for live test runner tracing;
- merge scheduled `appendFile` operations into a single `fs` call.
In some cases, this improves performance of UI mode by 61% and
performance of `trace: on` mode by 38%. Note that performance
improvement on the average test will not be as noticeable.
References #30875, #30635.
- Documents `PLAYWRIGHT_FORCE_TTY` and `FORCE_COLOR` across terminal
reporters.
- New `PLAYWRIGHT_LIST_PRINT_STEPS`. Removes undocumented test-only
`PW_TEST_DEBUG_REPORTERS_PRINT_STEPS`.
- Replaces `PLAYWRIGHT_HTML_REPORT` with `PLAYWRIGHT_HTML_OUTPUT_DIR`
and `PW_TEST_HTML_REPORT_OPEN` with `PLAYWRIGHT_HTML_OPEN` for
consistency, supports older versions for backwards compatibility.
- New `PLAYWRIGHT_HTML_HOST`, `PLAYWRIGHT_HTML_PORT` and
`PLAYWRIGHT_HTML_ATTACHMENTS_BASE_URL`.
- New `PLAYWRIGHT_JUNIT_STRIP_ANSI` and
`PLAYWRIGHT_JUNIT_INCLUDE_PROJECT_IN_TEST_NAME`.
- Removes `PW_HTML_REPORT` that was set for unknown reason.
Previously, terminal reporters consulted `process.stdout.isTTY`. Now it
is possible to control the tty behavior:
- `PLAYWRIGHT_FORCE_TTY=0` or `PLAYWRIGHT_FORCE_TTY=false` to disable
TTY;
- `PLAYWRIGHT_FORCE_TTY=1` or `PLAYWRIGHT_FORCE_TTY=true` to enable TTY,
defaults to 100 columns when real columns are unavailable;
- `PLAYWRIGHT_FORCE_TTY=<number>` to force enable TTY and set the
columns.
Fixes#29422.
Previously, terminating worker always had a 30 seconds force exit.
Now, regular worker termination assumes that process will eventually
finish tearing down all the fixtures and exits. However, the
self-destruction routine keeps the 30 seconds timeout to avoid zombies.
Fixes#30504.
Implements feature requested in
https://github.com/microsoft/playwright/issues/30457
The test runner treats flaky tests as failures when the flag is enabled,
but still reports flaky tests as flaky in the reporting interface. It
feels like something worth discussing as this behaviour makes sense to
me, but looked a bit odd to @BJSS-russell-pollock when I ran this past
him.
Closes#30457.
Matching bu `apiName@wallTime` fails when two actions start at the same
time, e.g. two parallel api requests. Moreover, it results in trace
actions that have parent set to themselves, which in turn causes
infinite loop in the trace viewer. To avoid this problems we write
stepId explicitly to the library trace and use those step ids to find
corresponding test runner steps.
The stepId is passed via zone in case of expect, because the protocol
step is quite deep in the call chain after or explicitly in case of API
call steps created by the test runner instrumentation.
Pre Node.js `18.20.0`:
- `assert` is supported
Post Node.js `18.20.0`
- `assert` and `with` is supported.
Before https://github.com/microsoft/playwright/pull/30482 we kept
`asserts` in the JS code, Node.js was interpreting them. The `with`
keyword was not supported, this was what the PR was fixing.
After https://github.com/microsoft/playwright/pull/30482 Babel is
converting `assert` (deprecated) into `with` (successor) since we use
the `deprecatedAssertSyntax` option.
This means, that the minimum Node.js version we support in order to use
import attributes is now `18.20.0` where they added the `with` support.
This follows our principle of supporting only the latest minor release
for Node.js versions.
See here for the 18.20 changelog:
> #### Added support for import attributes
>
> Support has been added for import attributes, to replace the old
import
> assertions syntax. This will aid migration by making the new syntax
available
> across all currently supported Node.js release lines.
>
> This adds the `with` keyword which should be used in place of the
previous
> `assert` keyword, which will be removed in a future semver-major
Node.js
> release.
>
> For example,
>
> ```console
> import "foo" assert { ... }
> ```
>
> should be replaced with
>
> ```console
> import "foo" with { ... }
> ```
Fixes https://github.com/microsoft/playwright/pull/30482 - the tests
were a noop before, since they were tree-shaked by Babel.
Similarly to page.close, we pass test-runner specific reason to
facilitate better error messages.
```
1) a.test.ts:10:11 › test
Error: apiRequestContext.fetch: Fixture { request } from beforeAll cannot be reused in a test.
- Recommended fix: use a separate { request } in the test.
- Alternatively, manually create APIRequestContext in beforeAll and dispose it in afterAll.
See https://playwright.dev/docs/api-testing#sending-api-requests-from-ui-tests for more details.
9 |
10 | test('test', async () => {
> 11 | await context.fetch('http://example.com');
| ^
12 | });
13 |
```
Closes#29260.
When comparing `outputDir` and html-reporter `outputFolder`, we now make
sure that both paths end with a forward-slash.
Fixes#28677
---------
Co-authored-by: Georg Unterholzner <georg.unterholzner@dynatrace.com>
There are plenty of edge cases in this area:
- interrupted test run;
- did not run because of serial mode failure;
- failed before `test.skip()` call (e.g. in `beforeEach`) in one of the
retries;
- and more...
Related issues: #28322, #28321, #27455, #17652.
Prior changes: #27762, #26385, #28360, probably more.
There is still some duplication between `outcome()` and similar logic in
`base.ts`, which might be deduped in a follow-up.
Fixes#28322.
Generate 'global timeout' and 'max failures' errors in the runner. Avoid
reading `config.globalTimeout` and `config.maxFailures` in the base
reporters.
Reference https://github.com/microsoft/playwright/issues/29768
Instead of filtering tests assuming there are no two projects with same
name we always rebuild test tree from scratch and restore previos test
results in the list mode.
Fixes https://github.com/microsoft/playwright/issues/30396
* FullConfig split into FullConfig and ConfigInWorker
* FullProject split into FullProject and ProjectInWorker
* Tests can only access {Config, Project}InWorker, while reporters can
only access Full{Config,Project}. This will allow in the future to
change the reporter API independently from the tests.
* Added documentation for the 4 classes.
Fixes#30040
# Context
In the `playwright.config` file, we can custom the default `timeout`
property for `expect.toPass` by the property:
```ts title='playwright.config.ts'
export default defineConfig({
// ...
expect: {
toPass: {
timeout: 60_000,
},
},
});
```
However, we can't customize the `intervals` option. So the default value
is always `[100, 250, 500, 1000]`.
# Goal
Add the possibility to customize the intervals option from the
`playwright.config` file.
```ts title='playwright.config.ts'
export default defineConfig({
// ...
expect: {
toPass: {
timeout: 60_000,
intervals: [1000, 2000, 3000]
},
},
});
```
* Add `Suite.entries` that returns tests and suites in their declaration
order
* Exposed `Suite.type` and `TestCase.type` for discriminating between
different entry types.
* Blob report format is updated to store entries instead of separate
lists for suites and tests.
* Bumped blob format version to 2, added modernizer.
Fixes https://github.com/microsoft/playwright/issues/29984
This brings up the question that we would show the tag name twice if its
a tag in a title. This would be aligned to how HTML report is doing it.
Fixes https://github.com/microsoft/playwright/issues/29927
---------
Signed-off-by: Max Schmitt <max@schmitt.mx>
Co-authored-by: Dmitry Gozman <dgozman@gmail.com>
With the recent change that gave after hooks a separate timeout, fixture
teardown does not imply that "test finished successfully, but fixture
teardown was slow".
This was historically done to make `console.log()` have colors. However,
this makes any other code that checks `process.stdout.isTTY` incorrectly
assume real TTY support.
Node18 and Node20 now respect `FORCE_COLOR=1` in console, so our default
behavior of forcing colors in the worker process just works out of the
box. See https://github.com/nodejs/node/pull/48034.
Previously, there was at most one "hard error", as opposite to multiple
"soft errors". This was done to preserve the historic behavior at the
time of introducing multiple `TestInfo.errors`.
With this change, every user callback that is executed `withRunnable()`
can throw an error and/or timeout, and both of these will end up in
`TestInfo.errors`.
Additionally, there is at most one "unhandled exception" error, to avoid
flooding the report with mass failures.
Drive-by: remove boolean arguments from `_failWithError()`.
Fixes#29876.
When collecting dependencies both from CJS loader and from ESM loader,
the latter would overwrite the dependencies set instead of appending.
Also make sure cts/cjs/mts/mjs are all supported equally.
References #29747.
Test-specific output dir and snapshot names are trimmed to 60 chars
instead of just 100. The snapshot names are still trimmed at 100 chars
for backwards compatibility.
Reference https://github.com/microsoft/playwright/issues/29719
This fixes a case where we incorrectly used the final config's base path
when resolving relative path mappings in the absence of the baseUrl.
Fixes#29816.
This will be useful to run `beforeAll`/`afterAll` hooks with a separate
`TestInfo` instance, as well as run use helpers like
`_runAndFailOnError()` during scope teardown.
This fixes some edge cases where fixtures and steps inside them were
attached to the wrong parent (see the new test).
This will also allow to replace some `runAsStep` calls with a flat list
of tasks to do that do not rely on lexical scope.
API changes:
- `test(title, details, body)` where details contain `tag` and
`annotation`.
- similar `details` property added to `test.skip`, `test.fail`,
`test.fixme`, `test.only`, `test.describe` and other `test.describe.*`
variations.
- `TestProject.tagFilter`/`TestConfig.tagFilter` that supports logical
tag expressions with `(`, `)`, `and`, `or` and `not`.
- `--tag` CLI option to filter by tags.
- New annotations are available in `TestInfo.annotations` and
`TestCase.annotations`.
- New tags are available in `TestCase.tags`.
Reporter changes:
- `json` reporter includes new tags in addition to old `@smoke`-style
tags. **Breaking**: tags are now listed with the leading `@` symbol.
- `html` reporter filters by old and new tags with the same `@smoke`
token.
Fixes#29229, fixes#23180.
- Modifiers that only depend on the worker fixtures are implemented as
`beforeAll` hooks.
- Modifiers that depend on test fixtures are implemented as `beforeEach`
hooks.
- Pushed `_runAndFailOnError` down the stack, wrapping individual hooks
instead of the whole "before hooks" section.
- Reused the same code to run `beforeAll` and `afterAll` hooks and
modifiers.
**Behavior change**: `test.skip()` inside a `beforeAll` now skips the
hook and all tests in the suite.
When `updateSnapshots === 'missing'` we generate new expectations on the
first attempt and don't retry the test afterwards instead of trying it
retries-1 times and only writing new expectation on the last attempt.
This logic infects all serial mode suites that contain the test with
missing expectations, so they also will not be retried.
Reference https://github.com/microsoft/playwright/issues/29073
Otherwise, merging two configs without `projects` property will create a
config with an empty project list, which is semantically different and
always leads to "No tests found".
- remove `onlyStartedTests` in favor of explicit branch with comments;
- produce one "test not found" error per test instead of a single large
error;
- extract `_failTestWithErrors` from `_massSkipTestsFromRemaining`.
Before this fix, unhandled error during test.fail():
- marks this test as "interrupted";
- fails next test in the file with "fatal error".
After this fix:
- marks this test as "failed as expected";
- restarts worker for the next test.
