Close https://github.com/microsoft/playwright/issues/19803
I had to manually inject `window.playwrightReportBase64` data into
`playwright/packages/html-reporter/index.html` when developing,
wondering if there's any method I don't know.
Introduce config.globalScripts. Tests from the matching files will run
before all projects. We'll only allow beforeAll/afterAll instead of
tests in such files (next PR).
Global scripts are executed as part of 'Global Scripts' project which is
not present in FullConfig.projects but may be referenced by
corresponding global setup Suites.
Signed-off-by: Yury Semikhatsky <yurys@chromium.org>
Co-authored-by: Dmitry Gozman <dgozman@gmail.com>
* Changed `setup` to `test.projectSetup`
* Only `test.projectSetup.only` is supported on the new method
* test.* methods except for before/after/Each/All hooks can be called
inside the project setup files
#19576 introduced a regression where the CLI reporters displayed some
times with way too many decimals. e.g. 7.123456789ms.
Prior to #19576, there were two monotonicTime implementations; that PR
updated the reporters to use the common definition that had existed in
utils.ts. However, that introduced a regression in the base.ts reporters
which used the ms duration humanizing package which did not account for
the more precise decimals used by the shared monotonicTime function.
This fix removes the dependency on the third-party ms package and now
consistently uses Pavel's humanize function which the HTML reporter had
been using and proved to have better defaults for decimals.
Additionally, we add more test coverage to limit future regressions
since this was caught in passing.
Closes#19556.
Relates #19576.
Note: this keeps existing behavior of `undefined` in `test.use()`
reverting to the config value and not to the original default value.
References #19615.
This patch implements a new image comparison function, codenamed
"ssim-cie94". The goal of the new comparison function is to cancel out
browser non-determenistic rendering.
To use the new comparison function:
```ts
await expect(page).toHaveScreenshot({
comparator: 'ssim-cie94',
});
```
As of Nov 30, 2022, we identified the following sources of
non-determenistic rendering for Chromium:
- Anti-aliasing for certain shapes might be different due to the
way skia rasterizes certain shapes.
- Color blending might be different on `x86` and `aarch64`
architectures.
The new function employs a few heuristics to fight these
differences.
Consider two non-equal image pixels `(r1, g1, b1)` and `(r2, g2, b2)`:
1. If the [CIE94] metric is less then 1.0, then we consider these pixels
**EQUAL**. (The value `1.0` is the [just-noticeable difference] for
[CIE94].). Otherwise, proceed to next step.
1. If all the 8 neighbors of the first pixel match its color, or
if the 8 neighbors of the second pixel match its color, then these
pixels are **DIFFERENT**. (In case of anti-aliasing, some of the
direct neighbors have to be blended up or down.) Otherwise, proceed
to next step.
1. If SSIM in some locality around the different pixels is more than
0.99, then consider this pixels to be **EQUAL**. Otherwise, mark them
as **DIFFERENT**. (Local SSIM for anti-aliased pixels turns out to be
very close to 1.0).
[CIE94]: https://en.wikipedia.org/wiki/Color_difference#CIE94
[just-noticeable difference]:
https://en.wikipedia.org/wiki/Just-noticeable_difference
The behavior regarding filters (both in config, command line and .only)
is the following:
- if some of tests match and none of setup match then we'll run all
setup files and all matching tests
- otherwise the filters apply to setup files the same way as to regular
tests
Running `npx playwright test file:123` will have the following behavior
- if only test files match then only matching subset of tests will run
but all setup files will run as well
- if only setup files match the filter then only those setup tests will
run
- if both setup and test files match an error will be thrown
This patch:
- updates documentation to lead users from `TestConfig.snapshotDir` and
`testInfo.snapshotSuffix` to `TestConfig.snapshotPathTemplate` as a
better and more flexible alternative.
- drops `{snapshotSuffix}` from documentation
- stops using `snapshotSuffix = ''` in our own tests and switches us
to the `snapshotPathTemplate`.
- adds `{testName}` token.
- Use `snapshotPathTemplate` for docker screenshots in html-reporter
- Mark the snapshot path template test as slow since it re-spawns
worker for each project.
- Fix docker smoke tests
This configuration option allows to set a string with template
values for precise control over snapshot path location.
An example of `snapshotPathTemplate` usage:
```ts
// playwright.config.ts
// Notice the `testDir` configuration!
export default {
testDir: './tests',
snapshotPathTemplate: './__screenshots__/{platform}/{projectName}/{testFilePath}/{arg}{ext}',
}
```
Currently supported "magic tokens" inside the `snapshotPathTemplate`
are:
- `{testDir}` - project's `testDir`
- `{snapshotDir}` - project's `snapshotDir`
- `{platform}` - `process.platform`
- `{projectName}` - Project's sanitized name
- `{testFileDir}` - Directories in relative path from `testDir` to test
file path (e.g. `page/` in the example below)
- `{testFileName}` - Test file name (with extension) (e.g.
`page-click.spec.ts` in the example below)
- `{testFilePath}` - Relative path from `testDir` to test file path
(e.g. `page/page-click.spec.ts` in the example below)
- `{ext}` - snapshot extension (with dots)
- `{arg}` - joined snapshot name parts, without extension (e.g.
`foo/bar/baz` in the example below)
- `{snapshotSuffix}` - `testInfo.snapshotSuffix` value.
Consider the following file structure:
```
playwright.config.ts
tests/
└── page/
└── page-click.spec.ts
```
The following `page-click.spec.ts`:
```ts
// page-click.spec.ts
import { test, expect } from '@playwright/test';
test('should work', async ({ page }) => {
await expect(page).toHaveScreenshot(['foo', 'bar', 'baz.png']);
});
```
Fixes#7792
Per discussion in #12950, adding sanitized name to the output filename
prefix. This can make debugging easier, and the filename structure more
human friendly.
We have to reliably know whether test timed out or not, and soft expect
error could have marked it with `status=failed` but it would still time
out. Now we have a separate `_didTimeout` flag for this.
Fixes#18023.
