This patch implements a new image comparison function, codenamed
"ssim-cie94". The goal of the new comparison function is to cancel out
browser non-determenistic rendering.
To use the new comparison function:
```ts
await expect(page).toHaveScreenshot({
comparator: 'ssim-cie94',
});
```
As of Nov 30, 2022, we identified the following sources of
non-determenistic rendering for Chromium:
- Anti-aliasing for certain shapes might be different due to the
way skia rasterizes certain shapes.
- Color blending might be different on `x86` and `aarch64`
architectures.
The new function employs a few heuristics to fight these
differences.
Consider two non-equal image pixels `(r1, g1, b1)` and `(r2, g2, b2)`:
1. If the [CIE94] metric is less then 1.0, then we consider these pixels
**EQUAL**. (The value `1.0` is the [just-noticeable difference] for
[CIE94].). Otherwise, proceed to next step.
1. If all the 8 neighbors of the first pixel match its color, or
if the 8 neighbors of the second pixel match its color, then these
pixels are **DIFFERENT**. (In case of anti-aliasing, some of the
direct neighbors have to be blended up or down.) Otherwise, proceed
to next step.
1. If SSIM in some locality around the different pixels is more than
0.99, then consider this pixels to be **EQUAL**. Otherwise, mark them
as **DIFFERENT**. (Local SSIM for anti-aliased pixels turns out to be
very close to 1.0).
[CIE94]: https://en.wikipedia.org/wiki/Color_difference#CIE94
[just-noticeable difference]:
https://en.wikipedia.org/wiki/Just-noticeable_difference
This configuration option allows to set a string with template
values for precise control over snapshot path location.
An example of `snapshotPathTemplate` usage:
```ts
// playwright.config.ts
// Notice the `testDir` configuration!
export default {
testDir: './tests',
snapshotPathTemplate: './__screenshots__/{platform}/{projectName}/{testFilePath}/{arg}{ext}',
}
```
Currently supported "magic tokens" inside the `snapshotPathTemplate`
are:
- `{testDir}` - project's `testDir`
- `{snapshotDir}` - project's `snapshotDir`
- `{platform}` - `process.platform`
- `{projectName}` - Project's sanitized name
- `{testFileDir}` - Directories in relative path from `testDir` to test
file path (e.g. `page/` in the example below)
- `{testFileName}` - Test file name (with extension) (e.g.
`page-click.spec.ts` in the example below)
- `{testFilePath}` - Relative path from `testDir` to test file path
(e.g. `page/page-click.spec.ts` in the example below)
- `{ext}` - snapshot extension (with dots)
- `{arg}` - joined snapshot name parts, without extension (e.g.
`foo/bar/baz` in the example below)
- `{snapshotSuffix}` - `testInfo.snapshotSuffix` value.
Consider the following file structure:
```
playwright.config.ts
tests/
└── page/
└── page-click.spec.ts
```
The following `page-click.spec.ts`:
```ts
// page-click.spec.ts
import { test, expect } from '@playwright/test';
test('should work', async ({ page }) => {
await expect(page).toHaveScreenshot(['foo', 'bar', 'baz.png']);
});
```
Fixes#7792
This patch opts into the `__screnshots__` folder snapshot management
for docker.
With this patch, docker-originating snapshots will be stored in the
following folder:
```
{testDir}/__screenshots__/{projectName}/{testFilePath}/{snapshotName}
```
Where `{testFilePath}` is a test file path relative to `testDir`
Drive-by: introduce and document the `PLAYWRIGHT_DOCKER` environment
variable that enables docker integration.
This patch changes `expect(locator).toHaveAttribute()` so that the
`value` argument can be omitted. When done so, the method will
assert attribute existance.
Fixes#16517
Note: all toHaveScreenshot tests still use `__screenshots__` directory
for their expectations. One more test was added to make sure that
by default, `toHaveScreenshot` uses old snapshots.
feat: support experimental doc entries
- Params/options/members are marked as experimental in the docs.
- `experimental.d.ts` is generated that contains all types and
includes experimental features.
- `experimental.d.ts` is references in our tests so that we
can test experimental features.
- `fonts` option is restored as experimental.
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.
This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
This patch adds support to multiple diffs. These are possible
due to soft assertions.
Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
* `fonts: "ready"`
* `animations: "disabled"`
* `size: "css"`
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)"
commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot (#12539)"
commit a3dff45974.
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.
The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.
For example, given the following config:
```js
// playwright.config.ts
module.exports = {
projects: [
{ name: 'Mobile Safari' },
{ name: 'Desktop Chrome' },
],
};
```
And the following test structure:
```
smoke-tests/
└── basic.spec.ts
```
Will result in the following screenshots folder structure by default:
```
__screenshots__/
└── darwin/
├── Mobile Safari/
│ └── smoke-tests/
│ └── basic.spec.ts/
│ └── screenshot-expectation.png
└── Desktop Chrome/
└── smoke-tests/
└── basic.spec.ts/
└── screenshot-expectation.png
```
This patch prepares for the `toHaveScreenshot` implementation
by splitting common parts from `toMatchSnapshot`.
Drive-by: fix default extension generation from `.bin` to `.dat`
for unknown buffers.
This patch adds additional options to `toMatchSnapshot` method:
- `pixelCount` - acceptable number of pixels that differ to still
consider images equal. Unset by default.
- `pixelRatio` - acceptable ratio of all image pixels (from 0 to 1) that differ to still
consider images equal. Unset by default.
Fixes#12167, #10219
Soft expects will still fail the test, but will not abort it's execution. As a consequence of this:
- `TestResult` now might have multiple errors, which is reflected with a new `testResult.erros: TestError[]` field.
- `TestInfo` now might have multiple errors as well, which is reflected with a new `testInfo.errors: TestError[]` field.
Fixes#7819
Textual snapshot diffs were previously broken in the HTML Report. The strikethrough'd text extended beyond the intended region.
HTML Report Before:
<img width="693" alt="Screen Shot 2021-12-27 at 4 43 35 PM" src="https://user-images.githubusercontent.com/11915034/147518750-a60f9002-6eed-48a1-a412-20fabd076fa6.png">
HTML Report After:
<img width="206" alt="Screen Shot 2021-12-27 at 4 48 37 PM" src="https://user-images.githubusercontent.com/11915034/147518762-19a4c8f9-ccc3-4a3c-a962-5a42edc6fc5d.png">
This now matches what's expected and shown in the terminal (which has always been correct):
<img width="1384" alt="Screen Shot 2021-12-27 at 4 36 29 PM" src="https://user-images.githubusercontent.com/11915034/147518799-f538259e-5a45-4d6f-916c-a12ccb620c5b.png">
NB: This MR is a workaround, but not a root cause fix. It works, but I never fully got to the root cause so a bug upstream may be required. It's unclear whether it's (1) in [`colors`](https://www.npmjs.com/package/colors), (2) in [`ansi-to-html`](https://www.npmjs.com/package/ansi-to-html), or (3) Playwright's use of the two. Since the terminal output is correct, I suspect it is in `ansi-to-html`. For example:
```js
const colors = require("colors");
const Convert = require('ansi-to-html');
const convert = new Convert();
// original (strike incorrectly wraps everything in the HTML)
console.log(convert.toHtml(colors.strikethrough("crossed out") + ' ' + colors.red("red")))
// prints: <strike>crossed out <span style="color:#A00">red<span style="color:#FFF"></span></span></strike>
// workaround
console.log(convert.toHtml(colors.reset(colors.strikethrough("crossed out")) + ' ' + colors.red("red")))
// prints: <strike>crossed out</strike> <span style="color:#A00">red<span style="color:#FFF"></span></span>
```
Fixes#11116
Makes it easier to understand that expect does indeed have a separate timeout.
```
Error: expect(received).toHaveCount(expected) // deep equality
Expected: 0
Received: 1
Call log:
- expect.toHaveCount with timeout 500ms
- waiting for selector "span"
- selector resolved to 1 element
- unexpected value "1"
- selector resolved to 1 element
- unexpected value "1"
- selector resolved to 1 element
- unexpected value "1"
```
