Filip/playwright
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

chore: pull expect.toHaveScreenshot() from 1.20 release (#12702)

Browse source 
This patch pulls `expect.toHaveScreenshot()` from 1.20 release.
We'd like to experiment more.

After this patch, there is no 'toHaveScreenshot' mention in the
codebase.
This commit is contained in:
Andrey Lushnikov 2022-03-11 20:56:43 -07:00 committed by GitHub
parent 4897967780
commit cf1a60bce8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 30 additions and 1012 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

30
docs/src/api/class-locatorassertions.md
View file

@ -998,36 +998,6 @@ Property value.
### option: LocatorAssertions.toHaveJSProperty.timeout = %%-js-assertions-timeout-%%
### option: LocatorAssertions.toHaveJSProperty.timeout = %%-csharp-java-python-assertions-timeout-%%
## async method: LocatorAssertions.toHaveScreenshot
* langs: js
Ensures that [Locator] resolves to a given screenshot. This function will re-take
screenshots until it matches with the saved expectation.
If there's no expectation yet, it will wait until two consecutive screenshots
yield the same result, and save the last one as an expectation.
```js
const locator = page.locator('button');
await expect(locator).toHaveScreenshot();
```
### option: LocatorAssertions.toHaveScreenshot.timeout = %%-js-assertions-timeout-%%
### option: LocatorAssertions.toHaveScreenshot.timeout = %%-csharp-java-python-assertions-timeout-%%
### option: LocatorAssertions.toHaveScreenshot.animations = %%-screenshot-option-animations-%%
### option: LocatorAssertions.toHaveScreenshot.omitBackground = %%-screenshot-option-omit-background-%%
### option: LocatorAssertions.toHaveScreenshot.mask = %%-screenshot-option-mask-%%
### option: LocatorAssertions.toHaveScreenshot.maxDiffPixels = %%-assertions-max-diff-pixels-%%
### option: LocatorAssertions.toHaveScreenshot.maxDiffPixelRatio = %%-assertions-max-diff-pixel-ratio-%%
### option: LocatorAssertions.toHaveScreenshot.threshold = %%-assertions-threshold-%%
## async method: LocatorAssertions.toHaveText
* langs:
- alias-java: hasText

32
docs/src/api/class-pageassertions.md
View file

@ -114,38 +114,6 @@ Expected substring or RegExp.
### option: PageAssertions.NotToHaveURL.timeout = %%-js-assertions-timeout-%%
### option: PageAssertions.NotToHaveURL.timeout = %%-csharp-java-python-assertions-timeout-%%
## async method: PageAssertions.toHaveScreenshot
* langs: js
Ensures that the page resolves to a given screenshot. This function will re-take
screenshots until it matches with the saved expectation.
If there's no expectation yet, it will wait until two consecutive screenshots
yield the same result, and save the last one as an expectation.
```js
await expect(page).toHaveScreenshot();
```
### option: PageAssertions.toHaveScreenshot.timeout = %%-js-assertions-timeout-%%
### option: PageAssertions.toHaveScreenshot.timeout = %%-csharp-java-python-assertions-timeout-%%
### option: PageAssertions.toHaveScreenshot.animations = %%-screenshot-option-animations-%%
### option: PageAssertions.toHaveScreenshot.omitBackground = %%-screenshot-option-omit-background-%%
### option: PageAssertions.toHaveScreenshot.fullPage = %%-screenshot-option-full-page-%%
### option: PageAssertions.toHaveScreenshot.clip = %%-screenshot-option-clip-%%
### option: PageAssertions.toHaveScreenshot.mask = %%-screenshot-option-mask-%%
### option: PageAssertions.toHaveScreenshot.maxDiffPixels = %%-assertions-max-diff-pixels-%%
### option: PageAssertions.toHaveScreenshot.maxDiffPixelRatio = %%-assertions-max-diff-pixel-ratio-%%
### option: PageAssertions.toHaveScreenshot.threshold = %%-assertions-threshold-%%
## async method: PageAssertions.toHaveTitle
* langs:
- alias-java: hasTitle

3
docs/src/api/class-screenshotassertions.md
View file

@ -2,8 +2,7 @@
* langs: js
Playwright provides methods for comparing page and element screenshots with
expected values stored in files. See also [`method: PageAssertions.toHaveScreenshot`] and
[`LocatorAssertions.toHaveScreenshot`].
expected values stored in files.
```js
expect(screenshot).toMatchSnapshot('landing-page.png');

28
docs/src/release-notes-js.md
View file

@ -7,28 +7,21 @@ title: "Release notes"
## Version 1.20
### Visual Regression Testing
### Highlights
- New options for methods [`method: Page.screenshot`], [`method: Locator.screenshot`] and [`method: ElementHandle.screenshot`]:
* Option `animations: "disabled"` rewinds all CSS animations and transitions to a consistent state
* Option `mask: Locator[]` masks given elements, overlaying them with pink `#FF00FF` boxes.
- New web-first assertions for screenshots: [`method: PageAssertions.toHaveScreenshot`] and [`method: LocatorAssertions.toHaveScreenshot`]. These methods will re-take screenshot until it matches the saved expectation. When generating a new expectation, the method will re-take screenshots until 2 consecutive screenshots match.
New methods support both named and anonymous (auto-named) expectations:
- `expect().toMatchSnapshot()` now supports anonymous snapshots: when snapshot name is missing, Playwright Test will generate one
automatically:
```js
// Take a full-page screenshot with a named expectation `fullpage.png`.
await expect(page).toHaveScreenshot('fullpage.png', { fullPage: true });
// Take a screenshot of an element with anonymous expectation.
await expect(page.locator('text=Booking')).toHaveScreenshot();
expect('Web is Awesome <3').toMatchSnapshot();
```
Methods support all screenshot options from [`method: Page.screenshot`] and [`method: Locator.screenshot`].
These methods also support new `maxDiffPixels` and `maxDiffPixelRatio` options for fine-grained screenshot comparison:
- New `maxDiffPixels` and `maxDiffPixelRatio` options for fine-grained screenshot comparison using `expect().toMatchSnapshot()`:
```js
await expect(page).toHaveScreenshot({
expect(await page.screenshot()).toMatchSnapshot({
fullPage: true, // take a full page screenshot
maxDiffPixels: 27, // allow no more than 27 different pixels.
});
@ -36,8 +29,6 @@ title: "Release notes"
It is most convenient to specify `maxDiffPixels` or `maxDiffPixelRatio` once in [`property: TestConfig.expect`].
### Other Updates
- Playwright Test now adds [`property: TestConfig.fullyParallel`] mode. By default, Playwright Test parallelizes between files. In fully parallel mode, tests inside a single file are also run in parallel. You can also use `--fully-parallel` command line flag.
```ts
@ -62,13 +53,6 @@ title: "Release notes"
```
- [Trace Viewer](./trace-viewer) now shows [API testing requests](./test-api-testing).
- `expect().toMatchSnapshot()` now supports anonymous snapshots: when snapshot name is missing, Playwright Test will generate one
automatically:
```js
expect('Web is Awesome <3').toMatchSnapshot();
```
- [`method: Locator.highlight`] visually reveals element(s) for easier debugging.
### Announcements

10
docs/src/test-api/class-testconfig.md
View file

@ -36,10 +36,6 @@ export default config;
## property: TestConfig.expect
- type: <[Object]>
- `timeout` <[int]> Default timeout for async expect matchers in milliseconds, defaults to 5000ms.
- `toHaveScreenshot` <[Object]>
- `threshold` <[float]> an acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
- `maxDiffPixels` <[int]> an acceptable amount of pixels that could be different, unset by default.
- `maxDiffPixelRatio` <[float]> an acceptable ratio of pixels that are different to the total amount of pixels, between `0` and `1` , unset by default.
- `toMatchSnapshot` <[Object]>
- `threshold` <[float]> an acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
- `maxDiffPixels` <[int]> an acceptable amount of pixels that could be different, unset by default.
@ -55,7 +51,7 @@ Configuration for the `expect` assertion library. Learn more about [various time
const config = {
expect: {
timeout: 10000,
toHaveScreenshot: {
toMatchSnapshot: {
maxDiffPixels: 10,
},
},
@ -71,7 +67,7 @@ import { PlaywrightTestConfig } from '@playwright/test';
const config: PlaywrightTestConfig = {
expect: {
timeout: 10000,
toHaveScreenshot: {
toMatchSnapshot: {
maxDiffPixels: 10,
},
},
@ -303,7 +299,7 @@ test('example test', async ({}, testInfo) => {
## property: TestConfig.snapshotDir
- type: <[string]>
The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot` and `toHaveScreenshot`. Defaults to [`property: TestConfig.testDir`].
The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot`. Defaults to [`property: TestConfig.testDir`].
The directory for each test can be accessed by [`property: TestInfo.snapshotDir`] and [`method: TestInfo.snapshotPath`].

2
docs/src/test-api/class-testinfo.md
View file

@ -382,7 +382,7 @@ The name of the snapshot or the path segments to define the snapshot file path.
## property: TestInfo.snapshotSuffix
- type: <[string]>
Suffix used to differentiate snapshots between multiple test configurations. For example, if snapshots depend on the platform, you can set `testInfo.snapshotSuffix` equal to `process.platform`. In this case both `expect(value).toMatchSnapshot(snapshotName)` and `expect(page).toHaveScreenshot(snapshotName)` will use different snapshots depending on the platform. Learn more about [snapshots](./test-snapshots.md).
Suffix used to differentiate snapshots between multiple test configurations. For example, if snapshots depend on the platform, you can set `testInfo.snapshotSuffix` equal to `process.platform`. In this case both `expect(value).toMatchSnapshot(snapshotName)` will use different snapshots depending on the platform. Learn more about [snapshots](./test-snapshots.md).
## property: TestInfo.status
- type: <[void]|[TestStatus]<"passed"|"failed"|"timedOut"|"skipped">>

6
docs/src/test-api/class-testproject.md
View file

@ -107,10 +107,6 @@ export default config;
## property: TestProject.expect
- type: <[Object]>
- `timeout` <[int]> Default timeout for async expect matchers in milliseconds, defaults to 5000ms.
- `toHaveScreenshot` <[Object]>
- `threshold` <[float]> an acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
- `maxDiffPixels` <[int]> an acceptable amount of pixels that could be different, unset by default.
- `maxDiffPixelRatio` <[float]> an acceptable ratio of pixels that are different to the total amount of pixels, between `0` and `1` , unset by default.
- `toMatchSnapshot` <[Object]>
- `threshold` <[float]> an acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between the same pixel in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
- `maxDiffPixels` <[int]> an acceptable amount of pixels that could be different, unset by default.
@ -155,7 +151,7 @@ Project name is visible in the report and during test execution.
## property: TestProject.snapshotDir
- type: <[string]>
The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot` and `toHaveScreenshot`. Defaults to [`property: TestProject.testDir`].
The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot`. Defaults to [`property: TestProject.testDir`].
The directory for each test can be accessed by [`property: TestInfo.snapshotDir`] and [`method: TestInfo.snapshotPath`].

22
docs/src/test-snapshots-js.md
View file

@ -3,7 +3,7 @@ id: test-snapshots
title: "Visual comparisons"
---
Playwright Test includes the ability to produce and visually compare screenshots using `await expect(pageOrLocator).toHaveScreenshot()`. On first execution, Playwright test will generate reference screenshots. Subsequent runs will compare against the reference.
Playwright Test includes the ability to produce and visually compare screenshots using `expect().toMatchSnapshot()`. On first execution, Playwright test will generate reference screenshots. Subsequent runs will compare against the reference.
```js js-flavor=js
// example.spec.js
@ -11,7 +11,7 @@ const { test, expect } = require('@playwright/test');
test('example test', async ({ page }) => {
await page.goto('https://playwright.dev');
await expect(page).toHaveScreenshot();
expect(await page.screenshot()).toMatchSnapshot();
});
```
@ -21,7 +21,7 @@ import { test, expect } from '@playwright/test';
test('example test', async ({ page }) => {
await page.goto('https://playwright.dev');
await expect(page).toHaveScreenshot();
expect(await page.screenshot()).toMatchSnapshot();
});
```
@ -45,10 +45,10 @@ drwxr-xr-x 3 user group 96 Jun 4 11:46 example.spec.ts-snapshots
The snapshot name `example-test-1-chromium-darwin.png` consists of a few parts:
- `example-test-1.png` - an auto-generated name of the snapshot. Alternatively you can specify snapshot name as the first argument of the `toMatchSnapshot()` method:
```js js-flavor=js
await expect(page).toHaveScreenshot('landing.png');
expect(await page.screenshot()).toMatchSnapshot('landing.png');
```
```js js-flavor=ts
await expect(page).toHaveScreenshot('landing.png');
expect(await page.screenshot()).toMatchSnapshot('landing.png');
```
- `chromium-darwin` - the browser name and the platform. Screenshots differ between browsers and platforms due to different rendering, fonts and more, so you will need different snapshots for them. If you use multiple projects in your [configuration file](./test-configuration.md), project name will be used instead of `chromium`.
@ -67,10 +67,10 @@ Sometimes you need to update the reference screenshot, for example when the page
npx playwright test --update-snapshots
```
> Note that `snapshotName` also accepts an array of path segments to the snapshot file such as `await expect(page).toHaveScreenshot(['relative', 'path', 'to', 'snapshot.png'])`.
> Note that `snapshotName` also accepts an array of path segments to the snapshot file such as `expect().toMatchSnapshot(['relative', 'path', 'to', 'snapshot.png'])`.
> However, this path must stay within the snapshots directory for each test file (i.e. `a.spec.js-snapshots`), otherwise it will throw.
Playwright Test uses the [pixelmatch](https://github.com/mapbox/pixelmatch) library. You can [pass various options](./test-assertions#expectpageorlocatortohavescreenshot-options) to modify its behavior:
You can pass various options to modify image comparison behavior - see [`method: ScreenshotAssertions.toMatchSnapshot`]:
```js js-flavor=js
// example.spec.js
@ -78,7 +78,7 @@ const { test, expect } = require('@playwright/test');
test('example test', async ({ page }) => {
await page.goto('https://playwright.dev');
await expect(page).toHaveScreenshot({ maxDiffPixels: 100 });
expect(await page.screenshot()).toMatchSnapshot({ maxDiffPixels: 100 });
});
```
@ -88,7 +88,7 @@ import { test, expect } from '@playwright/test';
test('example test', async ({ page }) => {
await page.goto('https://playwright.dev');
await expect(page).toHaveScreenshot({ maxDiffPixels: 100 });
expect(await page.screenshot()).toMatchSnapshot({ maxDiffPixels: 100 });
});
```
@ -97,7 +97,7 @@ If you'd like to share the default value among all the tests in the project, you
```js js-flavor=js
module.exports = {
expect: {
toHaveScreenshot: { maxDiffPixels: 100 },
toMatchSnapshot: { maxDiffPixels: 100 },
},
};
```
@ -106,7 +106,7 @@ module.exports = {
import { PlaywrightTestConfig } from '@playwright/test';
const config: PlaywrightTestConfig = {
expect: {
toHaveScreenshot: { maxDiffPixels: 100 },
toMatchSnapshot: { maxDiffPixels: 100 },
},
};
export default config;

3
packages/playwright-test/src/expect.ts
View file

@ -43,7 +43,7 @@ import {
toHaveURL,
toHaveValue
} from './matchers/matchers';
import { toMatchSnapshot, toHaveScreenshot } from './matchers/toMatchSnapshot';
import { toMatchSnapshot } from './matchers/toMatchSnapshot';
import type { Expect, TestError } from './types';
import matchers from 'expect/build/matchers';
import { currentTestInfo } from './globals';
@ -132,7 +132,6 @@ const customMatchers = {
toHaveURL,
toHaveValue,
toMatchSnapshot,
toHaveScreenshot,
};
type ExpectMetaInfo = {

107
packages/playwright-test/src/matchers/toMatchSnapshot.ts
View file

@ -14,14 +14,10 @@
* limitations under the License.
*/
import { Locator, Page } from 'playwright-core';
import type { Page as PageEx } from 'playwright-core/lib/client/page';
import type { Locator as LocatorEx } from 'playwright-core/lib/client/locator';
import type { Expect } from '../types';
import { currentTestInfo } from '../globals';
import { mimeTypeToComparator, ImageComparatorOptions, Comparator } from 'playwright-core/lib/utils/comparators';
import type { PageScreenshotOptions } from 'playwright-core/types/types';
import { addSuffixToFilePath, serializeError, sanitizeForFilePath, trimLongString, callLogText, currentExpectTimeout } from '../util';
import { addSuffixToFilePath, serializeError, sanitizeForFilePath, trimLongString, callLogText } from '../util';
import { UpdateSnapshots } from '../types';
import colors from 'colors/safe';
import fs from 'fs';
@ -248,107 +244,6 @@ export function toMatchSnapshot(
return helper.handleDifferent(received, expected, result.diff, result.errorMessage, undefined);
}
type HaveScreenshotOptions = ImageComparatorOptions & Omit<PageScreenshotOptions, 'type' | 'quality' | 'path'>;
export async function toHaveScreenshot(
this: ReturnType<Expect['getState']>,
pageOrLocator: Page | Locator,
nameOrOptions: NameOrSegments | { name?: NameOrSegments } & HaveScreenshotOptions = {},
optOptions: HaveScreenshotOptions = {}
): Promise<SyncExpectationResult> {
const testInfo = currentTestInfo();
if (!testInfo)
throw new Error(`toHaveScreenshot() must be called during the test`);
const helper = new SnapshotHelper(
testInfo, 'png',
testInfo.project.expect?.toHaveScreenshot || {},
nameOrOptions, optOptions);
const [page, locator] = pageOrLocator.constructor.name === 'Page' ? [(pageOrLocator as PageEx), undefined] : [(pageOrLocator as Locator).page() as PageEx, pageOrLocator as LocatorEx];
const screenshotOptions = {
...helper.allOptions,
mask: (helper.allOptions.mask || []) as LocatorEx[],
name: undefined,
threshold: undefined,
maxDiffPixels: undefined,
maxDiffPixelRatio: undefined,
};
const hasSnapshot = fs.existsSync(helper.snapshotPath);
if (this.isNot) {
if (!hasSnapshot)
return helper.handleMissingNegated();
// Having `errorMessage` means we timed out while waiting
// for screenshots not to match, so screenshots
// are actually the same in the end.
const isDifferent = !(await page._expectScreenshot({
expected: await fs.promises.readFile(helper.snapshotPath),
isNot: true,
locator,
comparatorOptions: helper.comparatorOptions,
screenshotOptions,
timeout: currentExpectTimeout(helper.allOptions),
})).errorMessage;
return isDifferent ? helper.handleDifferentNegated() : helper.handleMatchingNegated();
}
// Fast path: there's no screenshot and we don't intend to update it.
if (helper.updateSnapshots === 'none' && !hasSnapshot)
return { pass: false, message: () => `${helper.snapshotPath} is missing in snapshots.` };
if (helper.updateSnapshots === 'all' || !hasSnapshot) {
// Regenerate a new screenshot by waiting until two screenshots are the same.
const timeout = currentExpectTimeout(helper.allOptions);
const { actual, previous, diff, errorMessage, log } = await page._expectScreenshot({
expected: undefined,
isNot: false,
locator,
comparatorOptions: helper.comparatorOptions,
screenshotOptions,
timeout,
});
// We tried re-generating new snapshot but failed.
// This can be due to e.g. spinning animation, so we want to show it as a diff.
if (errorMessage) {
// TODO(aslushnikov): rename attachments to "actual" and "previous". They still should be somehow shown in HTML reporter.
const title = actual && previous ?
`Timeout ${timeout}ms exceeded while generating screenshot because ${locator ? 'element' : 'page'} kept changing:` :
`Timeout ${timeout}ms exceeded while generating screenshot:`;
return helper.handleDifferent(actual, previous, diff, undefined, log, title);
}
// We successfully (re-)generated new screenshot.
if (!hasSnapshot)
return helper.handleMissing(actual!);
writeFileSync(helper.snapshotPath, actual!);
/* eslint-disable no-console */
console.log(helper.snapshotPath + ' is re-generated, writing actual.');
return {
pass: true,
message: () => helper.snapshotPath + ' running with --update-snapshots, writing actual.'
};
}
// General case:
// - snapshot exists
// - regular matcher (i.e. not a `.not`)
// - no flags to update screenshots
const expected = await fs.promises.readFile(helper.snapshotPath);
const { actual, diff, errorMessage, log } = await page._expectScreenshot({
expected,
isNot: false,
locator,
comparatorOptions: helper.comparatorOptions,
screenshotOptions,
timeout: currentExpectTimeout(helper.allOptions),
});
return errorMessage ?
helper.handleDifferent(actual, expected, diff, errorMessage, log) :
helper.handleMatching();
}
function writeFileSync(aPath: string, content: Buffer | string) {
fs.mkdirSync(path.dirname(aPath), { recursive: true });
fs.writeFileSync(aPath, content);

25
packages/playwright-test/types/test.d.ts vendored
View file

@ -43,19 +43,6 @@ type ExpectSettings = {
* Default timeout for async expect matchers in milliseconds, defaults to 5000ms.
*/
timeout?: number;
toHaveScreenshot?: {
/** An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between pixels in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
*/
threshold?: number,
/**
* An acceptable amount of pixels that could be different, unset by default.
*/
maxDiffPixels?: number,
/**
* An acceptable ratio of pixels that are different to the total amount of pixels, between `0` and `1` , unset by default.
*/
maxDiffPixelRatio?: number,
}
toMatchSnapshot?: {
/** An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between pixels in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
*/
@ -170,8 +157,7 @@ interface TestProject {
*/
name?: string;
/**
* The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot` and
* `toHaveScreenshot`. Defaults to
* The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot`. Defaults to
* [testProject.testDir](https://playwright.dev/docs/api/class-testproject#test-project-test-dir).
*
* The directory for each test can be accessed by
@ -709,7 +695,7 @@ interface TestConfig {
* const config: PlaywrightTestConfig = {
* expect: {
* timeout: 10000,
* toHaveScreenshot: {
* toMatchSnapshot: {
* maxDiffPixels: 10,
* },
* },
@ -725,8 +711,7 @@ interface TestConfig {
metadata?: any;
name?: string;
/**
* The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot` and
* `toHaveScreenshot`. Defaults to
* The base directory, relative to the config file, for snapshot files created with `toMatchSnapshot`. Defaults to
* [testConfig.testDir](https://playwright.dev/docs/api/class-testconfig#test-config-test-dir).
*
* The directory for each test can be accessed by
@ -1580,8 +1565,8 @@ export interface TestInfo {
/**
* Suffix used to differentiate snapshots between multiple test configurations. For example, if snapshots depend on the
* platform, you can set `testInfo.snapshotSuffix` equal to `process.platform`. In this case both
* `expect(value).toMatchSnapshot(snapshotName)` and `expect(page).toHaveScreenshot(snapshotName)` will use different
* snapshots depending on the platform. Learn more about [snapshots](https://playwright.dev/docs/test-snapshots).
* `expect(value).toMatchSnapshot(snapshotName)` will use different snapshots depending on the platform. Learn more about
* [snapshots](https://playwright.dev/docs/test-snapshots).
*/
snapshotSuffix: string;
/**

24
packages/playwright-test/types/testExpect.d.ts vendored
View file

@ -222,18 +222,6 @@ interface LocatorMatchers {
* Asserts given DOM node visible on the screen.
*/
toBeVisible(options?: { timeout?: number }): Promise<Locator>;
/**
* Asserts element's screenshot is matching to the snapshot.
*/
toHaveScreenshot(options?: Omit<LocatorScreenshotOptions, 'path' | 'type' | 'quality'> & ImageComparatorOptions & {
name?: string | string[],
}): Promise<Locator>;
/**
* Asserts element's screenshot is matching to the snapshot.
*/
toHaveScreenshot(name: string | string[], options?: Omit<LocatorScreenshotOptions, 'path' | 'type' | 'quality'> & ImageComparatorOptions): Promise<Locator>;
}
interface PageMatchers {
/**
@ -245,18 +233,6 @@ interface PageMatchers {
* Asserts page's URL.
*/
toHaveURL(expected: string | RegExp, options?: { timeout?: number }): Promise<Page>;
/**
* Asserts page screenshot is matching to the snapshot.
*/
toHaveScreenshot(options?: Omit<PageScreenshotOptions, 'path' | 'quality' | 'type'> & ImageComparatorOptions & {
name?: string | string[],
}): Promise<Page>;
/**
* Asserts page screenshot is matching to the snapshot.
*/
toHaveScreenshot(name: string | string[], options?: Omit<PageScreenshotOptions, 'path' | 'quality' | 'type'> & ImageComparatorOptions): Promise<Page>;
}
interface APIResponseMatchers {

737
tests/playwright-test/to-have-screenshot.spec.ts
View file

@ -1,737 +0,0 @@
/**
* Copyright Microsoft Corporation. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { mimeTypeToComparator } from 'playwright-core/lib/utils/comparators';
import * as fs from 'fs';
import { PNG } from 'pngjs';
import * as path from 'path';
import { pathToFileURL } from 'url';
import { test, expect, stripAnsi, createImage, paintBlackPixels } from './playwright-test-fixtures';
const pngComparator = mimeTypeToComparator['image/png'];
test.describe.configure({ mode: 'parallel' });
const IMG_WIDTH = 1280;
const IMG_HEIGHT = 720;
const whiteImage = createImage(IMG_WIDTH, IMG_HEIGHT, 255, 255, 255);
const redImage = createImage(IMG_WIDTH, IMG_HEIGHT, 255, 0, 0);
const greenImage = createImage(IMG_WIDTH, IMG_HEIGHT, 0, 255, 0);
const blueImage = createImage(IMG_WIDTH, IMG_HEIGHT, 0, 0, 255);
const files = {
'helper.ts': `
export const test = pwt.test.extend({
auto: [ async ({}, run, testInfo) => {
testInfo.snapshotSuffix = '';
await run();
}, { auto: true } ]
});
`
};
test('should fail to screenshot a page with infinite animation', async ({ runInlineTest }, testInfo) => {
const infiniteAnimationURL = pathToFileURL(path.join(__dirname, '../assets/rotate-z.html'));
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await page.goto('${infiniteAnimationURL}');
await expect(page).toHaveScreenshot({ timeout: 2000 });
});
`
});
expect(result.exitCode).toBe(1);
expect(stripAnsi(result.output)).toContain(`Timeout 2000ms exceeded while generating screenshot because page kept changing`);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-actual.png'))).toBe(true);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-expected.png'))).toBe(true);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-diff.png'))).toBe(true);
expect(fs.existsSync(testInfo.outputPath('a.spec.js-snapshots', 'is-a-test-1.png'))).toBe(false);
});
test('should not fail when racing with navigation', async ({ runInlineTest }, testInfo) => {
const infiniteAnimationURL = pathToFileURL(path.join(__dirname, '../assets/rotate-z.html'));
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': createImage(10, 10, 255, 0, 0),
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await Promise.all([
page.goto('${infiniteAnimationURL}'),
expect(page).toHaveScreenshot({
name: 'snapshot.png',
animations: "disabled",
clip: { x: 0, y: 0, width: 10, height: 10 },
}),
]);
});
`
});
expect(result.exitCode).toBe(0);
});
test('should successfully screenshot a page with infinite animation with disableAnimation: true', async ({ runInlineTest }, testInfo) => {
const infiniteAnimationURL = pathToFileURL(path.join(__dirname, '../assets/rotate-z.html'));
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await page.goto('${infiniteAnimationURL}');
await expect(page).toHaveScreenshot({
animations: "disabled",
});
});
`
}, { 'update-snapshots': true });
expect(result.exitCode).toBe(0);
expect(fs.existsSync(testInfo.outputPath('a.spec.js-snapshots', 'is-a-test-1.png'))).toBe(true);
});
test('should support clip option for page', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': createImage(50, 50, 255, 255, 255),
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot({
name: 'snapshot.png',
clip: { x: 0, y: 0, width: 50, height: 50, },
});
});
`
});
expect(result.exitCode).toBe(0);
});
test('should support omitBackground option for locator', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await page.evaluate(() => {
document.body.style.setProperty('width', '100px');
document.body.style.setProperty('height', '100px');
});
await expect(page.locator('body')).toHaveScreenshot({
name: 'snapshot.png',
omitBackground: true,
});
});
`
}, { 'update-snapshots': true });
expect(result.exitCode).toBe(0);
const snapshotPath = testInfo.outputPath('a.spec.js-snapshots', 'snapshot.png');
expect(fs.existsSync(snapshotPath)).toBe(true);
const png = PNG.sync.read(fs.readFileSync(snapshotPath));
expect.soft(png.width, 'image width must be 100').toBe(100);
expect.soft(png.height, 'image height must be 100').toBe(100);
expect.soft(png.data[0], 'image R must be 0').toBe(0);
expect.soft(png.data[1], 'image G must be 0').toBe(0);
expect.soft(png.data[2], 'image B must be 0').toBe(0);
expect.soft(png.data[3], 'image A must be 0').toBe(0);
});
test('should fail to screenshot an element with infinite animation', async ({ runInlineTest }, testInfo) => {
const infiniteAnimationURL = pathToFileURL(path.join(__dirname, '../assets/rotate-z.html'));
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await page.goto('${infiniteAnimationURL}');
await expect(page.locator('body')).toHaveScreenshot({ timeout: 2000 });
});
`
});
expect(result.exitCode).toBe(1);
expect(stripAnsi(result.output)).toContain(`Timeout 2000ms exceeded while generating screenshot because element kept changing`);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-actual.png'))).toBe(true);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-expected.png'))).toBe(true);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-diff.png'))).toBe(true);
expect(fs.existsSync(testInfo.outputPath('a.spec.js-snapshots', 'is-a-test-1.png'))).toBe(false);
});
test('should fail to screenshot an element that keeps moving', async ({ runInlineTest }, testInfo) => {
const infiniteAnimationURL = pathToFileURL(path.join(__dirname, '../assets/rotate-z.html'));
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await page.goto('${infiniteAnimationURL}');
await expect(page.locator('div')).toHaveScreenshot({ timeout: 2000 });
});
`
});
expect(result.exitCode).toBe(1);
expect(stripAnsi(result.output)).toContain(`Timeout 2000ms exceeded`);
expect(stripAnsi(result.output)).toContain(`element is not stable - waiting`);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-actual.png'))).toBe(false);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-expected.png'))).toBe(false);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-diff.png'))).toBe(false);
expect(fs.existsSync(testInfo.outputPath('a.spec.js-snapshots', 'is-a-test-1.png'))).toBe(false);
});
test('should generate default name', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot();
});
`
});
expect(result.exitCode).toBe(1);
expect(fs.existsSync(testInfo.outputPath('test-results', 'a-is-a-test', 'is-a-test-1-actual.png'))).toBe(true);
expect(fs.existsSync(testInfo.outputPath('a.spec.js-snapshots', 'is-a-test-1.png'))).toBe(true);
});
test('should compile with different option combinations', async ({ runTSC }) => {
const result = await runTSC({
'a.spec.ts': `
const { test } = pwt;
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot();
await expect(page.locator('body')).toHaveScreenshot({ threshold: 0.2 });
await expect(page).toHaveScreenshot({ maxDiffPixelRatio: 0.2 });
await expect(page).toHaveScreenshot({
threshold: 0.2,
maxDiffPixels: 10,
maxDiffPixelRatio: 0.2,
animations: "disabled",
omitBackground: true,
timeout: 1000,
});
});
`
});
expect(result.exitCode).toBe(0);
});
test('should fail when screenshot is different size', async ({ runInlineTest }) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': createImage(22, 33),
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', { timeout: 2000 });
});
`
});
expect(result.exitCode).toBe(1);
expect(result.output).toContain('Expected an image 22px by 33px, received 1280px by 720px.');
});
test('should fail when screenshot is different pixels', async ({ runInlineTest }) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': paintBlackPixels(whiteImage, 12345),
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', { timeout: 2000 });
});
`
});
expect(result.exitCode).toBe(1);
expect(result.output).toContain('Screenshot comparison failed');
expect(result.output).toContain('12345 pixels');
expect(result.output).not.toContain('Call log');
expect(result.output).toContain('ratio 0.02');
expect(result.output).toContain('Expected:');
expect(result.output).toContain('Received:');
});
test('doesn\'t create comparison artifacts in an output folder for passed negated snapshot matcher', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': blueImage,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).not.toHaveScreenshot('snapshot.png');
});
`
});
expect(result.exitCode).toBe(0);
const outputText = stripAnsi(result.output);
const expectedSnapshotArtifactPath = testInfo.outputPath('test-results', 'a-is-a-test', 'snapshot-expected.png');
const actualSnapshotArtifactPath = testInfo.outputPath('test-results', 'a-is-a-test', 'snapshot-actual.png');
expect(outputText).not.toContain(`Expected: ${expectedSnapshotArtifactPath}`);
expect(outputText).not.toContain(`Received: ${actualSnapshotArtifactPath}`);
expect(fs.existsSync(expectedSnapshotArtifactPath)).toBe(false);
expect(fs.existsSync(actualSnapshotArtifactPath)).toBe(false);
});
test('should fail on same snapshots with negate matcher', async ({ runInlineTest }) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': whiteImage,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).not.toHaveScreenshot('snapshot.png', { timeout: 2000 });
});
`
});
expect(result.exitCode).toBe(1);
expect(result.output).toContain('Screenshot comparison failed:');
expect(result.output).toContain('Expected result should be different from the actual one.');
});
test('should write missing expectations locally twice and continue', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png');
await expect(page).toHaveScreenshot('snapshot2.png');
console.log('Here we are!');
});
`
});
expect(result.exitCode).toBe(1);
expect(result.failed).toBe(1);
const snapshot1OutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(result.output).toContain(`Error: ${snapshot1OutputPath} is missing in snapshots, writing actual`);
expect(pngComparator(fs.readFileSync(snapshot1OutputPath), whiteImage)).toBe(null);
const snapshot2OutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot2.png');
expect(result.output).toContain(`Error: ${snapshot2OutputPath} is missing in snapshots, writing actual`);
expect(pngComparator(fs.readFileSync(snapshot2OutputPath), whiteImage)).toBe(null);
expect(result.output).toContain('Here we are!');
const stackLines = stripAnsi(result.output).split('\n').filter(line => line.includes(' at ')).filter(line => !line.includes(testInfo.outputPath()));
expect(result.output).toContain('a.spec.js:8');
expect(stackLines.length).toBe(0);
});
test('shouldn\'t write missing expectations locally for negated matcher', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).not.toHaveScreenshot('snapshot.png');
});
`
});
expect(result.exitCode).toBe(1);
const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(result.output).toContain(`${snapshotOutputPath} is missing in snapshots, matchers using ".not" won\'t write them automatically.`);
expect(fs.existsSync(snapshotOutputPath)).toBe(false);
});
test('should update snapshot with the update-snapshots flag', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': blueImage,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png');
});
`
}, { 'update-snapshots': true });
expect(result.exitCode).toBe(0);
const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(result.output).toContain(`${snapshotOutputPath} is re-generated, writing actual.`);
expect(pngComparator(fs.readFileSync(snapshotOutputPath), whiteImage)).toBe(null);
});
test('shouldn\'t update snapshot with the update-snapshots flag for negated matcher', async ({ runInlineTest }, testInfo) => {
const EXPECTED_SNAPSHOT = blueImage;
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).not.toHaveScreenshot('snapshot.png');
});
`
}, { 'update-snapshots': true });
expect(result.exitCode).toBe(0);
const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(fs.readFileSync(snapshotOutputPath).equals(EXPECTED_SNAPSHOT)).toBe(true);
});
test('should silently write missing expectations locally with the update-snapshots flag', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png');
});
`
}, { 'update-snapshots': true });
expect(result.exitCode).toBe(0);
const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(result.output).toContain(`${snapshotOutputPath} is missing in snapshots, writing actual`);
const data = fs.readFileSync(snapshotOutputPath);
expect(pngComparator(data, whiteImage)).toBe(null);
});
test('should not write missing expectations locally with the update-snapshots flag for negated matcher', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).not.toHaveScreenshot('snapshot.png');
});
`
}, { 'update-snapshots': true });
expect(result.exitCode).toBe(1);
const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(result.output).toContain(`${snapshotOutputPath} is missing in snapshots, matchers using ".not" won\'t write them automatically.`);
expect(fs.existsSync(snapshotOutputPath)).toBe(false);
});
test('should match multiple snapshots', async ({ runInlineTest }) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/red.png': redImage,
'a.spec.js-snapshots/green.png': greenImage,
'a.spec.js-snapshots/blue.png': blueImage,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await Promise.all([
page.evaluate(() => document.documentElement.style.setProperty('background', '#f00')),
expect(page).toHaveScreenshot('red.png'),
]);
await Promise.all([
page.evaluate(() => document.documentElement.style.setProperty('background', '#0f0')),
expect(page).toHaveScreenshot('green.png'),
]);
await Promise.all([
page.evaluate(() => document.documentElement.style.setProperty('background', '#00f')),
expect(page).toHaveScreenshot('blue.png'),
]);
});
`
});
expect(result.exitCode).toBe(0);
});
test('should use provided name', async ({ runInlineTest }) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/provided.png': whiteImage,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('provided.png');
});
`
});
expect(result.exitCode).toBe(0);
});
test('should use provided name via options', async ({ runInlineTest }) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/provided.png': whiteImage,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot({ name: 'provided.png' });
});
`
});
expect(result.exitCode).toBe(0);
});
test('should respect maxDiffPixels option', async ({ runInlineTest }) => {
const BAD_PIXELS = 120;
const EXPECTED_SNAPSHOT = paintBlackPixels(whiteImage, BAD_PIXELS);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', { timeout: 2000 });
});
`
})).exitCode, 'make sure default comparison fails').toBe(1);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', {
maxDiffPixels: ${BAD_PIXELS}
});
});
`
})).exitCode, 'make sure maxDiffPixels option is respected').toBe(0);
expect((await runInlineTest({
...files,
'playwright.config.ts': `
module.exports = { projects: [
{ expect: { toHaveScreenshot: { maxDiffPixels: ${BAD_PIXELS} } } },
]};
`,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png');
});
`
})).exitCode, 'make sure maxDiffPixels option in project config is respected').toBe(0);
});
test('should satisfy both maxDiffPixelRatio and maxDiffPixels', async ({ runInlineTest }) => {
const BAD_RATIO = 0.25;
const BAD_COUNT = Math.floor(IMG_WIDTH * IMG_HEIGHT * BAD_RATIO);
const EXPECTED_SNAPSHOT = paintBlackPixels(whiteImage, BAD_COUNT);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', { timeout: 2000 });
});
`
})).exitCode, 'make sure default comparison fails').toBe(1);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', {
maxDiffPixels: ${Math.floor(BAD_COUNT / 2)},
maxDiffPixelRatio: ${BAD_RATIO},
timeout: 2000,
});
});
`
})).exitCode, 'make sure it fails when maxDiffPixels < actualBadPixels < maxDiffPixelRatio').toBe(1);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', {
maxDiffPixels: ${BAD_COUNT},
maxDiffPixelRatio: ${BAD_RATIO / 2},
timeout: 2000,
});
});
`
})).exitCode, 'make sure it fails when maxDiffPixelRatio < actualBadPixels < maxDiffPixels').toBe(1);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', {
maxDiffPixels: ${BAD_COUNT},
maxDiffPixelRatio: ${BAD_RATIO},
});
});
`
})).exitCode, 'make sure it passes when actualBadPixels < maxDiffPixelRatio && actualBadPixels < maxDiffPixels').toBe(0);
});
test('should respect maxDiffPixelRatio option', async ({ runInlineTest }) => {
const BAD_RATIO = 0.25;
const BAD_PIXELS = IMG_WIDTH * IMG_HEIGHT * BAD_RATIO;
const EXPECTED_SNAPSHOT = paintBlackPixels(whiteImage, BAD_PIXELS);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', { timeout: 2000 });
});
`
})).exitCode, 'make sure default comparison fails').toBe(1);
expect((await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', {
maxDiffPixelRatio: ${BAD_RATIO}
});
});
`
})).exitCode, 'make sure maxDiffPixelRatio option is respected').toBe(0);
expect((await runInlineTest({
...files,
'playwright.config.ts': `
module.exports = { projects: [
{ expect: { toHaveScreenshot: { maxDiffPixelRatio: ${BAD_RATIO} } } },
]};
`,
'a.spec.js-snapshots/snapshot.png': EXPECTED_SNAPSHOT,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png');
});
`
})).exitCode, 'make sure maxDiffPixels option in project config is respected').toBe(0);
});
test('should throw for invalid maxDiffPixels values', async ({ runInlineTest }) => {
expect((await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot({
maxDiffPixels: -1,
});
});
`
})).exitCode).toBe(1);
});
test('should throw for invalid maxDiffPixelRatio values', async ({ runInlineTest }) => {
expect((await runInlineTest({
...files,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot({
maxDiffPixelRatio: 12,
});
});
`
})).exitCode).toBe(1);
});
test('should attach expected/actual and no diff when sizes are different', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'a.spec.js-snapshots/snapshot.png': createImage(2, 2),
'a.spec.js': `
const { test } = require('./helper');
test.afterEach(async ({}, testInfo) => {
console.log('## ' + JSON.stringify(testInfo.attachments));
});
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png', { timeout: 2000 });
});
`
});
expect(result.exitCode).toBe(1);
const outputText = stripAnsi(result.output);
expect(outputText).toContain('Expected an image 2px by 2px, received 1280px by 720px.');
const attachments = outputText.split('\n').filter(l => l.startsWith('## ')).map(l => l.substring(3)).map(l => JSON.parse(l))[0];
for (const attachment of attachments)
attachment.path = attachment.path.replace(/\\/g, '/').replace(/.*test-results\//, '');
expect(attachments).toEqual([
{
name: 'expected',
contentType: 'image/png',
path: 'a-is-a-test/snapshot-expected.png'
},
{
name: 'actual',
contentType: 'image/png',
path: 'a-is-a-test/snapshot-actual.png'
},
]);
});
test('should fail with missing expectations and retries', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'playwright.config.ts': `
module.exports = { retries: 1 };
`,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png');
});
`
});
expect(result.exitCode).toBe(1);
expect(result.failed).toBe(1);
const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(result.output).toContain(`${snapshotOutputPath} is missing in snapshots, writing actual`);
const data = fs.readFileSync(snapshotOutputPath);
expect(pngComparator(data, whiteImage)).toBe(null);
});
test('should update expectations with retries', async ({ runInlineTest }, testInfo) => {
const result = await runInlineTest({
...files,
'playwright.config.ts': `
module.exports = { retries: 1 };
`,
'a.spec.js': `
const { test } = require('./helper');
test('is a test', async ({ page }) => {
await expect(page).toHaveScreenshot('snapshot.png');
});
`
}, { 'update-snapshots': true });
expect(result.exitCode).toBe(0);
expect(result.passed).toBe(1);
const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.png');
expect(result.output).toContain(`${snapshotOutputPath} is missing in snapshots, writing actual`);
const data = fs.readFileSync(snapshotOutputPath);
expect(pngComparator(data, whiteImage)).toBe(null);
});

13
utils/generate_types/overrides-test.d.ts vendored
View file

@ -42,19 +42,6 @@ type ExpectSettings = {
* Default timeout for async expect matchers in milliseconds, defaults to 5000ms.
*/
timeout?: number;
toHaveScreenshot?: {
/** An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between pixels in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
*/
threshold?: number,
/**
* An acceptable amount of pixels that could be different, unset by default.
*/
maxDiffPixels?: number,
/**
* An acceptable ratio of pixels that are different to the total amount of pixels, between `0` and `1` , unset by default.
*/
maxDiffPixelRatio?: number,
}
toMatchSnapshot?: {
/** An acceptable perceived color difference in the [YIQ color space](https://en.wikipedia.org/wiki/YIQ) between pixels in compared images, between zero (strict) and one (lax). Defaults to `0.2`.
*/