Commit graph

127 commits

Author SHA1 Message Date
Andrey Lushnikov 0f4090472c
feat: add maskColor option to the toHaveScreenshot method (#23555) 2023-06-06 17:15:55 -07:00
Dmitry Gozman 6b687b6d69
fix(toHaveScreenshot): attach newly created missing expectations (#23528)
Fixes #23090.
2023-06-05 14:57:06 -07:00
Pavel Feldman 84942aa992
chore: render successful toPass as such (#23411)
Fixes https://github.com/microsoft/playwright/issues/23302
2023-06-01 13:22:08 -07:00
Pavel Feldman 04070a59e1
chore: remove expect.configure({poll}) (#23060) 2023-05-16 18:45:03 -07:00
Dmitry Gozman f469e4b1eb
fix(expect): mark step as failed when async custom matcher throws (#23035)
Fixes #23021.
2023-05-15 19:37:12 -07:00
Pavel Feldman 083d13a13d
chore: surface syntax error in ui mode (#22982)
Fixes https://github.com/microsoft/playwright/issues/22863
2023-05-12 14:23:22 -07:00
Pavel Feldman efad19b332
chore: render test steps in the trace (#22837) 2023-05-05 15:12:18 -07:00
Pavel Feldman 7fdd7a20fb
chore: fix soft after poll (#22642) 2023-04-26 08:50:10 -07:00
Pavel Feldman d34c4e99f5
chore: add step attachments into the expect trace event (#22543) 2023-04-25 17:38:12 -07:00
Pavel Feldman a1007bbe2c
chore: introduce expect.configure (#22533) 2023-04-25 10:29:56 -07:00
Pavel Feldman 17df03c043
chore: better zone stack preservation in expect (#22563) 2023-04-21 16:13:35 -07:00
Pavel Feldman bbc47ba315
chore: ensure web assertions are merged in trace (#22544) 2023-04-21 10:07:23 -07:00
Pavel Feldman d95268ffc0
chore: remove refined title (#22541) 2023-04-20 20:34:07 -07:00
Pavel Feldman e7b9c08833
chore: should not spill toPass between tests (#22473) 2023-04-19 14:45:58 -07:00
Pavel Feldman c5aeab4d7e
revert(20509, 20596): expect.toPass is broken with these (#22254)
Reverts https://github.com/microsoft/playwright/pull/20509 and
https://github.com/microsoft/playwright/pull/20596
Fixes #22215
2023-04-10 17:39:49 -07:00
Pavel Feldman a42567d549
chore: inside out the config & project internal (#22260) 2023-04-07 09:54:01 -07:00
Pavel Feldman b1fdf0bcb6
chore: nest test steps based on zones (#22108)
Fixes: #21423
2023-03-30 21:05:07 -07:00
Dmitry Gozman 6929214dbf
feat(expect): expect(locator).toBeAttached() (#22067)
Fixes #13467.
2023-03-29 11:09:17 -07:00
Pavel Feldman c45d8749b0
chore: split trace events into phases (#21696) 2023-03-15 22:33:40 -07:00
Pavel Feldman de3a5e2a91
chore(trace): include expect steps in a trace (#21199) 2023-02-28 13:26:23 -08:00
Dmitry Gozman 27027658dc
chore: fix all Proxy() to account for symbol properties (#21272)
Fixes #20940.
2023-02-28 12:45:14 -08:00
Pavel Feldman 09f77c41dd
chore: migrate to protocol's StackFrame type (#21198) 2023-02-24 18:36:15 -08:00
Pavel Feldman 3fa19e80ad
chore: wrap expect call in a zone (#21191) 2023-02-24 12:17:03 -08:00
Pavel Feldman 8ccc02a9d3
chore: use client wall time in traces (#21163) 2023-02-23 14:37:53 -08:00
Pavel Feldman 3f8f2a0fdd
chore: do not use library stack capturer in test runner (#21075) 2023-02-21 19:24:17 -08:00
Pavel Feldman 06fc72b6ed
revert(#12706): also fix related bugs it introduced (#21070) 2023-02-21 14:15:11 -08:00
Andrey Lushnikov 1db04f2839
fix: simplify protocol for toBeInViewport (#20960) 2023-02-16 11:02:19 -08:00
Andrey Lushnikov fdcd7b549d
chore: mark comparator option back as experimental (#20816)
This reverts commit 303c5998f8.

Reason for revert: I tried enabling `ssim-cie94` by default on
ionic-framework test suite, and it proves to be overly strict for their
usecase.
2023-02-13 11:11:44 -08:00
Andrey Lushnikov 68e170ef89
feat: implement expect(locator).toBeInViewport() (#20668)
The method accepts a `ratio` option to assert the ratio
of the element in viewport. `ratio` defaults to `Number.MIN_VALUE`.

NOTE: this reverts commit d950f5b6ee and
adds `ratio` option support + does the rename.

Fixes #8740
2023-02-10 04:33:22 -08:00
Dmitry Gozman ab4cd83e44
chore: move testInfo and timeoutManager to worker/ (#20803) 2023-02-09 19:22:17 -08:00
Andrey Lushnikov 303c5998f8
feat: release "comparator" option from experiment (#20720)
The option defines a comparator to be used to compare images.
Possible values are `"pixelmatch"` and `"ssim-cie94"`.

Note: This reverts commit 8167f8bf54.
2023-02-07 08:51:48 -08:00
Andrey Lushnikov f45f20a8c9
fix: handle not.toPass() with expect.soft inside (#20596)
Fixes #20518
2023-02-03 09:14:05 -08:00
Andrey Lushnikov 4f61105032
fix: swallow expect.soft errors inside successful toPass matcher (#20509)
Fixes #20437
2023-02-02 10:47:20 -08:00
Pavel Feldman cb9ace6035
chore(test): move run options into config (#20568) 2023-02-01 15:25:26 -08:00
Yury Semikhatsky a03f3223c4
fix: trim long expect messages (#20517)
Fixes #20324
2023-01-30 17:24:12 -08:00
Pavel Feldman b91bb1af9a
chore: minimize configLoader use (#20431) 2023-01-27 12:44:15 -08:00
Pavel Feldman 7d7e66f450
chore: group test runner files by process (#20425) 2023-01-26 17:26:47 -08:00
Dmitry Gozman eafa6fda13
fix(expect): poll/toPass should not wait over specified timeout (#20266)
Drive-by: unflake some timeout-dependent tests.
2023-01-20 15:47:24 -08:00
Dmitry Gozman d950f5b6ee
chore: remove toIntersectViewport for the next release (#20232)
Mostly reverts #19901.
2023-01-19 15:04:09 -08:00
Pavel Feldman d1c161ce99
chore: merge the util exports (#20110) 2023-01-13 13:50:38 -08:00
Andrey Lushnikov 2a49c5e498
feat(expect): introduce expect(locator).toIntersectViewport() (#19901)
This is a new web-first assertion that should be used like this:

```ts
test('should work', async ({ page }) => {
  const locator = page.locator('body');
  // New web-first assertion.
  await expect(locator).toIntersectViewport();
  // The same functionality.
  await expect.poll(() => locator.viewportRatio()).toBeGreaterThan(0);
});
```

Fixes #8740
2023-01-06 16:56:24 -08:00
Andrey Lushnikov 1afa38d5a7
chore(expect): extract polling from expect.poll and expect().toPass (#19882)
This extracts & unifies polling machinery from `expect.poll` and
`expect.toPass` methods.
2023-01-05 11:14:37 -08:00
Andrey Lushnikov d7e7cab44a
fix: properly handle negated timed-out toPass matcher (#19580) 2022-12-20 08:41:32 -08:00
Pavel Feldman 0e2732decf
feat(api): introduce expect().toPass (#19463) 2022-12-14 19:23:13 -08:00
Andrey Lushnikov 8167f8bf54
chore: hide "comparator" option from documentation and types (#19441) 2022-12-14 09:58:19 -08:00
Andrey Lushnikov 6d82460a02
feat: implement a new image comparison function (#19166)
This patch implements a new image comparison function, codenamed
"ssim-cie94". The goal of the new comparison function is to cancel out
browser non-determenistic rendering.

To use the new comparison function:

```ts
await expect(page).toHaveScreenshot({
  comparator: 'ssim-cie94',
});
```

As of Nov 30, 2022, we identified the following sources of
non-determenistic rendering for Chromium:
- Anti-aliasing for certain shapes might be different due to the
  way skia rasterizes certain shapes.
- Color blending might be different on `x86` and `aarch64`
architectures.

The new function employs a few heuristics to fight these
differences.

Consider two non-equal image pixels `(r1, g1, b1)` and `(r2, g2, b2)`:
1. If the [CIE94] metric is less then 1.0, then we consider these pixels
   **EQUAL**. (The value `1.0` is the [just-noticeable difference] for
   [CIE94].). Otherwise, proceed to next step.
1. If all the 8 neighbors of the first pixel match its color, or
   if the 8 neighbors of the second pixel match its color, then these
   pixels are **DIFFERENT**. (In case of anti-aliasing, some of the
   direct neighbors have to be blended up or down.) Otherwise, proceed
   to next step.
1. If SSIM in some locality around the different pixels is more than
   0.99, then consider this pixels to be **EQUAL**. Otherwise, mark them
   as **DIFFERENT**. (Local SSIM for anti-aliased pixels turns out to be
   very close to 1.0).

[CIE94]: https://en.wikipedia.org/wiki/Color_difference#CIE94
[just-noticeable difference]:
https://en.wikipedia.org/wiki/Just-noticeable_difference
2022-12-02 15:22:05 -08:00
Shubham Kanodia 3d804ff7cd
chore: rewrite error message for missing snapshot (#19104) 2022-11-29 12:51:15 -08:00
Dmitry Gozman 4f72a895e9
fix(inspector): render expect.not correctly (#19125)
Also fixes the same in expect logs.

References #19083.
2022-11-28 20:50:16 -08:00
Pavel Feldman 4e58b0c2ea
chore: render timed out error message when expect timeouts (#18863)
Fixes https://github.com/microsoft/playwright/issues/18859
2022-11-16 17:00:42 -08:00
Andrey Lushnikov 6d491f928d
feat(playwright-test): introduce snapshotPathTemplate configuration (#18568)
This configuration option allows to set a string with template
values for precise control over snapshot path location.

An example of `snapshotPathTemplate` usage:

```ts
// playwright.config.ts
// Notice the `testDir` configuration!
export default {
  testDir: './tests',
  snapshotPathTemplate: './__screenshots__/{platform}/{projectName}/{testFilePath}/{arg}{ext}',
}
```

Currently supported "magic tokens" inside the `snapshotPathTemplate`
are:

- `{testDir}` - project's `testDir`
- `{snapshotDir}` - project's `snapshotDir`
- `{platform}` - `process.platform`
- `{projectName}` - Project's sanitized name
- `{testFileDir}` - Directories in relative path from `testDir` to test
  file path (e.g. `page/` in the example below)
- `{testFileName}` - Test file name (with extension) (e.g.
  `page-click.spec.ts` in the example below)
- `{testFilePath}` - Relative path from `testDir` to test file path
  (e.g. `page/page-click.spec.ts` in the example below)
- `{ext}` - snapshot extension (with dots)
- `{arg}` - joined snapshot name parts, without extension (e.g.
`foo/bar/baz` in the example below)
- `{snapshotSuffix}` - `testInfo.snapshotSuffix` value.

Consider the following file structure:

```
playwright.config.ts
tests/
└── page/
    └── page-click.spec.ts
```

The following `page-click.spec.ts`:

```ts
// page-click.spec.ts
import { test, expect } from '@playwright/test';

test('should work', async ({ page }) => {
  await expect(page).toHaveScreenshot(['foo', 'bar', 'baz.png']);
});
```

Fixes #7792
2022-11-09 15:29:07 -08:00