Commit graph

148 commits

Author SHA1 Message Date
Pavel Feldman 7bfa6f9b5f
feat(html): group similar items in the report (#11160) 2022-01-03 21:17:17 -08:00
Pavel Feldman a0aeaeb929
test: expose repeatEachIndex (#11158) 2022-01-03 17:29:54 -08:00
Ross Wollman dadb5cbc30
fix(html reporter): fix too much strikethrough in diffs (#11115)
Textual snapshot diffs were previously broken in the HTML Report. The strikethrough'd text extended beyond the intended region.

HTML Report Before: 
<img width="693" alt="Screen Shot 2021-12-27 at 4 43 35 PM" src="https://user-images.githubusercontent.com/11915034/147518750-a60f9002-6eed-48a1-a412-20fabd076fa6.png">

HTML Report After:
<img width="206" alt="Screen Shot 2021-12-27 at 4 48 37 PM" src="https://user-images.githubusercontent.com/11915034/147518762-19a4c8f9-ccc3-4a3c-a962-5a42edc6fc5d.png">

This now matches what's expected and shown in the terminal (which has always been correct):

<img width="1384" alt="Screen Shot 2021-12-27 at 4 36 29 PM" src="https://user-images.githubusercontent.com/11915034/147518799-f538259e-5a45-4d6f-916c-a12ccb620c5b.png">

NB: This MR is a workaround, but not a root cause fix. It works, but I never fully got to the root cause so a bug upstream may be required. It's unclear whether it's (1) in [`colors`](https://www.npmjs.com/package/colors), (2) in [`ansi-to-html`](https://www.npmjs.com/package/ansi-to-html), or (3) Playwright's use of the two. Since the terminal output is correct, I suspect it is in `ansi-to-html`. For example:

```js
const colors = require("colors");
const Convert = require('ansi-to-html');
const convert = new Convert();

// original (strike incorrectly wraps everything in the HTML)
console.log(convert.toHtml(colors.strikethrough("crossed out") + ' ' + colors.red("red")))
// prints: <strike>crossed out <span style="color:#A00">red<span style="color:#FFF"></span></span></strike>

// workaround
console.log(convert.toHtml(colors.reset(colors.strikethrough("crossed out")) + ' ' + colors.red("red")))
// prints: <strike>crossed out</strike> <span style="color:#A00">red<span style="color:#FFF"></span></span>
```

Fixes #11116
2021-12-28 09:56:34 -08:00
Dmitry Gozman f933759ad1
chore(test runner): minor improvements (#11067)
- Types for fixture options and more.
- Refined type for deadline runner.
2021-12-22 09:59:58 -08:00
Dmitry Gozman f5304e3bda
fix(fixtures): await fixture teardown when shutting down the worker (#11033) 2021-12-20 16:19:21 -08:00
Dmitry Gozman 2d00836f0e
fix(test runner): show the location of afterAll timeout (#11007) 2021-12-18 09:32:41 -08:00
Dmitry Gozman c9ba49936f
feat(reporters): show retry #x when running a test (#11003) 2021-12-17 21:07:04 -08:00
Pavel Feldman 8f98074fc8
chore: add blink-diff third party library (#10984) 2021-12-17 15:53:37 -08:00
Dmitry Gozman b6aad54b9f
fix(test runner): test.setTimeout whould not break debugging (#11004)
We ignore test.setTimeout() when timeout is already zero for debugging.
2021-12-17 15:17:48 -08:00
Dmitry Gozman f5780be41b
fix(list reporter): make sure that duration suffix survives truncation (#11002) 2021-12-17 13:08:02 -08:00
Dmitry Gozman 93ad12978c
fix(test runner): disable expect, action and navigation timeouts on debug (#10958)
We disable these timeouts when test timeout is set to zero.
This covers PWDEBUG=1, --debug and manual `test.setTimeout(0)` scenarios.
2021-12-16 18:32:46 -08:00
Dmitry Gozman 192071d5bc
feat(test runner): save traces for beforeAll/afterAll hooks (#10950) 2021-12-15 16:06:10 -08:00
Dmitry Gozman 19b08332ce
feat(test runner): disable test timeout on page.pause() call (#10944) 2021-12-15 11:12:52 -08:00
Dmitry Gozman 0d54afab9c
feat(test runner): show beforeAll/afterAll hooks similar to tests (#10923)
Reporters now get notified about hooks start/end/steps.
2021-12-15 10:39:49 -08:00
Pavel Feldman f579f9c806
chore: parse tsx tests (#10917) 2021-12-14 19:25:07 -08:00
Dmitry Gozman 34b84841b0
chore(test runner): create TestResult instances lazily in dispatcher (#10921)
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
2021-12-14 14:10:56 -08:00
Dmitry Gozman 5c4ebdce54
fix(line reporter): clarify about retries when going over total counter (#10901) 2021-12-13 19:06:13 -08:00
Joel Einbinder c27491cd4d
feat(test-runner): shorten long output paths (#10523) 2021-12-13 10:56:03 -08:00
Joel Einbinder 7a02c52144
feat(test-runner): specific playwright types for expect (#10670) 2021-12-13 13:42:36 -05:00
Dmitry Gozman 308c7b4e32
docs: update test advanced guides (#10861)
Linking to the API reference, using better examples and newer docs.
2021-12-10 11:15:01 -08:00
Dmitry Gozman 897e41c6c1
docs: document TestOptions.video.size (#10827) 2021-12-09 07:38:58 -08:00
Pavel Feldman b5933db279
feat(tsconfig): respect baseUrl and paths from tsconfig (#10525) 2021-12-08 22:43:00 -08:00
Pavel Feldman 4d683cef7f
fix(html): render text attachments as text (#10778) 2021-12-08 08:51:44 -08:00
Pavel Feldman a08a41f6c9
chore: render annotations in html report (#10774) 2021-12-07 18:35:06 -08:00
Pavel Feldman feb4c62da1
fix(html): html reporter fixes (#10770) 2021-12-07 16:47:47 -08:00
Pavel Feldman 7765131a14
feat(acceptDownload): revert acceptDownload (#10709) 2021-12-06 09:25:24 -08:00
Pavel Feldman 518d67add5
feat(test.info): expose information on the currently running test (#10708) 2021-12-06 09:25:11 -08:00
Dmitry Gozman 98e2f40bb0
docs: replace TestCase.suite with TestCase.parent (#10687)
It is there by mistake.
2021-12-02 14:24:43 -08:00
Pavel Feldman 31e0a63fcd
feat(toBeChecked): allow passing checked: false (#10665) 2021-12-02 10:31:26 -08:00
Pavel Feldman b9731a904e
chore: add validations into check_deps (#10661) 2021-12-01 18:14:13 -08:00
Saransh Miglani f05252874a
chore: minor code rearrangement (#10650) 2021-12-01 09:18:16 -08:00
Yury Semikhatsky d66b7aab3b
feat(expext): toBeOK for APIResponse (#10596) 2021-11-30 18:12:19 -08:00
Dmitry Gozman 729da65eba
fix(test runner): allow multiple missing snapshots per test (#10621)
Instead of failing right away, continue test execution but mark
the test as failed.
2021-11-30 17:50:19 -08:00
Max Schmitt 3337920c76
docs: clarify webServer ipv4/ipv6 and baseURL protocol (#10610) 2021-11-30 20:01:28 +01:00
Max Schmitt 682d1f41c7
fix(test-runner): wrong test-runner options got determined (#10607)
Follow-up for #10507
2021-11-30 19:59:35 +01:00
Max Schmitt 9337aa3cdf
fix(test-runner): webServer port detection on Node.js 17 (#10538) 2021-11-29 19:36:35 +01:00
Joel Einbinder 15053d9b3b
fix(test-runner): apply default options to all browserTypes (#10507) 2021-11-29 12:21:15 -05:00
Max Schmitt 293c233a49
docs(test-runner): add webServer (#10531) 2021-11-29 18:01:14 +01:00
Joel Einbinder 20c0facfb9
fix(test runner): dont mangle test names with multiple dashes (#10447) 2021-11-24 19:36:38 -05:00
Pavel Feldman 5d19f16601
feat(esm): introduce experimental PW_EXPERIMENTAL_TS_ESM option (#10519) 2021-11-24 14:17:01 -08:00
Pavel Feldman 7eb3f76f49
feat(esm): allow running tests in type module projects (#10503) 2021-11-24 12:42:48 -08:00
Max Schmitt 85197e68c9
chore: support range requests in trace viewer http server (#10434) 2021-11-23 21:37:55 +01:00
Ross Wollman 854f321532
feat(api): add explicit async testInfo.attach (#10121)
feat(api): add explicit async testInfo.attach

We add an explicit async API for attaching file paths (and Buffers) to
tests that can be awaited to help users ensure they are attaching files
that actually exist at both the time of the invocation and later when
reporters (like the HTML Reporter) run and package up test artifacts.

This is intended to help surface attachment issues as soon as possible
so you aren't silently left with a missing attachment
minutes/days/months later when you go to debug a suddenly breaking test
expecting an attachment to be there.

NB: The current implemntation incurs an extra file copy compared to
manipulating the raw attachments array. If users encounter performance
issues because of this, we can consider an option parameter that uses
rename under the hood instead of copy. However, that would need to be
used with care if the file were to be accessed later in the test.
2021-11-23 09:30:53 -08:00
Dmitry Gozman e647f0420c
docs: add more references to TestInfo.retry (#10472) 2021-11-22 10:06:20 -08:00
Dmitry Gozman fde2f6a77f
docs: separate doc for test timeouts (#10448) 2021-11-19 17:06:46 -08:00
Dmitry Gozman 4eaeb3b59c
docs: explain that beforeAll/afterAll run again in the new worker process (#10446) 2021-11-19 13:47:30 -08:00
Dmitry Gozman 0302e759df
feat(test runner): allow top-level test.fixme similar to test.skip (#10250)
```js
test.fixme('my test name', () => {});
```
2021-11-19 11:40:40 -08:00
Dmitry Gozman d9f849fb14
feat(test runner): replace declare/define with "options" (#10293)
1. Fixtures defined in test.extend() can now have `{ option: true }` configuration that makes them overridable in the config. Options support all other properties of fixtures - value/function, scope, auto.
```
const test = base.extend<MyOptions>({
  foo: ['default', { option: true }],
});
```

2. test.declare() and project.define are removed.

3. project.use applies overrides to default option values and nothing else. Any test.extend() and test.use() calls take priority over config options.

Required user changes: if someone used to define fixture options with test.extend(), overriding them in config will stop working. The solution is to add `{ option: true }`.

```
// Old code
export const test = base.extend<{ myOption: number, myFixture: number }>({
  myOption: 123,
  myFixture: ({ myOption }, use) => use(2 * myOption),
});

// New code
export const test = base.extend<{ myOption: number, myFixture: number }>({
  myOption: [123, { option: true }],
  myFixture: ({ myOption }, use) => use(2 * myOption),
});
```
2021-11-18 15:45:52 -08:00
Dmitry Gozman 8f43f4c98f
feat(serial): better errors from beforeAll (#10419)
When beforeAll hook times out or fails with an exception, we now
close the context and show a nice error.
2021-11-18 14:36:55 -08:00
Dmitry Gozman bd93fc499f
fix(html reporter): show missing attachments as warnings (#10400) 2021-11-17 18:03:13 -08:00