Pavel Feldman
4bcca2c87e
chore: add a parallel mode note ( #11943 )
2022-02-08 14:36:14 -08:00
Dmitry Gozman
8a5c93436d
fix(reporter): do not report parallel tests as slow ( #11921 )
2022-02-07 20:10:13 -08:00
Dmitry Gozman
7912c515a3
fix(fixtures): account for default options being undefined ( #11916 )
2022-02-07 17:11:36 -08:00
Pavel Feldman
9116adc684
chore: allow opt-into the legacy global setup mode ( #11888 )
2022-02-07 10:41:56 -08:00
Yury Semikhatsky
46dfa45b4e
docs: fix parallel test docs build ( #11877 )
2022-02-04 16:44:04 -08:00
Yury Semikhatsky
afc4b505aa
docs: fix duplicate sections ( #11874 )
2022-02-04 13:50:46 -08:00
Dmitry Gozman
d9a8bb057d
fix(test-fail): allow unhandled expects in test.fail ( #11850 )
...
Previously, we would consider this a worker error, but
we make an exception for "expect()" calls.
2022-02-03 17:14:12 -08:00
Pavel Feldman
72424dc904
chore: allow setting reporter via env ( #11848 )
2022-02-03 16:10:39 -08:00
Pavel Feldman
fdda759a9d
feat(parallel): allow setting enclosing scope parallel mode ( #11822 )
2022-02-02 20:44:11 -08:00
Andrey Lushnikov
ba0c7e679b
feat(test-runner): support expect.soft ( #11800 )
...
Soft expects will still fail the test, but will not abort it's execution. As a consequence of this:
- `TestResult` now might have multiple errors, which is reflected with a new `testResult.erros: TestError[]` field.
- `TestInfo` now might have multiple errors as well, which is reflected with a new `testInfo.errors: TestError[]` field.
Fixes #7819
2022-02-02 18:33:51 -08:00
Max Schmitt
c752b28516
chore: pin production dependencies ( #11793 )
2022-02-02 11:14:41 +01:00
Andrey Lushnikov
b0daa7754f
feat: filter stack traces to exclude test runner frames ( #11795 )
...
Before:
```bash
Running 1 test using 1 worker
1) [chromium] › tests/example.spec.ts:3:1 › should work ==========================================
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
2 |
3 | test('should work', async({page}) => {
> 4 | expect(1).toBe(2);
| ^
5 | });
6 |
at Proxy.<anonymous> (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/expect.ts:151:30)
at /Users/andreylushnikov/tmp/tests/example.spec.ts:4:13
at /Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:335:13
at runNextTicks (node:internal/process/task_queues:61:5)
at processImmediate (node:internal/timers:437:9)
at TestInfoImpl._runFn (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/testInfo.ts:164:7)
at WorkerRunner._runTestWithBeforeHooks (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:317:24)
at TimeoutRunner.run (/Users/andreylushnikov/prog/playwright/packages/playwright-core/src/utils/async.ts:48:14)
at TestInfoImpl._runWithTimeout (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/testInfo.ts:151:7)
at WorkerRunner._runTestOrAllHook (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:276:5)
at WorkerRunner._runSuite (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:190:11)
at WorkerRunner.run (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:137:9)
at process.<anonymous> (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/worker.ts:87:5)
```
after:
```
Running 1 test using 1 worker
1) [chromium] › tests/example.spec.ts:3:1 › should work ==========================================
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
2 |
3 | test('should work', async({page}) => {
> 4 | expect(1).toBe(2);
| ^
5 | });
6 |
at /Users/andreylushnikov/tmp/tests/example.spec.ts:4:13
```
2022-02-01 18:40:44 -08:00
Pavel Feldman
ceb3027bdf
chore(test-runner): force colors in worker processes ( #11799 )
2022-02-01 17:29:15 -08:00
Pavel Feldman
6e2fcc4700
chore: do not expose suite load error ( #11797 )
2022-02-01 15:34:16 -08:00
Pavel Feldman
f875ebe730
fix(test-runner): respect source maps when reporting test files ( #11770 )
2022-02-01 13:04:54 -08:00
Max Schmitt
028afb167b
chore: bump production dependencies ( #11787 )
2022-02-01 21:27:34 +01:00
Pavel Feldman
a3bc911a3d
fix(debug): don't apply infinite timeouts when using debugger ( #11785 )
2022-02-01 11:51:37 -08:00
Caio Agiani
b6b60decdf
fix: typos ( #11789 )
2022-02-01 11:09:41 -08:00
Pavel Feldman
3a5e8184b5
fix(html-reporter): open tests from required file ( #11784 )
2022-02-01 11:01:52 -08:00
Pavel Feldman
c82f2641d7
fet(list-files): report per-project test dir and filters ( #11764 )
2022-02-01 08:08:56 -08:00
Pavel Feldman
c5d852f1bb
fix(test-runner): escape backslashes in win cli ( #11763 )
2022-01-31 17:48:05 -08:00
Andrey Lushnikov
2336692e8a
feat: support clarification message for expect ( #11735 )
...
The clarification message is displayed in the HTML report as the name of the step:
It is also shown in terminal output:
Fixes #7816
2022-01-31 17:14:59 -08:00
Pavel Feldman
2b55adaafa
feat(breaking): always report onBegin/onEnd, report file errors ( #11758 )
2022-01-31 17:09:04 -08:00
Andrey Lushnikov
3a4e506479
chore: extract SigIntWatcher ( #11749 )
...
This is to reduce the size of the long `_run` method in the `runner.ts`.
It also might come handy around the codebase.
2022-01-31 08:51:22 -08:00
Andrey Lushnikov
b58b004f0f
fix: run reporter.onEnd after webserver teardown ( #11712 )
...
Fixes #11647
2022-01-31 06:19:33 -08:00
Dmitry Gozman
eeebcd53ae
chore(test runner): extract TestInfoImpl ( #11725 )
2022-01-28 17:39:42 -08:00
Dmitry Gozman
30f3cbd6da
fix(fixtures): make sure defaultLaunchOptions are always there ( #11713 )
2022-01-28 10:51:55 -08:00
Pavel Feldman
29a84b2df8
chore: allow pre-processing scripts ( #11702 )
2022-01-27 14:32:23 -08:00
Feng Yu
0698254253
docs: fix broken link ( #11687 )
...
fix #11680
2022-01-27 13:00:09 -08:00
Dmitry Gozman
b1fbc4fdbe
fix(test runner): resolve tsconfig for each file ( #11662 )
...
This allows us to properly handle path mappings
that are not too ambiguous.
2022-01-26 18:28:42 -08:00
divdavem
512a245f13
feat(test-runner): wait for a url before starting tests ( #10138 )
...
The webServer configuration in @playwright/test now accepts a url as an
alternative to a port number to wait for a url to return a 2xx status code.
2022-01-26 16:32:58 -08:00
Dmitry Gozman
19820de7a9
fix(test runner): correctly save videos when running remotely ( #11633 )
2022-01-26 07:43:07 -08:00
Dmitry Gozman
4f42484c64
chore(test runner): simplify runTestWithBeforeHooks ( #11629 )
2022-01-25 16:46:50 -08:00
Dmitry Gozman
800b813d4b
chore(test runner): prepare to per-fixture timeout ( #11605 )
...
This reworks DeadlineRunner to use exception to signal timeout. This way,
we'll be able to run fixtures against a shared deadline vs their own
deadline and still get an easy control-flow timeout handling.
2022-01-25 11:22:28 -08:00
Andrey Lushnikov
42876a0528
fix: support PWDEBUG=0 to disable debug ( #11611 )
...
Fixes #11606
2022-01-25 08:40:24 -08:00
Ross Wollman
64e7557fb9
fix: falsey behavior in route.continue, page.post, testInfo.attach ( #11421 )
...
In several of the Playwright APIs, falsey values were not handled correctly. This changeset adds tests (and some fixes):
- route.continue: If options.postData was the empty string, the continue failed to override the post data.
- page.post (application/json with options.data: false|''|0|null): Raw falsey values were getting dropped (i.e. you can't do the equivalent of curl --header application/json … -d 'false'). This has been fixed with most values across all browsers, but an additional fix is needed for 'null' which the channel serializer treats extra specially.
- testInfo.attach: This didn't get reported as an error when options.path was the empty string, but should have been.
#11413 (and its fix #11414 ) inspired this search as they are the same
class of bug.
2022-01-24 15:06:36 -08:00
Pavel Feldman
9542b007cf
fix(esm): don't emit module shorthand warning ( #11596 )
2022-01-24 11:22:56 -08:00
Pavel Feldman
0b99d83911
chore: include config dir in quick list report ( #11563 )
2022-01-22 17:02:46 -08:00
Pavel Feldman
baf0e5b47a
chore: implement lightweight test listing ( #11551 )
2022-01-21 19:11:22 -08:00
Pavel Feldman
6b3e596fd8
fix(baseurl): support path-less baseurl ( #11527 )
2022-01-20 18:11:56 -08:00
Pavel Feldman
50fa839e64
chore: include transpilation mode in the compute cache hash ( #11521 )
2022-01-20 13:33:40 -08:00
Ross Wollman
33f32368e5
fix(test runner): dispatcher transformation of empty attachments ( #11414 )
2022-01-17 18:47:24 -08:00
Pavel Feldman
8ecf581b45
chore: focus by line should override only ( #11427 )
2022-01-16 08:47:09 -08:00
Pavel Feldman
8b0ef4ad6b
chore: introduce oop testing mode for debugging ( #11428 )
2022-01-15 19:25:32 -08:00
Max Schmitt
badb5b4d13
chore(pw-test): forward web-server stdout to debug target ( #11384 )
2022-01-13 23:55:46 +01:00
Dmitry Gozman
9d5bf0e90d
fix(test runner): fixture teardown double error, testInfo.attach() ( #11365 )
...
- Use file path, not content to calculate the attachment hash.
- Always cleanup fixture from the list on teardown, to avoid reporting
teardown error multiple times: from the test, and from the cleanup.
2022-01-13 10:38:47 -08:00
Pavel Feldman
feb7148b3f
chore: don't write into stdout from the test runner, use reporters instead ( #11367 )
2022-01-12 19:52:40 -08:00
Andrey Lushnikov
9285596806
chore: cut v1.18.0 ( #11358 )
...
Drive-by: fix the `//utils/bump_package_versions.js` script.
2022-01-12 11:50:44 -08:00
Andrey Lushnikov
6f932fcb4a
fix: default list reporter should dump stderr to stderr ( #11351 )
...
This is consistent with all other reporters we have in place.
2022-01-12 07:43:25 -08:00
Dmitry Gozman
4efb30999f
feat(request): show request context methods as steps ( #11337 )
2022-01-11 17:33:41 -08:00
Andrey Lushnikov
2a0930c7a4
docs: clarification about trailing slash in docs ( #11326 )
...
Fixes #10557
2022-01-11 07:50:58 -08:00
Dmitry Gozman
da1f39fb29
chore(test runner): refactor worker runner parts ( #11316 )
...
This will make it easier to change in the future.
2022-01-10 20:25:56 -08:00
Dmitry Gozman
d36ff8a96c
fix(test runner): remove test output for failures ( #11308 )
2022-01-10 13:22:09 -08:00
Dmitry Gozman
14fd837e94
fix(test runner): hide beforeAll/afterAll hooks from the reporter api ( #11306 )
...
This api is not ready yet.
2022-01-10 12:09:51 -08:00
Andrey Lushnikov
7b1fecc009
fix: default to en-US locale in all browsers in Playwright Test ( #11133 )
...
It makes sense to default to en-US by default across all our browsers in Playwright Test.
Fixes #11127
2022-01-10 08:37:55 -08:00
Michaël De Boey
ff5f75dc10
fix: pin colors to 1.4.0 ( #11274 )
2022-01-09 16:18:21 -08:00
Dmitry Gozman
058f98d3dd
fix(test runner): revert error location from top frame ( #11250 )
...
This does not play nicely with some internal Playwright errors, so
it needs more tweaks.
2022-01-07 11:06:47 -08:00
Dmitry Gozman
19460d50e4
chore(test runner): process-wide cache of required test files ( #11230 )
...
This is a preparation to TestRunner api, where we should be
able to run tests multiple times in the same runner process.
2022-01-06 15:38:10 -08:00
Pavel Feldman
373042ed89
chore: don't allow importing @playwright/test twice ( #11218 )
2022-01-06 09:29:05 -08:00
Dmitry Gozman
3ecac56cc0
fix(test runner): testInfo.attach api review changes ( #11211 )
...
Remove overload, require name, merge options.
2022-01-05 16:39:33 -08:00
Dmitry Gozman
f77c874e8a
feat(test runner): make _extendTest experimental ( #11210 )
...
Hidden from types and docs.
2022-01-05 15:54:00 -08:00
Dmitry Gozman
ccc61e31ea
chore(test runner): preparation for TestRunner api ( #11209 )
2022-01-05 15:49:01 -08:00
Dmitry Gozman
3839917eb6
chore(test runner): move config loading to runner ( #11186 )
...
In preparation for the TestRunner api.
2022-01-05 13:44:29 -08:00
Dmitry Gozman
16a779a5ff
fix(test runner): show codeframe and location from the error top stack frame ( #11179 )
...
Previously, reporter would look for a stack frame directly in the test file.
Often times, that is not a top stack frame, especially when the test uses
some helper functions.
This changes error snippets and locations to use the top frame. When top
frame does not match the test file, we additionally show the location
to avoid confusion:
```
1) a.spec.ts:7:7 › foobar ========================================================================
Error: oh my
at helper.ts:5
3 |
4 | export function ohMy() {
> 5 | throw new Error('oh my');
| ^
6 | }
7 |
at ohMy (.../reporter-base-should-print-codeframe-from-a-helper/helper.ts:5:15)
at .../reporter-base-should-print-codeframe-from-a-helper/a.spec.ts:8:9
at FixtureRunner.resolveParametersAndRunHookOrTest (.../src/fixtures.ts:281:12)
```
2022-01-04 16:00:55 -08:00
Pavel Feldman
7bfa6f9b5f
feat(html): group similar items in the report ( #11160 )
2022-01-03 21:17:17 -08:00
Pavel Feldman
a0aeaeb929
test: expose repeatEachIndex ( #11158 )
2022-01-03 17:29:54 -08:00
Ross Wollman
dadb5cbc30
fix(html reporter): fix too much strikethrough in diffs ( #11115 )
...
Textual snapshot diffs were previously broken in the HTML Report. The strikethrough'd text extended beyond the intended region.
HTML Report Before:
<img width="693" alt="Screen Shot 2021-12-27 at 4 43 35 PM" src="https://user-images.githubusercontent.com/11915034/147518750-a60f9002-6eed-48a1-a412-20fabd076fa6.png ">
HTML Report After:
<img width="206" alt="Screen Shot 2021-12-27 at 4 48 37 PM" src="https://user-images.githubusercontent.com/11915034/147518762-19a4c8f9-ccc3-4a3c-a962-5a42edc6fc5d.png ">
This now matches what's expected and shown in the terminal (which has always been correct):
<img width="1384" alt="Screen Shot 2021-12-27 at 4 36 29 PM" src="https://user-images.githubusercontent.com/11915034/147518799-f538259e-5a45-4d6f-916c-a12ccb620c5b.png ">
NB: This MR is a workaround, but not a root cause fix. It works, but I never fully got to the root cause so a bug upstream may be required. It's unclear whether it's (1) in [`colors`](https://www.npmjs.com/package/colors ), (2) in [`ansi-to-html`](https://www.npmjs.com/package/ansi-to-html ), or (3) Playwright's use of the two. Since the terminal output is correct, I suspect it is in `ansi-to-html`. For example:
```js
const colors = require("colors");
const Convert = require('ansi-to-html');
const convert = new Convert();
// original (strike incorrectly wraps everything in the HTML)
console.log(convert.toHtml(colors.strikethrough("crossed out") + ' ' + colors.red("red")))
// prints: <strike>crossed out <span style="color:#A00">red<span style="color:#FFF"></span></span></strike>
// workaround
console.log(convert.toHtml(colors.reset(colors.strikethrough("crossed out")) + ' ' + colors.red("red")))
// prints: <strike>crossed out</strike> <span style="color:#A00">red<span style="color:#FFF"></span></span>
```
Fixes #11116
2021-12-28 09:56:34 -08:00
Dmitry Gozman
f933759ad1
chore(test runner): minor improvements ( #11067 )
...
- Types for fixture options and more.
- Refined type for deadline runner.
2021-12-22 09:59:58 -08:00
Dmitry Gozman
f5304e3bda
fix(fixtures): await fixture teardown when shutting down the worker ( #11033 )
2021-12-20 16:19:21 -08:00
Dmitry Gozman
2d00836f0e
fix(test runner): show the location of afterAll timeout ( #11007 )
2021-12-18 09:32:41 -08:00
Dmitry Gozman
c9ba49936f
feat(reporters): show retry #x when running a test ( #11003 )
2021-12-17 21:07:04 -08:00
Pavel Feldman
8f98074fc8
chore: add blink-diff third party library ( #10984 )
2021-12-17 15:53:37 -08:00
Dmitry Gozman
b6aad54b9f
fix(test runner): test.setTimeout whould not break debugging ( #11004 )
...
We ignore test.setTimeout() when timeout is already zero for debugging.
2021-12-17 15:17:48 -08:00
Dmitry Gozman
f5780be41b
fix(list reporter): make sure that duration suffix survives truncation ( #11002 )
2021-12-17 13:08:02 -08:00
Dmitry Gozman
93ad12978c
fix(test runner): disable expect, action and navigation timeouts on debug ( #10958 )
...
We disable these timeouts when test timeout is set to zero.
This covers PWDEBUG=1, --debug and manual `test.setTimeout(0)` scenarios.
2021-12-16 18:32:46 -08:00
Dmitry Gozman
192071d5bc
feat(test runner): save traces for beforeAll/afterAll hooks ( #10950 )
2021-12-15 16:06:10 -08:00
Dmitry Gozman
19b08332ce
feat(test runner): disable test timeout on page.pause() call ( #10944 )
2021-12-15 11:12:52 -08:00
Dmitry Gozman
0d54afab9c
feat(test runner): show beforeAll/afterAll hooks similar to tests ( #10923 )
...
Reporters now get notified about hooks start/end/steps.
2021-12-15 10:39:49 -08:00
Pavel Feldman
f579f9c806
chore: parse tsx tests ( #10917 )
2021-12-14 19:25:07 -08:00
Dmitry Gozman
34b84841b0
chore(test runner): create TestResult instances lazily in dispatcher ( #10921 )
...
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
2021-12-14 14:10:56 -08:00
Dmitry Gozman
5c4ebdce54
fix(line reporter): clarify about retries when going over total counter ( #10901 )
2021-12-13 19:06:13 -08:00
Joel Einbinder
c27491cd4d
feat(test-runner): shorten long output paths ( #10523 )
2021-12-13 10:56:03 -08:00
Joel Einbinder
7a02c52144
feat(test-runner): specific playwright types for expect ( #10670 )
2021-12-13 13:42:36 -05:00
Dmitry Gozman
308c7b4e32
docs: update test advanced guides ( #10861 )
...
Linking to the API reference, using better examples and newer docs.
2021-12-10 11:15:01 -08:00
Dmitry Gozman
897e41c6c1
docs: document TestOptions.video.size ( #10827 )
2021-12-09 07:38:58 -08:00
Pavel Feldman
b5933db279
feat(tsconfig): respect baseUrl and paths from tsconfig ( #10525 )
2021-12-08 22:43:00 -08:00
Pavel Feldman
4d683cef7f
fix(html): render text attachments as text ( #10778 )
2021-12-08 08:51:44 -08:00
Pavel Feldman
a08a41f6c9
chore: render annotations in html report ( #10774 )
2021-12-07 18:35:06 -08:00
Pavel Feldman
feb4c62da1
fix(html): html reporter fixes ( #10770 )
2021-12-07 16:47:47 -08:00
Pavel Feldman
7765131a14
feat(acceptDownload): revert acceptDownload ( #10709 )
2021-12-06 09:25:24 -08:00
Pavel Feldman
518d67add5
feat(test.info): expose information on the currently running test ( #10708 )
2021-12-06 09:25:11 -08:00
Dmitry Gozman
98e2f40bb0
docs: replace TestCase.suite with TestCase.parent ( #10687 )
...
It is there by mistake.
2021-12-02 14:24:43 -08:00
Pavel Feldman
31e0a63fcd
feat(toBeChecked): allow passing checked: false ( #10665 )
2021-12-02 10:31:26 -08:00
Pavel Feldman
b9731a904e
chore: add validations into check_deps ( #10661 )
2021-12-01 18:14:13 -08:00
Saransh Miglani
f05252874a
chore: minor code rearrangement ( #10650 )
2021-12-01 09:18:16 -08:00
Yury Semikhatsky
d66b7aab3b
feat(expext): toBeOK for APIResponse ( #10596 )
2021-11-30 18:12:19 -08:00
Dmitry Gozman
729da65eba
fix(test runner): allow multiple missing snapshots per test ( #10621 )
...
Instead of failing right away, continue test execution but mark
the test as failed.
2021-11-30 17:50:19 -08:00
Max Schmitt
3337920c76
docs: clarify webServer ipv4/ipv6 and baseURL protocol ( #10610 )
2021-11-30 20:01:28 +01:00
Max Schmitt
682d1f41c7
fix(test-runner): wrong test-runner options got determined ( #10607 )
...
Follow-up for #10507
2021-11-30 19:59:35 +01:00
Max Schmitt
9337aa3cdf
fix(test-runner): webServer port detection on Node.js 17 ( #10538 )
2021-11-29 19:36:35 +01:00
