Filip/playwright
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
5769 commits 34 branches 141 tags 223 MiB
Commit graph

166 commits

Author SHA1 Message Date
Pavel Feldman 710cec80a0
feat(test-runner): render step titles (#8270) 2021-08-17 16:41:36 -07:00
Pavel Feldman 97ba4f22f3
feat(test-runner): report nested steps (#8266) 2021-08-17 13:57:26 -07:00
Joel Einbinder de4464cb9a
fix(expect): highlight diff in toHaveText (#8262) 2021-08-17 16:56:21 -04:00
Dmitry Gozman 2aff06ec73
fix(test runner): make sure tracing is not running on non-retries (#8232) 
When sharing a context between tests and using `'on-first-retry'` we
could end up with tracing still running in non-retried tests. That's
extra overhead without a reason.
 2021-08-16 16:46:35 -07:00
Dmitry Gozman d32d50a906 fix(test runner): make sure we always teardown all fixtures (#8158) 
Even if one of the fixtures throws, we should teardown all of them
so that we can run afterAll hooks.
 2021-08-12 09:08:56 -07:00
Pavel Feldman d2d71c4cdb
fix(reporter): group fixture initialization under before hooks (#8072) 2021-08-12 07:58:00 -07:00
Dmitry Gozman 44cdda43fe
feat(test runner): show stdio for failures in terminal reporters (#8150) 2021-08-11 16:44:19 -07:00
Dmitry Gozman a5e0965087
feat(test runner): implement test.describe.serial (#8132) 2021-08-10 21:26:45 -07:00
Joel Einbinder 64da74fba8
feat(test-runner): allow non-ascii characters in the output dir path (#8093) 2021-08-10 21:24:35 -07:00
Pavel Feldman b800c1d35c
feat(reporter): render attachments in html report (#8125) 2021-08-10 17:06:25 -07:00
Dmitry Gozman c9c1ea6546
fix(test runner): disallow use(workerFixture) in describes (#8119) 
Using a worker fixture forces a new worker. This might be unexpected
when part of the test file runs in one worker, and another runs
in another worker. Top-level use of worker fixtures is still fine.
 2021-08-10 16:32:32 -07:00
Dmitry Gozman 9d6c7cdf20
fix(test runner): ensure we run after hooks after failures (#8102) 2021-08-10 10:54:05 -07:00
Dmitry Gozman 2744cd6c9a
feat(test runner): remove createContext fixture (#8109) 
We can now use `browser.newContext()` or `browser.newPage()` instead.
 2021-08-10 09:26:36 -07:00
Dmitry Gozman 3bf3318350
fix(test runner): make sure options, trace and screenshot apply to all contexts (#8083) 
- Uses some auto fixtures to set default options and instrumentation on BrowserType.
- Moves screenshot, trace and video to worker-scoped fixtures.
- Throws in page/context when used from beforeAll/afterAll.
- Plumbs around BrowserType to be accessible from Browser and BrowserContext.
 2021-08-09 18:09:11 -07:00
Dmitry Gozman e638c4597f
fix(test runner): do not swallow afterAll failure (#8099) 2021-08-09 14:21:53 -07:00
Dmitry Gozman 87548f94c1
feat(test runner): support test fixtures in beforeAll/afterAll (#8082) 
Each hook gets its own test scope. This is not too useful for
object fixtures like `page` (although one can use a page in
`beforeAll` to save storage state), but much more useful for option
fixtures like `viewport`.
 2021-08-09 13:26:33 -07:00
Dmitry Gozman 41949e559e
Revert "feat(test runner): file scope fixtures (#7969)" (#8081) 
This reverts commit 1bbf86d060,
leaving small improvements around.
 2021-08-09 12:33:16 -07:00
Pavel Feldman 290f601dae
feat(expect): introduce explicit default async expect timeout (#8071) 2021-08-07 22:08:56 -07:00
Andrey Lushnikov 755cf60496
feat(test): write missing snapshots to test-results (#7879) 
By default, CI's are set up to upload `//test-results` folder to
artifacts storage. Storing missing snapshots in the `//test-results`
folder too will simplify collecting new baselines from the CI.
 2021-08-07 16:52:11 -07:00
Pavel Feldman 40fb9d85e0
feat(test-result): render image diff (#8061) 2021-08-07 15:47:03 -07:00
Pavel Feldman b1dc84c953
chore: add debug logging to localize steps error (#8062) 2021-08-06 20:37:57 -07:00
Pavel Feldman 3424f59e67
chore(expect): polish matcher names, remote arguable ones (#8060) 2021-08-06 16:58:42 -07:00
Dmitry Gozman 5f297b6894
feat(junit reporter): add attachments to stdout (#8059) 
`JUnitReporter` follows the common format for attachments in JUnit reports,
recognized by GitLab and Jenkins among others.
 2021-08-06 15:47:54 -07:00
Pavel Feldman 79e8592146
docs: refresh getting started docs (#8054) 2021-08-06 14:02:41 -07:00
Dmitry Gozman d846c05619
fix(test runner): make obtainWorker() resolve with null when stopping (#8018) 
This ensures that we properly exit from `Dispatcher.run()`, print
epilogue and set the right exit code.
 2021-08-05 15:00:00 -07:00
Pavel Feldman a8d404cd29
feat(test-runner): basic html reporter (#7994) 2021-08-05 13:36:47 -07:00
Pavel Feldman 0a97e87817
feat(expect): log-scale expect pollers (#8010) 2021-08-05 10:55:37 -07:00
Dmitry Gozman 1bbf86d060
feat(test runner): file scope fixtures (#7969) 
These are reset after running tests from a single file.
 2021-08-04 21:11:02 -07:00
Dmitry Gozman ea4f42b7ed
feat(report): add video to attachments (#7976) 2021-08-03 17:07:57 -07:00
Max Schmitt 385d489b35
feat(test-runner): re-enable web server (#7906) 
Co-authored-by: Joel Einbinder <joel.einbinder@gmail.com>
 2021-08-03 23:24:14 +02:00
Pavel Feldman 3b34e57ee4
feat(test-runner): introduce test.step (#7956) 2021-08-02 22:11:37 -07:00
Pavel Feldman 5803035c1b
feat(test-runner): introduce steps (#7952) 2021-08-02 17:17:20 -07:00
Pavel Feldman af30d267b6
feat(test-runner): experiemental expect plumbing (#7926) 2021-07-30 16:07:02 -07:00
Dmitry Gozman 081b8683a3
feat(test runner): expect(locator) matchers to show a nice error on timeout (#7935) 2021-07-30 13:12:49 -07:00
Dmitry Gozman 5a3ebfc9f9
fix(test runner): dot reporter incorrectly splits by 80 (#7925) 2021-07-30 01:34:28 -07:00
Dmitry Gozman 2e387b3a3a
chore(test runner): reuse TestGroup instead of DispatcherEntry (#7924) 2021-07-29 21:41:06 -07:00
Pavel Feldman 4163cec93b
feat(test-runner): introduce actionTimeout and navigationTimeout (#7919) 2021-07-29 21:03:50 -07:00
Dmitry Gozman 34c0c342fa
fix(test runner): make test order stable when fixtures are changing (#7923) 
We used to sort based on workerHash, and that changes depending on
the exact worker fixtures list. Now we replace workerHash with
an ordinal when constructing the TestGroup list to preserve the
natural order.
 2021-07-29 18:27:47 -07:00
Dmitry Gozman a18b4fb49a
fix(test runner): workaround npm bug that sends multiple sigints (#7916) 
When running through NPM, we can get multiple SIGINT signals
for a single Ctrl+C - the more npm layers you use, the more SIGINTs you get.

This is a known NPM bug present since at least NPM v6.
Workaround is to give NPM one second for the fake signals, and
only then remove out listener.
 2021-07-29 16:01:22 -07:00
Dmitry Gozman 40901e8b9a
feat(test runner): test.skip(title, testFunction) syntax (#7922) 2021-07-29 14:33:37 -07:00
Dmitry Gozman dd0b089d13
feat(test runner): createContext fixture for multi-context scenarios (#7779) 2021-07-29 14:03:58 -07:00
Pavel Feldman 1807142eb7
feat(expect): even more matchers (#7902) 2021-07-29 07:33:19 -07:00
Pavel Feldman 9e07029973
chore: relayout matchers (#7901) 2021-07-28 22:30:37 -07:00
Pavel Feldman 3187ffdebf
feat(expect): add more matchers (#7891) 2021-07-28 15:44:44 -07:00
Dmitry Gozman 3f0485486d
feat(test runner): show errors from interrupted tests when available (#7874) 
This shows the exact operation that is timing out (like click) when
user hits Ctrl+C.
 2021-07-28 15:43:37 -07:00
Pavel Feldman 49e9f8c15e
feat(expect): add text and true matchers (#7873) 2021-07-28 12:07:11 -07:00
Pavel Feldman b8dc0b9156
feat(expect): implement toMatchText (#7871) 2021-07-27 20:26:12 -07:00
Dmitry Gozman da9b488d0d
fix(test-runner): sharding tests does not show a lot of skips (#7708) 2021-07-27 11:04:38 -07:00
Dmitry Gozman cf886b3829
fix(test runner): align shard info to be one-based everywhere (#7859) 
We used to treat shard from cli and shard from config differently.
 2021-07-27 09:13:04 -07:00
Dmitry Gozman bf6482a58d
fix(test runner): do not override browserName when using without --browser (#7806) 2021-07-23 09:04:20 -07:00
Dmitry Gozman 03ebe21323
faet(test runner): help when describe() is misused (#7753) 2021-07-22 12:34:37 -07:00
Max Schmitt 9e3ac1f3a2
fix(test-runner): warn users who invoke the test-runner with Jest (#7743) 2021-07-22 18:27:01 +02:00
Joel Einbinder 483b7d6a0d
feat(test-runner): mark launch as experimental (#7757) 2021-07-21 09:51:27 -05:00
Joel Einbinder 20ba7c3a96
feat(test-runner): use require.resolve for globalSetup and globalTeardown (#7752) 2021-07-20 15:13:40 -05:00
Joel Einbinder 051dc332a6
feat(test-runner): use require to resolve reporters (#7749) 2021-07-20 15:03:01 -05:00
Joel Einbinder bdbf9c9dda
perf(test-runner): lazy require babel (#7750) 2021-07-20 13:24:29 -05:00
Max Schmitt c84c5c8c9b
fix(test-runner): do not list tests to stdout when JSON reporter is used (#7730) 2021-07-20 01:10:43 +02:00
Dmitry Gozman 2ac0c59156
feat(test-runner): rename Test to TestCase (#7725) 2021-07-19 14:54:18 -07:00
Joel Einbinder d72efbe181
feat(test-runner): smaller error message paths (#7384) 2021-07-19 12:20:24 -05:00
Joel Einbinder 0cf9cf0829
fix(test-runner): set expand: false for expect. (#7722) 2021-07-19 11:59:53 -05:00
Dmitry Gozman 66ea613c4d
feat(test-runner): small changes to Reporter api (#7709) 
- `TestResult.startTime`
- `Suite.location` is optional now
- `Test.status()` renamed to `Test.outcome()` to differentiate against a
  `Test.expectedStatus` and `TestResult.status` of the different type.
 2021-07-18 17:40:59 -07:00
Dmitry Gozman 602d815981
fix(test-runner): list mode should print tests (#7665) 
It was not doing anything before.
 2021-07-16 22:34:55 -07:00
Dmitry Gozman 167db03f05
feat(test-runner): export reporter api as @playwright/test/reporter (#7692) 2021-07-16 21:15:03 -07:00
Dmitry Gozman 18be5f5319
feat(test-runner): suite per project (#7688) 
This makes our suite structure the following:
```
Root(title='') > Project(title=projectName) > File(title=relativeFilePath) > ...suites > test
```

Removed `fullTitle()` because it is not used directly by anyone.
Default reporters now report each test as
```
[project-name] › relative/file/path.spec.ts:42:42 › suite subsuite test title
```
 2021-07-16 15:23:50 -07:00
Pavel Feldman bde764085c
feat(test-runner): introduce attachments (#7685) 2021-07-16 13:48:37 -07:00
Dmitry Gozman 31572fc372
chore(test-runner): misc changes to reporter api (#7664) 
- `Location` with `file`, `line` and `column`.
- `fullTitle` does not include project name.
- `titlePath` method.
- All methods of `Reporter` are optional.
- Removed `Test.skipped` property that is superseeded by `Test.status()`.
- Replaced `Suite.findTest()` with `Suite.allTests()`.
- Removed `Test.suite` property.
 2021-07-16 12:40:33 -07:00
Dmitry Gozman 8b2dd2e3d1
chore(test-runner): remove the notion of Spec (#7661) 
We now have Suites and Tests. When running multiple projects the whole
suite is cloned for each project. Same happens for repeatEach.

This simplifies the reporters API, but there is still room for improvement.

JSON reporter continues to produce old json output.
 2021-07-15 22:02:10 -07:00
Dmitry Gozman bb34d7a953
fix(test-runner): sanitize snapshot name before constructing a path (#7620) 
This avoids problems with `toMatchSnapshot('../../dir/file.png')`
where we append this path to `snapshotDir` and end up in some random
place.

Also added a note to documentation.
 2021-07-14 16:31:19 -07:00
Max Schmitt 6cc2fe178e
feat(test-runner): migrate to launch config/server (#7603) 2021-07-15 01:19:45 +02:00
Max Schmitt a26fe65db3
fix(test-runner): start webServer before globalSetup/teardown (#7604) 2021-07-14 19:01:46 +02:00
Alex Plischke 27d89732ce
Use correct skip check when generating junit reports (#7571) 2021-07-13 05:05:34 -07:00
Joel Einbinder eb31b9e4a9
feat(test-runner): support esm modules in more places (#7542) 2021-07-12 11:59:58 -05:00
Dmitry Gozman 77deca1d6b
feat(test runner): export testInfo.data (#7525) 
This is a key-value storage for any information that goes into the report.
Also export JSONReport types.
 2021-07-08 17:16:36 -07:00
Max Schmitt e604f185ca
fix(test-runner): with automatic fixtures workerInfo was undefined after conditional skip (#7521) 2021-07-08 21:55:43 +02:00
Yury Semikhatsky b2742976a8
fix(runner): do not restart worker after skipping tests (#7511) 2021-07-08 09:36:28 -07:00
Max Schmitt ae489b1c43
fix(test-runner): do not override error with unhandled error (#7507) 2021-07-08 18:24:07 +02:00
Dmitry Zakharov 5464ad849e
fix(test-runner): handle negated toMatchSnapshot result (#7345) 2021-07-07 18:51:38 -07:00
Max Schmitt 08da9d207e
chore: move processLauncher into src/utils (#7504) 2021-07-07 21:14:16 +02:00
Dmitry Gozman 2073193c36
fix(test runner): keep track of remaining tests on the runner side (#7486) 
This fixes two issues:
- Sudden worker process exit is properly accounted for.
- We can stop() workers willy-nilly, e.g. after reaching maxFailures.

Details:
- DonePayload does not send `reamining` anymore, and worker does not track it.
- Instead, `Dispatcher._runJob` track remaining tests and acts accordingly.
- Upon worker exit, we emulate a fatal error for all remaining tests.

Drive-by:
- Do not report onTestBegin after reaching maxFailures to avoid confusion.
  Before, we did report onTestBegin, but not onTestEnd.
- List reporter aligned between "running" and "finished" state - it was
  one character misplaced.
- Added a SIGINT test.
 2021-07-07 12:04:43 -07:00
Max Schmitt 98bcf26656
feat(test-runner): add webServer (#7368) 2021-07-07 20:19:42 +02:00
Dmitry Gozman 444d1eb51a
feat(test runner): run modifier functions once if they do not depend on test fixtures (#7436) 2021-07-02 15:49:05 -07:00
Joel Einbinder a1f9fc3f32
fix(test runner): properly match test suffix (#7392) 2021-06-29 17:49:56 -07:00
Joel Einbinder 7629587914
fix(test-runner): work with .mjs files (#7373) 2021-06-29 15:28:41 -07:00
Dmitry Gozman 6b3614fd4c
feat(test runner): support test.setTimeout for the whole block (#7387) 
This enables `test.setTimeout()` outside of the test body, that
affects all tests in the block (either file or describe).
 2021-06-29 13:33:13 -07:00
Joel Einbinder 368880962f
fix(test-runner): better test match glob (#7382) 2021-06-29 11:49:50 -07:00
Dmitry Gozman 6aefa02e91
feat(test runner): improve reporters api (#7370) 
- onEnd may return a Promise
- onEnd now takes a result for the full run
- onTimeout is replaced with onEnd(result)
 2021-06-29 10:55:46 -07:00
Max Schmitt 0776cf76a2
feat(test-runner): do only allow unique spec titles per suite (#7300) 2021-06-28 22:13:35 +02:00
Andrey Lushnikov 1157f622f3
fix(test-runner): default @playwright/test timeout to 0 (#7312) 
All actions for the `@playwright/test` actions inside test should
have default 0 timeout. This way they will never clash with the
test timeout.
 2021-06-24 23:28:01 -07:00
Max Schmitt 908ad6921f
fix(test-runner): make it compatible when running as a sub-process (#7298) 
Works around the following bug: https://github.com/nodejs/node/issues/12921
 2021-06-25 00:54:06 +02:00
Max Schmitt bd86e70465
feat(test-runner): allow to focus a test in a location (#7208) 2021-06-24 10:02:34 +02:00
Dmitry Gozman 4c6fa42810
fix(test runner): expose real stack traces and speed up locations (#7265) 
Stop wrapping/prepending error messages so that we do not loose the stack trace. For this, update a few manually thrown errors with better messages (usually including a file path).

Speed up locations by doing manual `sourceMapSupport.wrapCallSite()` for a single call site. Performance gain in the runner process with 100 files x 100 tests each:
- 25% on the fresh run without babel cache;
- 80% on the cached run where babel is almost instant.

Also some obvious cleanups around stack traces (removing unused code).
 2021-06-23 10:30:54 -07:00
Max Schmitt 103f8ddd06
Revert "chore: update eslint config (#6840)" partly (#7268) 
This reverts commit ec7d37d92f.
 2021-06-23 11:08:35 +02:00
Max Schmitt 99bbc51760
fix(test-runner): support ANSII terminals with list reporter (#7258) 2021-06-22 19:04:24 +02:00
Dmitry Gozman 73a43fce60
chore(test runner): self profile under PWTEST_PROFILE_DIR (#7252) 
`PWTEST_CLI_ALLOW_TEST_COMMAND=1 PWTEST_PROFILE_DIR=./profile node lib/cli/cli.js test -c <dir/config>`
 2021-06-21 14:49:43 -07:00
Dmitry Gozman 6118d16edd
fix(test runner): properly keep track of requireFile to support helpers/wrappers (#7243) 
This fixes an issue where we incorrectly labeled and assigned ids for tests
that declared tests in require'd files or used test wrappers.
See new tests for examples.
 2021-06-21 11:25:15 -07:00
Pavel Feldman ea4eebeb2d
feat(test-runner): document tagging, implement grep-invert (#7227) 2021-06-18 17:56:59 -07:00
Dmitry Gozman 388c6f5196
feature(test runner): capture error and pending calls when timing out (#7193) 2021-06-17 15:09:38 -07:00
Dmitry Gozman 5e471a3ece
feat(test runner): add type to TestInfo.error (#7207) 
It has a very special `TestError` type, but we say it's `any` for some reason.
 2021-06-17 12:14:28 -07:00
Pavel Feldman 82a50b0e1d
fix(test-runner): property handle artifacts in context of preserveOutput (#7181) 2021-06-16 16:05:30 -07:00
Pavel Feldman 184f2c2e93
feat(test-runner): allow specifying video size (#7158) 2021-06-16 07:51:54 -07:00