Commit graph

166 commits

Author SHA1 Message Date
Pavel Feldman 63ff405e6e
fix(inspector): stop on all snapshottable actions (#8990) 2021-09-17 15:24:15 -07:00
Dmitry Gozman 43213614a1
fix(test runner): after hooks step should not be nested (#8969) 2021-09-16 15:51:27 -07:00
Andrey Lushnikov 4f762ba90a
feat: introduce experimental general-purpose grid (#8941)
This patch adds a general-purpose grid framework to parallelize
Playwright across multiple agents.

This patch adds two CLI commands to manage grid:

- `npx playwright experimental-grid-server` - to launch grid
- `npx playwrigth experimental-grid-agent` - to launch agent in a host
  environment.

Grid server accepts an `--agent-factory` argument. A simple
`factory.js` might look like this:

```js
const child_process = require('child_process');

module.exports = {
  name: 'My Simple Factory',
  capacity: Infinity, // How many workers launch per agent
  timeout: 10_000, // 10 seconds timeout to create agent
  launch: ({agentId, gridURL, playwrightVersion}) => child_process.spawn(`npx`, [
    'playwright'
    'experimental-grid-agent',
    '--grid-url', gridURL,
    '--agent-id', agentId,
  ], {
    cwd: __dirname,
    shell: true,
    stdio: 'inherit',
  }),
};
```

With this `factory.js`, grid server could be launched like this:

```bash
npx playwright experimental-grid-server --factory=./factory.js
```

Once launched, it could be used with Playwright Test using env variable:

```bash
PW_GRID=http://localhost:3000 npx playwright test
```
2021-09-16 01:20:36 -07:00
Dmitry Gozman 5379b2dcba
fix(test runner): account for errors with inconsistent stack/message (#8950) 2021-09-15 21:28:36 -07:00
Dmitry Gozman 9b08871b4d
fix(test runner): print flaky tests summary in yellow, not in red (#8951) 2021-09-15 16:28:57 -07:00
Darrell Breeden d82cb9a2ff
feat: addition of tags to JSON reporter output (#8920) 2021-09-15 12:30:22 -07:00
Max Schmitt 131239569d
feat(test-runner): add --debug CLI flag (#8938) 2021-09-15 21:19:31 +02:00
Pavel Feldman 14bc663742
chrome: don't nest pw:api steps (#8932) 2021-09-15 11:34:23 -07:00
Dmitry Gozman b8a46580dd
fix(expect): toHaveText, toContainText and toHaveTitle normalize whitespace (#8929) 2021-09-14 19:24:29 -07:00
Pavel Feldman 5253a7eb54
feat(html): restore trace, video, screenshot (#8925) 2021-09-14 16:26:31 -07:00
Pavel Feldman e91243ac90
feat(html): auto-open report (#8908) 2021-09-14 13:55:31 -07:00
Joel Einbinder 351c20be48
chore(expect): remove some dependencies on jest internals (#8782) 2021-09-14 11:47:06 -04:00
Pavel Feldman 16baaa317d
chore: remove stale html experiments (#8905) 2021-09-13 20:34:46 -07:00
Pavel Feldman 1925c85dfb
feat(report): render attachment as a part of failure (#8903) 2021-09-13 18:07:40 -07:00
Pavel Feldman b76e993951
chore: remove log from htmlreport (#8898) 2021-09-13 18:07:15 -07:00
Dmitry Gozman d9d2d809a2
fix(test): make use in config accept option values only (#8828)
Also include default options in FullConfig/FullProject.
Also make examples compile and add a test.
2021-09-13 17:50:31 -07:00
Dmitry Gozman ed34a67d4a
fix(junit reporter): put stdio under testcase (#8900)
When output happened during test execution, it should be
under `<testcase><system-out>...</system-out></testcase>`.
2021-09-13 17:50:08 -07:00
Pavel Feldman bb33b8923e
feat(report): add show-report option (#8895) 2021-09-13 15:19:40 -07:00
Joel Einbinder bf35da3656
fix(test-runner): accept unix separators even on windows (#8881)
.
2021-09-13 12:09:38 -04:00
Pavel Feldman ccff6e3036
chore(html): render steps and errors (#8826) 2021-09-10 07:52:29 -07:00
Pavel Feldman 665143d629
chore(report): don't generate file per test (#8822) 2021-09-09 17:35:31 -07:00
Pavel Feldman 7bbb63d143
feat(test): start authoring raw reporter (#8790) 2021-09-09 14:17:18 -07:00
Yury Semikhatsky afe92a6fcf
fix(runner): avoid line wrapping in list reporter (#8705) 2021-09-07 17:42:17 -07:00
Daniel Dyssegaard Kallick b2c59a15dd
feat(test-runner): include test step in json report (#8476) 2021-09-07 13:35:30 -07:00
Dmitry Gozman 3739113e74
fix(test runner): allow dot-files and dot-directories with tests (#8751) 2021-09-07 10:32:47 -07:00
Max Schmitt e40b805782
feat(test-runner): support baseURL in toHaveURL (#8743) 2021-09-07 18:34:02 +02:00
Dmitry Gozman e2b092c1a0
feat(html reporter): show log for pw:api and expect steps (#8692) 2021-09-03 13:08:17 -07:00
Dmitry Gozman 620712a5d9
feat(expect): support array of RegExp objects in toHaveText/toHaveClass (#8667) 2021-09-02 15:48:04 -07:00
Dmitry Gozman e691b649de
feat(test runner): describe.parallel (#8662) 2021-09-02 15:42:07 -07:00
Fokke Zandbergen 0ae38b5aec
fix(test-runner): WebServer: use socket for reuseExistingServer to detect servers that aren't using SO_REUSEADDR (#8537) 2021-09-02 18:39:41 +02:00
Yury Semikhatsky bafa426231
feat(runner): support multiple names in project filter (#8600) 2021-09-02 09:29:55 -07:00
Dmitry Gozman 798f894f47
feat(html reporter): show test source (#8643) 2021-09-01 21:15:11 -07:00
Dmitry Gozman 4f4cf448c2
fix(test runner): generate unique outputDir for beforeAll/afterAll (#8633) 2021-09-01 13:41:35 -07:00
Dmitry Gozman 1a9215a6ec
fix(html reporter): do not throw when attachment is actually missing (#8630) 2021-09-01 12:20:28 -07:00
Dmitry Gozman b8f8ca7493
feat(tracing): tracing.{start,stop}Chunk instead of tracing._export (#8521) 2021-08-31 17:03:31 -07:00
Dmitry Gozman bee8ed117b
feat(html reporter): preview source code, steps and step errors (#8598) 2021-08-31 16:34:52 -07:00
Pavel Feldman 26e7c2825b
chore: audit code for new Promise(async) (#8601) 2021-08-31 15:55:32 -07:00
Pavel Feldman 1c169289b2
chore: move async utils into a separate file (#8595) 2021-08-31 14:44:08 -07:00
Dmitry Gozman 900362ec0b
fix(test runner): report unhandled rejections during worker teardown (#8592) 2021-08-31 10:50:30 -07:00
Pavel Feldman dd31f3bd43
chore: introduce manual promise helper (#8533) 2021-08-29 11:21:06 -07:00
Dmitry Gozman 37a897c9b4
fix(test runner): report beforeAll timeout instead of hanging (#8529)
We used to not report fatal error and hang forever because worker
did not run any tests but also did not report any errors.

Also properly show stack-less errors.
2021-08-28 07:19:45 -07:00
Michael Rienstra ed0c47a06f
fix(test-runner): minor error message typo (#8489) 2021-08-27 10:15:16 +02:00
Dmitry Gozman de85d8bb83
fix(test runner): do not special case test.fail (#8447)
This makes `test.fail` tests considered as passing when they actually fail:
- Stop restarting the worker.
- Retry when it passes instead of a fail.
- Behaves similar to regular tests in a `describe.serial` suite.
2021-08-25 12:19:50 -07:00
Joel Einbinder fc991fe897
chore(typescript): turn on noImplicitOverride (#7810) 2021-08-25 10:11:18 -04:00
Max Schmitt 712a36649b
fix(test-runner): do not start web-server when tests are listed (#8414) 2021-08-24 23:39:24 +02:00
Joel Einbinder a8a3799e9d
fix(test-runner): use describe name in output dir (#8282) 2021-08-24 10:33:40 -04:00
Dmitry Gozman f9b87268a7
fix(test runner): afterAll error should not mask beforeAll error (#8358) 2021-08-23 09:21:40 -07:00
Pavel Feldman 0997c13151
fix(test-runner): do not attach non-existent diff (#8297) 2021-08-20 13:40:27 -07:00
Dmitry Gozman e5be2c9205
feat(test runner): show failure details for flaky test runs (#8332)
Currently, we just say "foo.spec.ts > my test" is flaky, but do not
show how exactly the failed run went.
2021-08-19 18:20:53 -07:00
Dmitry Gozman 70eca0401d
fix(test runner): avoid internal error for step end without begin (#8327)
fix(test runner): avoid internal error for step end without begin

Consider the following scenario:
- Test finishes and starts tearing down fixtures.
- Fixture teardown starts a step S and then times out.
- We declare the test finished (with timeout).
- Dispatcher shuts down the worker and spins a new one for a retry.
  Additionally, it clears steps information for the test to be
  ready for the new retry. Step S information is lost.
- Meanwhile, during worker teardown, the step S does
  actually finish (usually with an error), and we send stepEnd for S.
- Dispatcher does not know what to do with step S end and
  prints an internal error.

The fix is to ignore certain messages from the shutting down worker that failed.
2021-08-19 17:04:09 -07:00