Filip/playwright
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
7109 commits 34 branches 141 tags 223 MiB
Commit graph

349 commits

Author SHA1 Message Date
Dmitry Gozman 729da65eba
fix(test runner): allow multiple missing snapshots per test (#10621) 
Instead of failing right away, continue test execution but mark
the test as failed.
 2021-11-30 17:50:19 -08:00
Pavel Feldman 3997671ab7
fix(selectors): do not hide selector errors (#10595) 2021-11-29 17:13:24 -08:00
Joel Einbinder 15053d9b3b
fix(test-runner): apply default options to all browserTypes (#10507) 2021-11-29 12:21:15 -05:00
Max Schmitt 63abf0950b
test: using custom baseURL with webServer (#10539) 2021-11-29 17:41:26 +01:00
Joel Einbinder 20c0facfb9
fix(test runner): dont mangle test names with multiple dashes (#10447) 2021-11-24 19:36:38 -05:00
Pavel Feldman 5d19f16601
feat(esm): introduce experimental PW_EXPERIMENTAL_TS_ESM option (#10519) 2021-11-24 14:17:01 -08:00
Pavel Feldman 7eb3f76f49
feat(esm): allow running tests in type module projects (#10503) 2021-11-24 12:42:48 -08:00
Ross Wollman 854f321532
feat(api): add explicit async testInfo.attach (#10121) 
feat(api): add explicit async testInfo.attach

We add an explicit async API for attaching file paths (and Buffers) to
tests that can be awaited to help users ensure they are attaching files
that actually exist at both the time of the invocation and later when
reporters (like the HTML Reporter) run and package up test artifacts.

This is intended to help surface attachment issues as soon as possible
so you aren't silently left with a missing attachment
minutes/days/months later when you go to debug a suddenly breaking test
expecting an attachment to be there.

NB: The current implemntation incurs an extra file copy compared to
manipulating the raw attachments array. If users encounter performance
issues because of this, we can consider an option parameter that uses
rename under the hood instead of copy. However, that would need to be
used with care if the file were to be accessed later in the test.
 2021-11-23 09:30:53 -08:00
Dmitry Gozman 0302e759df
feat(test runner): allow top-level test.fixme similar to test.skip (#10250) 
```js
test.fixme('my test name', () => {});
```
 2021-11-19 11:40:40 -08:00
Pavel Feldman b302152789
chore(zones): prepare to remove wrapApiCall, introduce zones (#10427) 2021-11-18 22:30:09 -08:00
Dmitry Gozman d9f849fb14
feat(test runner): replace declare/define with "options" (#10293) 
1. Fixtures defined in test.extend() can now have `{ option: true }` configuration that makes them overridable in the config. Options support all other properties of fixtures - value/function, scope, auto.
```
const test = base.extend<MyOptions>({
  foo: ['default', { option: true }],
});
```

2. test.declare() and project.define are removed.

3. project.use applies overrides to default option values and nothing else. Any test.extend() and test.use() calls take priority over config options.

Required user changes: if someone used to define fixture options with test.extend(), overriding them in config will stop working. The solution is to add `{ option: true }`.

```
// Old code
export const test = base.extend<{ myOption: number, myFixture: number }>({
  myOption: 123,
  myFixture: ({ myOption }, use) => use(2 * myOption),
});

// New code
export const test = base.extend<{ myOption: number, myFixture: number }>({
  myOption: [123, { option: true }],
  myFixture: ({ myOption }, use) => use(2 * myOption),
});
```
 2021-11-18 15:45:52 -08:00
Dmitry Gozman 8f43f4c98f
feat(serial): better errors from beforeAll (#10419) 
When beforeAll hook times out or fails with an exception, we now
close the context and show a nice error.
 2021-11-18 14:36:55 -08:00
Dmitry Gozman bd93fc499f
fix(html reporter): show missing attachments as warnings (#10400) 2021-11-17 18:03:13 -08:00
Dmitry Gozman ce2c0c59a7
feat(expect): show expect timeout in the error message (#10388) 
Makes it easier to understand that expect does indeed have a separate timeout.

```
    Error: expect(received).toHaveCount(expected) // deep equality

    Expected: 0
    Received: 1

    Call log:
      - expect.toHaveCount with timeout 500ms
      - waiting for selector "span"
      -   selector resolved to 1 element
      -   unexpected value "1"
      -   selector resolved to 1 element
      -   unexpected value "1"
      -   selector resolved to 1 element
      -   unexpected value "1"
```
 2021-11-17 17:28:30 -08:00
Dmitry Gozman 638ebd6dd6
fix(test runner): do not validate fixtures in tests/hooks that are never run (#10328) 2021-11-15 13:17:26 -08:00
Dmitry Gozman 4bb123d4b7
fix(test runner): route more errors to reporter (#10263) 
For example, top-level errors in files or global setup issues.
 2021-11-12 06:47:41 -08:00
Dmitry Gozman 7eec66d0f9
chore(test runner): route runner errors through Reporter.onError (#10257) 2021-11-11 16:48:08 -08:00
Dmitry Gozman f38f611478
chore: add npm run lint-tests (#10252) 2021-11-11 13:27:50 -08:00
Dmitry Gozman 4c93417e8a
fix(github reporter): handle global onError (#10256) 2021-11-11 13:25:38 -08:00
Dmitry Gozman 9622704a8a
fix(test runner): update TestInfo.duration before running afterEach hooks (#10228) 2021-11-10 16:02:27 -08:00
Pavel Feldman 0d0f1690cd
fix(trace-viewer): do not rely upon request mode, it is inconsistent (#10198) 2021-11-09 15:12:37 -08:00
Pavel Feldman 7b64161a37
feat(test-runner): allow specifying fine-grained trace options (#10147) 2021-11-08 15:39:58 -08:00
Joel Einbinder 0cad0de3e3
fix(test runner): better error message when importing typescript from esmodule (#10061) 2021-11-08 16:25:40 -05:00
Dmitry Gozman 0a104bc500
docs: clarify that slow tests are about files (#10134) 2021-11-08 12:54:18 -08:00
Dmitry Gozman 3c1aaa5338
fix(trace): do not call tracing.stopChunk() twice (#10054) 2021-11-04 21:08:42 -07:00
Andrey Lushnikov 1a577ac344
chore: introduce docker integration dogfood (#9998) 
This patch:
- rolls stable-test-runner to Nov 2, 2021 tip-of-tree
- introduces a new npm script, `npm run vtest`, to run Visual Regression
  Tests for our HTML reporter
 2021-11-03 15:26:18 -07:00
Dmitry Gozman 3120f81629
fix(reporters): move Running X tests using Y workers to reporters (#10014) 2021-11-03 11:17:23 -07:00
Dmitry Gozman 9cebe60831
feat(reporters): augment non-stdio reporters with dot/line (#10003) 2021-11-03 08:25:16 -07:00
Nick Partridge a51ac39275
feat: add snapshotDir to set base snapshot directory (#9260) 2021-11-02 08:02:49 -07:00
Pavel Feldman 56ca3a18f5
feat(trace-viewer): show test name (#9957) 2021-11-01 21:23:35 -07:00
Pavel Feldman 3673776330
Revert "chore(test-runner): use test name as trace file name prefix (#9619)" (#9956) 
This reverts commit 3c420a7cf1.
 2021-11-01 20:27:41 -07:00
Pavel Feldman 9ac8829583
feat(html): bake report zip into the html report, allow opening from fs (#9939) 2021-11-01 16:14:52 -07:00
Dmitry Gozman 4f1027bdd0
feat(test runner): introduce TestInfo.parallelIndex (#9762) 
This is a worker number between `0` and `workers - 1` that
does not change after worker process restart.
 2021-11-01 10:37:34 -07:00
Max Schmitt 3c420a7cf1
chore(test-runner): use test name as trace file name prefix (#9619) 2021-11-01 18:15:19 +01:00
Dmitry Gozman 227e3da62f
fix(test runner): do not reuse worker that did not teardown scopes (#9872) 
Two bug fixes:
- Do not use the worker that is being shutdown for a new job.
- Report unhandled errors during "expected to fail" tests as
  fatal errors.
 2021-10-29 13:36:12 -07:00
Yury Semikhatsky 1886897e5c
fix(matchers): support builtin types (#9865) 2021-10-28 18:45:59 -07:00
Pavel Feldman 75ac579fac
fix(toHaveCount): another edge case fix (#9860) 2021-10-28 16:04:21 -07:00
Pavel Feldman 2e4722d460
test: migrate to upstream fixtures (#9835) 2021-10-28 08:31:30 -07:00
Andrey Lushnikov 4b22e1d54a
chore: bump stable test runner to 1.16.2 (#9836) 2021-10-28 01:11:26 -07:00
Pavel Feldman 5bf1c03b51
test: add several toBeEnabled methods (#9821) 2021-10-27 16:12:54 -07:00
Pavel Feldman 31faa7d9a7
chore: sort out base fixtures (#9809) 2021-10-27 08:28:53 -07:00
Dmitry Gozman 22e4a0d580
fix(snapshots): --update-snapshots + retries should work (#9799) 2021-10-26 13:50:16 -07:00
Dmitry Gozman 13ed1dee50
fix(test runner): do not mask uncaught error in beforeEach (#9764) 2021-10-25 14:17:27 -07:00
Pavel Feldman 2d4db7a6f0
fix(stack): hide test runner stack frames (#9735) 2021-10-25 11:49:59 -07:00
Pavel Feldman f08c22b467
fix(trace-viewer): show source files in local version (#9732) 2021-10-23 11:23:39 -07:00
Dmitry Gozman 23aa0be15b
docs: update afterEach docs with an example (#9727) 
Also add a test for TestInfo.status in afterEach.
 2021-10-22 16:32:22 -07:00
Joel Einbinder f2888395a6
fix: support even older node 12 (#9718) 2021-10-22 15:59:52 -04:00
Dmitry Gozman 193c79a685
feat(test runner): rewrite dispatcher to avoid unneded stalls (#9629) 
In some circumstances, dispatcher was waiting for all exisitng jobs
to finish before scheduling a new one. This leads to unneded stalls.
Instead, we can schedule jobs right away, if we have a worker
available.
 2021-10-22 11:10:37 -07:00
Pavel Feldman 6a3e08d1ac
fix(reporter): fix locator stacks to hide internal detail (#9693) 2021-10-21 16:10:47 -07:00
Pavel Feldman 299dffbdb3
fix(expect): fix toHaveCount(0) (#9690) 2021-10-21 14:46:54 -07:00
Max Schmitt 5426e24248
chore: roll stable-test-runner to ToT (#9655) 2021-10-20 10:35:37 -07:00
Joel Einbinder 53c866c0f2
fix(monorepo): explicitly list package.json exports instead of using glob patterns (#9584) 2021-10-19 12:28:02 -04:00
Dmitry Gozman fc6a93d24e
test: expect.not.toBeChecked when no elements match (#9607) 2021-10-19 09:13:16 -07:00
Dmitry Gozman 6d554a5e30
feat(reporter): expose more apis (#9603) 2021-10-19 08:38:04 -07:00
Pavel Feldman c06a6e1f63
fix(html): always highlight error code (#9604) 2021-10-18 21:37:19 -07:00
Pavel Feldman e37660b068
feat(api): expose step location (#9602) 2021-10-18 21:06:18 -07:00
Dmitry Gozman 4977edcaf3
fix(tracing): fix inconsistent calls of tracing.{start,stop} in fixtures (#9594) 
Drive-by: fix error logging.
 2021-10-18 21:05:59 -07:00
Pavel Feldman 432fb453e4
chore(html): only copy trace viewer for reports with traces (#9579) 2021-10-18 08:03:04 -07:00
Pavel Feldman 6afa85927e
chore(ui): redesign html report a bit (#9577) 2021-10-17 20:58:06 -07:00
Pavel Feldman 99f3389b3b
feat(html): automatically use console reporter with html (#9559) 2021-10-15 19:18:56 -07:00
Pavel Feldman 9135847950
feat(ui): merge stdio in html report (#9560) 2021-10-15 16:15:06 -07:00
Yury Semikhatsky 235cd10a43
fix(runner): clarify where test.use() can be called (#9486) 2021-10-15 12:03:26 -07:00
Pavel Feldman 75cfe5d1f5
chore: start adding html runner e2e tests (#9518) 2021-10-14 15:48:05 -07:00
Pavel Feldman 584014f6fa
feat(html): unhide html reporter (#9512) 2021-10-14 11:17:35 -07:00
Yury Semikhatsky e827bde1c2
fix(runner): skip global hooks in --list mode (#9480) 2021-10-13 14:18:15 -07:00
Dmitry Gozman 64a3099655
chore(expect): simplify expect implementation (#9459) 2021-10-13 08:56:57 -07:00
Pavel Feldman 876e08315b
fix(expect): continue polling in array-based matchers, fix edge cases (#9430) 2021-10-11 14:01:09 -07:00
Max Schmitt b1160ec239
chore(test-runner): make outputDir based on cwd instead of testDir (#9360) 2021-10-11 22:24:46 +02:00
Joel Einbinder d34b53a0d7
feat(test runner): allow comparing binary files to strings (#9315) 
Co-authored-by: Max Schmitt <max@schmitt.mx>
 2021-10-11 21:57:59 +02:00
Max Schmitt 76142a33ae
test: fix tests which include special signal handling (#9433) 2021-10-11 12:07:40 -07:00
Max Schmitt 9dd6bb8c6a Revert "feat(test-runner): allow parametrized tests that don't extend base (#9301)" 
This reverts commit f63af830de.
 2021-10-11 17:32:31 +02:00
Joel Einbinder c89d5a50dd
chore: migrate to monorepo (#9421) 2021-10-11 16:52:17 +02:00
Max Schmitt b69b8889fe test: fix broken relative outputDir test 
#9380
 2021-10-09 00:12:21 +02:00
Max Schmitt 804ee6e979
chore: roll stable-test-runner to ToT (#9397) 2021-10-08 18:37:06 +02:00
Max Schmitt b3a8ccd064
test(test-runner): add test for relative outputDir (#9381) 2021-10-08 09:20:41 +02:00
Pavel Feldman 083d882b2b
test: add missing request tests (#9343) 2021-10-06 10:15:13 -07:00
Pavel Feldman bc71d20d0f
feat(request): add global request fixture (#9332) 2021-10-06 10:09:27 -07:00
Pavel Feldman f63af830de
feat(test-runner): allow parametrized tests that don't extend base (#9301) 2021-10-04 17:16:33 -07:00
Sidharth Vinod f4cbd9d574
test: remove checking duration in github reporter to reduce flakiness 2021-10-04 15:53:26 +02:00
Sidharth Vinod be30f9f1c4
feat(test-runner): Add GitHub Actions reporter (#9191) 2021-10-04 10:32:56 +02:00
Pavel Feldman 913821f675
chore: roll Electron to 12.2.1 (#9271) 2021-10-01 19:40:47 -07:00
Nick Partridge b126a5685b
feat: add path option to toMatchSnapshot (#9156) 2021-10-01 09:15:44 -07:00
Max Schmitt e674d873a3
feat(test-runner): add reuse context mode to share a single context between tests (#9115) 2021-10-01 09:16:03 +02:00
Dmitry Gozman 5e3ad63b42
fix(test runner): do not write missing snapshot until the last retry (#9246) 
This prevents future retries from passing because of the actual
snapshot being written.

In theory, we can avoid running the retry since it should fail anyway.
However, this brings problems, for example in the `describe.serial` mode
where running a test also has some side effects and so it should not be
skipped. Since running a test without a snapshot is rare, it should be
fine to retry it.
 2021-09-30 16:44:52 -07:00
Sidharth Vinod fcb7d2b15a
feat(reporters): Add error position to JSON Report (#9151) 2021-09-30 14:18:36 -07:00
Pavel Feldman f78302e8dd
fix(toBeHidden): return true to missing elements (#9205) 2021-09-28 17:11:04 -07:00
Dmitry Gozman ed9b42a92d
feat(test runner): collect test error from worker teardown (#9190) 
When the test fails (usually with timeout), we wait until all hooks are run
and worker scope is teared down before reporting test end result.

This allows us to collect any error details populated by teardown
in addition to the "timed out" message.
 2021-09-28 10:56:50 -07:00
Dmitry Gozman fa536786f2
fix(test runner): proper serial mode with beforeAll/afterAll failures (#9183) 2021-09-27 15:58:26 -07:00
Dmitry Gozman 8dc8777ab4
feat(expect): toContainText(array) (#9160) 
This matches when each expected item from the array
is matched to one of the resolved elements, in order.
Note this performs both "sub-array" and "substring" matching.

Drive-by: documentation fixes.
Drive-by: added "selector resolved to 3 elements" log line
when expecting arrays.
 2021-09-27 11:14:35 -07:00
Max Schmitt cd22072685
chore: enable object-curly-spacing in ESLint (#9168) 2021-09-27 18:58:08 +02:00
Pavel Feldman 241411ad42
chore: render expect in trace viewer (#9141) 2021-09-27 09:19:59 -07:00
Max Schmitt 4573ce0cf8
chore(test-runner): launch -> webServer (#9167) 2021-09-27 11:32:57 +02:00
Dmitry Gozman d22dd4a4e7
feat(test runner): improve fixture typings for function fixtures (#9138) 
When fixture value `R` is a function, TypeScript sometimes confuses
function `R` and function `async ({}, use) => {}`. This leads to
`any` types in the latter because it could be either of the functions
as TS thinks.

The solution is to only accept the second syntax, assuming that noone
passes fixture value that is a function as is:

```js
// This will stop working.
test.extend<{ foo: (x: number) => number }>({
  foo: x => 2 * x,
});

// This will get inferred types and autocomplete.
test.extend<{ foo: (x: number) => number }>({
  foo: async ({}, use) => {
    await use(x => 2 * x);
  },
});
```
 2021-09-24 19:59:30 -07:00
Dmitry Gozman f4aaebfba0
fix(expect): produce "waiting for selector" log, corner cases (#9140) 2021-09-24 18:55:45 -07:00
Pavel Feldman db6c55af51
chore: add a test for the log scale polling (#9136) 2021-09-24 12:54:33 -07:00
Pavel Feldman 0908dc98c8
chore: migrate expect(locator).toMatchText to protocol (#9117) 2021-09-23 16:46:46 -07:00
Dmitry Gozman 568ec05a97
fix(junit): produce a single system-out entry (#9091) 
Instead of multiple `system-out` entries we produce a single one
with concatenated content. This is compatible with various junit xml
parsers in the wild.
 2021-09-22 15:28:57 -07:00
Dmitry Gozman 018467911b
test: introduce some common test fixtures (#9060) 2021-09-21 16:24:48 -07:00
Max Schmitt f9c5279c3a
fix(test-runner): toHaveURL respect baseURL (#9050) 2021-09-21 21:41:24 +02:00
Dmitry Gozman 7ec1035b98
test: improve child process utilities in tests (#9036) 2021-09-20 17:17:12 -07:00
Pavel Feldman 63ff405e6e
fix(inspector): stop on all snapshottable actions (#8990) 2021-09-17 15:24:15 -07:00
Dmitry Gozman 43213614a1
fix(test runner): after hooks step should not be nested (#8969) 2021-09-16 15:51:27 -07:00
Dmitry Gozman 5379b2dcba
fix(test runner): account for errors with inconsistent stack/message (#8950) 2021-09-15 21:28:36 -07:00
Darrell Breeden d82cb9a2ff
feat: addition of tags to JSON reporter output (#8920) 2021-09-15 12:30:22 -07:00
Pavel Feldman 14bc663742
chrome: don't nest pw:api steps (#8932) 2021-09-15 11:34:23 -07:00
Dmitry Gozman b8a46580dd
fix(expect): toHaveText, toContainText and toHaveTitle normalize whitespace (#8929) 2021-09-14 19:24:29 -07:00
Pavel Feldman 5253a7eb54
feat(html): restore trace, video, screenshot (#8925) 2021-09-14 16:26:31 -07:00
Pavel Feldman 16baaa317d
chore: remove stale html experiments (#8905) 2021-09-13 20:34:46 -07:00
Pavel Feldman 1925c85dfb
feat(report): render attachment as a part of failure (#8903) 2021-09-13 18:07:40 -07:00
Pavel Feldman b76e993951
chore: remove log from htmlreport (#8898) 2021-09-13 18:07:15 -07:00
Dmitry Gozman d9d2d809a2
fix(test): make use in config accept option values only (#8828) 
Also include default options in FullConfig/FullProject.
Also make examples compile and add a test.
 2021-09-13 17:50:31 -07:00
Dmitry Gozman ed34a67d4a
fix(junit reporter): put stdio under testcase (#8900) 
When output happened during test execution, it should be
under `<testcase><system-out>...</system-out></testcase>`.
 2021-09-13 17:50:08 -07:00
Dmitry Gozman 440651e05c
test(test runner): calling test.step() from fixture (#8899) 2021-09-13 17:49:58 -07:00
Dmitry Gozman 6c41ad3329
test: expect(locator).toHaveCount should wait (#8901) 2021-09-13 17:06:19 -07:00
Joel Einbinder bf35da3656
fix(test-runner): accept unix separators even on windows (#8881) 
.
 2021-09-13 12:09:38 -04:00
Dmitry Gozman bcb0c1745b
fix(waitForEventInfo): reply from the server side (#8825) 
Otherwise, client thinks that `waitForEventInfo` is a pending operation.
 2021-09-09 21:20:26 -07:00
Pavel Feldman 665143d629
chore(report): don't generate file per test (#8822) 2021-09-09 17:35:31 -07:00
Pavel Feldman 7bbb63d143
feat(test): start authoring raw reporter (#8790) 2021-09-09 14:17:18 -07:00
Yury Semikhatsky afe92a6fcf
fix(runner): avoid line wrapping in list reporter (#8705) 2021-09-07 17:42:17 -07:00
Daniel Dyssegaard Kallick b2c59a15dd
feat(test-runner): include test step in json report (#8476) 2021-09-07 13:35:30 -07:00
Dmitry Gozman 3739113e74
fix(test runner): allow dot-files and dot-directories with tests (#8751) 2021-09-07 10:32:47 -07:00
Max Schmitt e40b805782
feat(test-runner): support baseURL in toHaveURL (#8743) 2021-09-07 18:34:02 +02:00
Yury Semikhatsky e5e461c0de
fix: use logger from config if specified (#8697) 2021-09-03 14:21:25 -07:00
Dmitry Gozman e2b092c1a0
feat(html reporter): show log for pw:api and expect steps (#8692) 2021-09-03 13:08:17 -07:00
Dmitry Gozman 2b4a93972d
test: use ToT test runner for non-test-runner tests (#8671) 2021-09-03 11:22:25 -07:00
Dmitry Gozman 620712a5d9
feat(expect): support array of RegExp objects in toHaveText/toHaveClass (#8667) 2021-09-02 15:48:04 -07:00
Dmitry Gozman e691b649de
feat(test runner): describe.parallel (#8662) 2021-09-02 15:42:07 -07:00
Fokke Zandbergen 0ae38b5aec
fix(test-runner): WebServer: use socket for reuseExistingServer to detect servers that aren't using SO_REUSEADDR (#8537) 2021-09-02 18:39:41 +02:00
Yury Semikhatsky bafa426231
feat(runner): support multiple names in project filter (#8600) 2021-09-02 09:29:55 -07:00
Dmitry Gozman b1260602ac
docs: update global setup docs (#8637) 
Changing example to "authenticate once", and also updating the auth doc.
Adding a test with the same setup.
 2021-09-01 15:35:46 -07:00
Dmitry Gozman 4f4cf448c2
fix(test runner): generate unique outputDir for beforeAll/afterAll (#8633) 2021-09-01 13:41:35 -07:00
Dmitry Gozman 1a9215a6ec
fix(html reporter): do not throw when attachment is actually missing (#8630) 2021-09-01 12:20:28 -07:00
Dmitry Gozman b8f8ca7493
feat(tracing): tracing.{start,stop}Chunk instead of tracing._export (#8521) 2021-08-31 17:03:31 -07:00
Dmitry Gozman bee8ed117b
feat(html reporter): preview source code, steps and step errors (#8598) 2021-08-31 16:34:52 -07:00
Dmitry Gozman 900362ec0b
fix(test runner): report unhandled rejections during worker teardown (#8592) 2021-08-31 10:50:30 -07:00
Dmitry Gozman 37a897c9b4
fix(test runner): report beforeAll timeout instead of hanging (#8529) 
We used to not report fatal error and hang forever because worker
did not run any tests but also did not report any errors.

Also properly show stack-less errors.
 2021-08-28 07:19:45 -07:00
Michael Rienstra ed0c47a06f
fix(test-runner): minor error message typo (#8489) 2021-08-27 10:15:16 +02:00
Dmitry Gozman a479cb6f52
fix(test runner): add an overload for test.skip(title, fn) (#8454) 
We shipped this feature, but forgot to add the right overload to d.ts.
 2021-08-25 14:36:36 -07:00
Dmitry Gozman de85d8bb83
fix(test runner): do not special case test.fail (#8447) 
This makes `test.fail` tests considered as passing when they actually fail:
- Stop restarting the worker.
- Retry when it passes instead of a fail.
- Behaves similar to regular tests in a `describe.serial` suite.
 2021-08-25 12:19:50 -07:00
Dmitry Gozman 75fb77355a
test(test runner): check retries with beforeAll failure and multiple tests (#8413) 2021-08-24 12:22:16 -07:00
Joel Einbinder a8a3799e9d
fix(test-runner): use describe name in output dir (#8282) 2021-08-24 10:33:40 -04:00
Dmitry Gozman f9b87268a7
fix(test runner): afterAll error should not mask beforeAll error (#8358) 2021-08-23 09:21:40 -07:00
Pavel Feldman 0997c13151
fix(test-runner): do not attach non-existent diff (#8297) 2021-08-20 13:40:27 -07:00
Dmitry Gozman e5be2c9205
feat(test runner): show failure details for flaky test runs (#8332) 
Currently, we just say "foo.spec.ts > my test" is flaky, but do not
show how exactly the failed run went.
 2021-08-19 18:20:53 -07:00
Dmitry Gozman 70eca0401d
fix(test runner): avoid internal error for step end without begin (#8327) 
fix(test runner): avoid internal error for step end without begin

Consider the following scenario:
- Test finishes and starts tearing down fixtures.
- Fixture teardown starts a step S and then times out.
- We declare the test finished (with timeout).
- Dispatcher shuts down the worker and spins a new one for a retry.
  Additionally, it clears steps information for the test to be
  ready for the new retry. Step S information is lost.
- Meanwhile, during worker teardown, the step S does
  actually finish (usually with an error), and we send stepEnd for S.
- Dispatcher does not know what to do with step S end and
  prints an internal error.

The fix is to ignore certain messages from the shutting down worker that failed.
 2021-08-19 17:04:09 -07:00
Pavel Feldman 710cec80a0
feat(test-runner): render step titles (#8270) 2021-08-17 16:41:36 -07:00
Pavel Feldman 97ba4f22f3
feat(test-runner): report nested steps (#8266) 2021-08-17 13:57:26 -07:00
Dmitry Gozman 2aff06ec73
fix(test runner): make sure tracing is not running on non-retries (#8232) 
When sharing a context between tests and using `'on-first-retry'` we
could end up with tracing still running in non-retried tests. That's
extra overhead without a reason.
 2021-08-16 16:46:35 -07:00
Dmitry Gozman d32d50a906 fix(test runner): make sure we always teardown all fixtures (#8158) 
Even if one of the fixtures throws, we should teardown all of them
so that we can run afterAll hooks.
 2021-08-12 09:08:56 -07:00
Pavel Feldman d2d71c4cdb
fix(reporter): group fixture initialization under before hooks (#8072) 2021-08-12 07:58:00 -07:00