Commit graph

52 commits

Author SHA1 Message Date
Andrey Lushnikov 05b019f1ba
reland: testrunner: make environment a simple class (#2812)
This re-lands PR https://github.com/microsoft/playwright/pull/2769

It was reverted before in https://github.com/microsoft/playwright/pull/2790
because it was breaking the new CHANNEL bot.
2020-07-02 11:05:38 -07:00
Pavel Feldman e467ea57e7
revert: testrunner: make environment a simple class (#2769) (#2790) 2020-07-01 10:56:19 -07:00
Andrey Lushnikov 1605cb453c
testrunner: make environment a simple class (#2769)
This patch:
- makes environment a simple class with optional methods `beforeEach`, `afterEach`, `beforeAll`, `afterAll`, `globalSetup` and `globalTeardown`
- removes capability to have multiple hooks of the same name inside suite
- removes default environment for test. (`dit` now adds a `TraceTestEnvironment` to the test)
- extracts all environments that we use in our tests in `//test/environments.js`

Downsides:
- we no longer know hook locations for the environments. This, however, should not be a big deal since stack traces (if any) will still point into it.
- this also regresses hook locations for suites for simplicity. We can get them back, but it shouldn't be pressing since we now have only one hook of each kind in every suite.
2020-06-30 16:51:03 -07:00
Andrey Lushnikov 1865c5685a
testrunner: support globalSetup and globalTeardown hooks. (#2686) 2020-06-24 22:08:46 -07:00
Andrey Lushnikov 07dbd0baae
testrunner: teach runHook to accept hook arguments (#2682) 2020-06-23 18:17:14 -07:00
Andrey Lushnikov c61e2d6cc9
testrunner: drop nested test environments (#2681) 2020-06-23 16:45:00 -07:00
Dmitry Gozman cf82e2c945
fix(testrunner): await terminations before reporting test results (#1855)
This way we ensure that all errors are picked up.
2020-04-17 18:42:12 -07:00
Dmitry Gozman a3571c24c4
fix(testrunner): respect timeout=0 in hooks (#1764) 2020-04-13 14:30:51 -07:00
Dmitry Gozman 2d57fff0be
fix(tests): fix multiple browsers tests (#1718)
- Use a server only once, not once per browser.
- Avoid require cache for test files.
- Improve testrunner hooks debugging.
2020-04-08 15:54:17 -07:00
Dmitry Gozman 20ff327827
feat(testrunner): catch delegate errors (#1704)
This ensures we get a proper error when something goes wrong. Should
also help with producing the right error code in the case of internal error.

Drive-by: fix location issue which manifests on the bots.
Drive-by: remove the use of Array.prototype.flat to make it work on bots.
2020-04-07 22:56:21 -07:00
Dmitry Gozman 39e06f0286
feat(testrunner): improve reporting of unhandled errors/rejections (#1697) 2020-04-07 16:02:33 -07:00
Dmitry Gozman f2b13c0e93
chore(testrunner): split TestRunner into parts (#1679) 2020-04-06 17:21:42 -07:00
Dmitry Gozman 823f961d8d
feat(testrunner): migrate from events to a delegate (#1647)
This allows an async handler for each event that can be awaited.
Drive-by: merge TestPass into TestRunner.
2020-04-03 15:47:25 -07:00
Dmitry Gozman 1f2803bbc6
feat(testrunner): removeEnvironment (#1650) 2020-04-03 09:48:01 -07:00
Dmitry Gozman d38baae01c
feat(testrunner): nested environments (#1635) 2020-04-02 13:12:22 -07:00
Dmitry Gozman f87e64544c
feat(testrunner): introduce environments (#1593) 2020-04-01 10:49:47 -07:00
Dmitry Gozman b6166c9669
chore(testrunner): introduce Location class (#1585)
Drive-by: fix an edge when testing continued after termination.
2020-03-29 21:38:30 -07:00
Dmitry Gozman c49b856d6d
chore(testrunner): remove setup() helper (#1584) 2020-03-29 11:37:45 -07:00
Dmitry Gozman a41836b1f1
chore(testrunner): introduce TestRun (#1582) 2020-03-28 14:25:57 -07:00
Dmitry Gozman b85ab891a1
chore(testrunner): make most modifiers external (#1581) 2020-03-28 08:49:00 -07:00
Dmitry Gozman 4bd46bafa9
feat(testurnner): allow multiple hooks isntances and per-test hooks (#1571) 2020-03-27 22:57:22 -07:00
Dmitry Gozman a042466125
chore(testrunner): remove effectiveMode and effectiveExpectation (#1569) 2020-03-27 15:35:14 -07:00
Dmitry Gozman af7a16c360
chore(testrunner): merge test spec with test, suite spec with suite (#1566) 2020-03-26 22:47:13 -07:00
Dmitry Gozman aad82e00bd
chore(testrunner): decouple UserCallback from location and timeout (#1557)
This will make it easier to change lifetimes of Test and Suite.
2020-03-26 14:43:28 -07:00
Dmitry Gozman 81bd8de00b
feat(testrunner): composable and bindable attributes and modifiers (#1547)
This allows to make things like `fit`, `skip()` and soon even `dit` be implemented externally.
2020-03-25 22:42:09 -07:00
Dmitry Gozman b61198458c
feat(testrunner): modifiers and attributes (#1538)
This generalizes test modifiers to support custom ones.
2020-03-25 14:40:57 -07:00
Dmitry Gozman a2ee7a1f2a
fix(testrunner): do not spam output after termination (#1511) 2020-03-24 14:40:59 -07:00
Dmitry Gozman 63906454a7
fix(testrunner): attribute unhandle promise reject to a single worker (#1491)
When there is a single worker, we are almost sure the error originated in that worker. Attributing it helps with context by showing last run tests.
2020-03-23 14:45:21 -07:00
Andrey Lushnikov a9ab9b0dc0
fix(testrunner): sourcemapify stack traces for test errors (#1409)
Fixes #1403
2020-03-17 09:04:44 -07:00
Dmitry Gozman 3960b179a1
fix(testrunner): fit.fail should run the test (#1407) 2020-03-16 19:12:52 -07:00
Dmitry Gozman e7eeefe4c7
chore(testrunner): separate expectations from run mode (#1395)
Run/Focus/Skip is orthogonal to expect to Pass/Fail/Flake.
This change separates the two, in a preparation to run Fail/Flaky tests.
2020-03-15 23:10:49 -07:00
Dmitry Gozman 0d7cb29329
test: continue running tests after crash, report crashes separately (#1362) 2020-03-12 17:32:53 -07:00
Andrey Lushnikov 1cd00bd068
feat(testrunner): allow filtering by name and show all focused tests (#1354) 2020-03-11 18:30:43 -07:00
Dmitry Gozman 0ce8efab7b
test: rework testrunner workers (#1296)
This change introduces a TestWorker that can be in a certain state,
meaning it has run some beforeAll hooks of a certain test suite stack.

TestWorker can be created at any time, which allows for a number of features:
- don't run hooks for suites with no runnable tests;
- smarter test distribution (and possibility for variuos strategies);
- recovering from hook failures and test failure by creating a new worker;
- possible isolation between workers by running them in separate environments.
2020-03-10 11:30:02 -07:00
Andrey Lushnikov d542ef849f
fix(testrunner): handle uncaught errors (#1317) 2020-03-10 11:16:54 -07:00
Andrey Lushnikov a0e12e0a14
feat(testrunner): support .slow() for slow tests (#1274)
Slow tests have 3x timeout.
2020-03-06 15:52:24 -08:00
Pavel ba06fb2f0d test: mark some tests as skipped 2020-03-02 14:57:54 -08:00
Pavel Feldman 5bd6e4970b
test: it.skip skips and it.fail expects to fail now (#1178) 2020-03-02 13:47:08 -08:00
Pavel Feldman cbf65a9c56
test: chain test modifiers (#1175) 2020-03-02 11:18:42 -08:00
Dmitry Gozman ac2f04f10f
api(selectors): pass selector name when registering, allow file path (#1162) 2020-02-28 15:34:07 -08:00
Andrey Lushnikov 0ded511d0b
feat(testrunner): better matchers (#1077)
This patch re-implements matching and reporting for test runner.
Among other improvements:
- test failures now show a short snippet from test
- test failures now explicitly say what received and what was expected
- `expect.toBe()` now does text diff when gets strings as input
- `expect.toEqual` now does object diff
2020-02-20 22:55:39 -08:00
Andrey Lushnikov 8487ef2821
feat(testrunner): add DEBUG to testrunner (#1014)
The plan is to collect logs for the whole test run and upload it later on
using https://github.com/actions/upload-artifact

Produced log size:
- 163MB (8.5MB zipped) for Chromium: `DEBUG=* npm run unit 2>log`
- 135MB (4.8MB zipped) for WebKit: `DEBUG=*,-pw:wrapped* npm run wunit 2>log`
- 29MB (4.0MB zipped) for Firefox: `DEBUG=* npm run funit 2>log`
2020-02-14 15:21:08 -08:00
Andrey Lushnikov 7128628d70 feat(testrunner): ability to repeat test suites (#681)
Now you can `fdescribe.repeat(10)` to repeat test suites.
2020-01-27 12:48:11 -08:00
Andrey Lushnikov 710554bff5
feat(testrunner): add it.repeat to repeat test multiple times (#568) 2020-01-22 15:31:52 -08:00
Andrey Lushnikov 98bf9ac1d5
fix(testrunner): return correct exit code (#474) 2020-01-13 15:30:16 -08:00
Andrey Lushnikov 02c504ca59
feat(testrunner): "sourcemapify" crash error stacktraces (#441)
Whenever test runner crashes due to UnhandledPromiseRejection, the
related error should be run through "sourcemap decoding".
2020-01-09 16:37:19 +00:00
Andrey Lushnikov a18777673e
devops(testrunner): support source maps (#340)
This patch adds a basic source map support to test runner.

SourceMap support is powered by Chromium DevTools source map
implementation (thus copyright). Unlike popular `source-map` npm
module, it's sync and pretty straight-forward.

The `SourceMap.js` file has a few modifications wrt upstream
Chromium version:
- reverse mappings API is removed. There's no need to ever compute them
- the `upperBoundary` function from DevTools' platform is inlined
2020-01-08 16:16:54 +00:00
Andrey Lushnikov a7f3490e48 feat(testrunner): implement "dit" command (#312) 2019-12-19 16:47:35 -07:00
Andrey Lushnikov 2acf36debc
fix(testrunner): xdescribe.skip should be a no-op 2019-12-19 00:46:03 -08:00
Andrey Lushnikov 8cc682c7b8 feat(testrunner): allow focusing all tests in a file
With this change, exporting a "fdescribe" function from a file
will focus all tests inside the file.
2019-12-18 17:13:39 -08:00