Commit graph

27 commits

Author SHA1 Message Date
Andrey Lushnikov 05b019f1ba
reland: testrunner: make environment a simple class (#2812)
This re-lands PR https://github.com/microsoft/playwright/pull/2769

It was reverted before in https://github.com/microsoft/playwright/pull/2790
because it was breaking the new CHANNEL bot.
2020-07-02 11:05:38 -07:00
Pavel Feldman e467ea57e7
revert: testrunner: make environment a simple class (#2769) (#2790) 2020-07-01 10:56:19 -07:00
Andrey Lushnikov 1605cb453c
testrunner: make environment a simple class (#2769)
This patch:
- makes environment a simple class with optional methods `beforeEach`, `afterEach`, `beforeAll`, `afterAll`, `globalSetup` and `globalTeardown`
- removes capability to have multiple hooks of the same name inside suite
- removes default environment for test. (`dit` now adds a `TraceTestEnvironment` to the test)
- extracts all environments that we use in our tests in `//test/environments.js`

Downsides:
- we no longer know hook locations for the environments. This, however, should not be a big deal since stack traces (if any) will still point into it.
- this also regresses hook locations for suites for simplicity. We can get them back, but it shouldn't be pressing since we now have only one hook of each kind in every suite.
2020-06-30 16:51:03 -07:00
Andrey Lushnikov 1865c5685a
testrunner: support globalSetup and globalTeardown hooks. (#2686) 2020-06-24 22:08:46 -07:00
Andrey Lushnikov c61e2d6cc9
testrunner: drop nested test environments (#2681) 2020-06-23 16:45:00 -07:00
Andrey Lushnikov a9f0c40af3
feat(testrunner): improvements to testrunner (#2070)
This patch:
- teaches test runner to understand custom argument spelling, e.g. `--file=evalu` and `-j10`
- fixes `--file` filter to actually focus file paths instead of focusing
all tests with given path
2020-05-01 10:09:45 -07:00
Andrey Lushnikov 167d265e56
fix(testrunner): make .repeat() retain test order (#1814) 2020-04-15 18:41:55 -07:00
Dmitry Gozman 20ff327827
feat(testrunner): catch delegate errors (#1704)
This ensures we get a proper error when something goes wrong. Should
also help with producing the right error code in the case of internal error.

Drive-by: fix location issue which manifests on the bots.
Drive-by: remove the use of Array.prototype.flat to make it work on bots.
2020-04-07 22:56:21 -07:00
Dmitry Gozman f2b13c0e93
chore(testrunner): split TestRunner into parts (#1679) 2020-04-06 17:21:42 -07:00
Dmitry Gozman 823f961d8d
feat(testrunner): migrate from events to a delegate (#1647)
This allows an async handler for each event that can be awaited.
Drive-by: merge TestPass into TestRunner.
2020-04-03 15:47:25 -07:00
Dmitry Gozman 1f2803bbc6
feat(testrunner): removeEnvironment (#1650) 2020-04-03 09:48:01 -07:00
Dmitry Gozman d38baae01c
feat(testrunner): nested environments (#1635) 2020-04-02 13:12:22 -07:00
Dmitry Gozman f87e64544c
feat(testrunner): introduce environments (#1593) 2020-04-01 10:49:47 -07:00
Dmitry Gozman b6166c9669
chore(testrunner): introduce Location class (#1585)
Drive-by: fix an edge when testing continued after termination.
2020-03-29 21:38:30 -07:00
Dmitry Gozman c49b856d6d
chore(testrunner): remove setup() helper (#1584) 2020-03-29 11:37:45 -07:00
Dmitry Gozman a41836b1f1
chore(testrunner): introduce TestRun (#1582) 2020-03-28 14:25:57 -07:00
Dmitry Gozman b85ab891a1
chore(testrunner): make most modifiers external (#1581) 2020-03-28 08:49:00 -07:00
Dmitry Gozman 4bd46bafa9
feat(testurnner): allow multiple hooks isntances and per-test hooks (#1571) 2020-03-27 22:57:22 -07:00
Dmitry Gozman a042466125
chore(testrunner): remove effectiveMode and effectiveExpectation (#1569) 2020-03-27 15:35:14 -07:00
Dmitry Gozman af7a16c360
chore(testrunner): merge test spec with test, suite spec with suite (#1566) 2020-03-26 22:47:13 -07:00
Dmitry Gozman b61198458c
feat(testrunner): modifiers and attributes (#1538)
This generalizes test modifiers to support custom ones.
2020-03-25 14:40:57 -07:00
Dmitry Gozman 3960b179a1
fix(testrunner): fit.fail should run the test (#1407) 2020-03-16 19:12:52 -07:00
Andrey Lushnikov 1cd00bd068
feat(testrunner): allow filtering by name and show all focused tests (#1354) 2020-03-11 18:30:43 -07:00
Dmitry Gozman 0ce8efab7b
test: rework testrunner workers (#1296)
This change introduces a TestWorker that can be in a certain state,
meaning it has run some beforeAll hooks of a certain test suite stack.

TestWorker can be created at any time, which allows for a number of features:
- don't run hooks for suites with no runnable tests;
- smarter test distribution (and possibility for variuos strategies);
- recovering from hook failures and test failure by creating a new worker;
- possible isolation between workers by running them in separate environments.
2020-03-10 11:30:02 -07:00
Andrey Lushnikov 0ded511d0b
feat(testrunner): better matchers (#1077)
This patch re-implements matching and reporting for test runner.
Among other improvements:
- test failures now show a short snippet from test
- test failures now explicitly say what received and what was expected
- `expect.toBe()` now does text diff when gets strings as input
- `expect.toEqual` now does object diff
2020-02-20 22:55:39 -08:00
Andrey Lushnikov a7f3490e48 feat(testrunner): implement "dit" command (#312) 2019-12-19 16:47:35 -07:00
Pavel Feldman 9ba375c063 Initial commit 2019-11-19 10:58:15 -08:00