- Use file path, not content to calculate the attachment hash.
- Always cleanup fixture from the list on teardown, to avoid reporting
teardown error multiple times: from the test, and from the cleanup.
We do not have a timeout for any other close method, such as
browserContext.close or browser.close, and hitting default
30 seconds is very realistic with large Electron apps.
Previously, reporter would look for a stack frame directly in the test file.
Often times, that is not a top stack frame, especially when the test uses
some helper functions.
This changes error snippets and locations to use the top frame. When top
frame does not match the test file, we additionally show the location
to avoid confusion:
```
1) a.spec.ts:7:7 › foobar ========================================================================
Error: oh my
at helper.ts:5
3 |
4 | export function ohMy() {
> 5 | throw new Error('oh my');
| ^
6 | }
7 |
at ohMy (.../reporter-base-should-print-codeframe-from-a-helper/helper.ts:5:15)
at .../reporter-base-should-print-codeframe-from-a-helper/a.spec.ts:8:9
at FixtureRunner.resolveParametersAndRunHookOrTest (.../src/fixtures.ts:281:12)
```
In case of self-hosted github runners, it's much easier to checkout
under `playwright-internal` folder name instead of a default
`playwright` name. This confuses our stack collection.
This patch makes it generic.
This changes previous layout shift attempt (see #9546)
to account for more valid usecases:
- On the first event that is intercepted we enforce the hit target. This
is similar to the current mode that checks hit target before the action,
but is better timed.
- On subsequent events we assume that everything is fine. This covers more
scenarios like react rerender, glass pane on mousedown, detach on mouseup.
This check is enabled by default, with `process.env.PLAYWRIGHT_NO_LAYOUT_SHIFT_CHECK`
to opt out.
Textual snapshot diffs were previously broken in the HTML Report. The strikethrough'd text extended beyond the intended region.
HTML Report Before:
<img width="693" alt="Screen Shot 2021-12-27 at 4 43 35 PM" src="https://user-images.githubusercontent.com/11915034/147518750-a60f9002-6eed-48a1-a412-20fabd076fa6.png">
HTML Report After:
<img width="206" alt="Screen Shot 2021-12-27 at 4 48 37 PM" src="https://user-images.githubusercontent.com/11915034/147518762-19a4c8f9-ccc3-4a3c-a962-5a42edc6fc5d.png">
This now matches what's expected and shown in the terminal (which has always been correct):
<img width="1384" alt="Screen Shot 2021-12-27 at 4 36 29 PM" src="https://user-images.githubusercontent.com/11915034/147518799-f538259e-5a45-4d6f-916c-a12ccb620c5b.png">
NB: This MR is a workaround, but not a root cause fix. It works, but I never fully got to the root cause so a bug upstream may be required. It's unclear whether it's (1) in [`colors`](https://www.npmjs.com/package/colors), (2) in [`ansi-to-html`](https://www.npmjs.com/package/ansi-to-html), or (3) Playwright's use of the two. Since the terminal output is correct, I suspect it is in `ansi-to-html`. For example:
```js
const colors = require("colors");
const Convert = require('ansi-to-html');
const convert = new Convert();
// original (strike incorrectly wraps everything in the HTML)
console.log(convert.toHtml(colors.strikethrough("crossed out") + ' ' + colors.red("red")))
// prints: <strike>crossed out <span style="color:#A00">red<span style="color:#FFF"></span></span></strike>
// workaround
console.log(convert.toHtml(colors.reset(colors.strikethrough("crossed out")) + ' ' + colors.red("red")))
// prints: <strike>crossed out</strike> <span style="color:#A00">red<span style="color:#FFF"></span></span>
```
Fixes#11116
## Before
* When docker wasn't running: crashed with a TypeError
* When any image without `RepoTags` was present: crashed with TypeError
* Typo in recommended command ("playwight")
* No indication of `PW_TEST_IMAGE` env var in error output
<details>
<summary>Docker not running</summary>
```
Using config at ~redacted/playwright.config.ts
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused
TypeError: Cannot read property 'find' of null
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
test.ts:9:1 › some redacted description
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused
TypeError: Cannot read property 'find' of null
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js
```
</details>
<details>
<summary>Image without repo tags</summary>
```
Using config at ~redacted/playwright.config.ts
TypeError: Cannot read property 'includes' of undefined
at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
at Array.find (<anonymous>)
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
some-test.ts › some description
TypeError: Cannot read property 'includes' of undefined
at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
at Array.find (<anonymous>)
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element (26ms)
1) some-test.ts › some description =====
'Grid agent creation failed'
1 failed
some-test.ts › some description ======
```
</details>
## After
* Helpful error message if docker isn't running
* Doesn't crash when local-only images in list
* No typo in `playwright install docker-image` command
* When other playwright images are found they are listed and `PW_TEST_IMAGE` is mentioned
### After: Docker not running
```
Using config at ~redacted/playwright.config.ts
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images ║
║ Please ensure docker daemon is running. ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
test.ts:9:1 › description
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images ║
║ Please ensure docker daemon is running. ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ test.ts:9:1 › description
1) test.ts:9:1 › description =====
'Grid agent creation failed'
1 failed
test.ts:9:1 › description ======
```
## After: No exact match found
```
Using config at ~redacted/playwright.config.ts
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image. ║
║ ║
║ Available images: ║
║ - mcr.microsoft.com/playwright:v1.16.3 ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal ║
║ ║
║ Use available images via PWTEST_IMAGE_NAME environment variable: ║
║ PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║ ║
║ Alternatively, please pull docker image with the following command: ║
║ ║
║ npx playwright install docker-image ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)
Running 1 test using 1 worker
editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image. ║
║ ║
║ Available images: ║
║ - mcr.microsoft.com/playwright:v1.16.3 ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal ║
║ ║
║ Use available images via PWTEST_IMAGE_NAME environment variable: ║
║ PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║ ║
║ Alternatively, please pull docker image with the following command: ║
║ ║
║ npx playwright install docker-image ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════╝
at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j ✘ test.ts:9:1 › description
1) test.ts:9:1 › description =====
'Grid agent creation failed'
1 failed
test.ts:9:1 › description ======
```
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
