Commit graph

91 commits

Author SHA1 Message Date
Yury Semikhatsky 8b454cb3c3
devops: merge reports fall pw tests in tests 2 (#24309) 2023-07-19 13:01:52 -07:00
Yury Semikhatsky d92db9a513
devops: always create blob report on CI, write PR number within action (#24241) 2023-07-14 15:10:29 -07:00
Yury Semikhatsky 1b1cf87e0a
devops: add upload report action (#24222)
* Fix report downloading from Azure (reports are now zipped)
* Extracted upload logic into an action
* Extracted PR number file generation into its own job
2023-07-14 08:21:15 -07:00
Yury Semikhatsky 51b8f609fb
Revert "devops: zip blob report artifact before uploading (#23667)" (#23687)
This reverts commit a1cdae6bff.

The problem with this approach is that each job overwrites the zip
artifact whereas previously it was merging all reports in the same
directory. We are going to zip .jsonl files instead.
2023-06-13 12:36:04 -07:00
Yury Semikhatsky a1cdae6bff
devops: zip blob report artifact before uploading (#23667)
Compressed size for `tests 1` blob report is 19Mb whil uncompressed one
is 211Mb. Also according to [GitHub
policy](https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
) it is uncompressed size that is used for billing:

"Artifacts are uploaded during a workflow run, and you can view an
artifact's name and size in the UI. When an artifact is downloaded using
the GitHub UI, all files that were individually uploaded as part of the
artifact get zipped together into a single file. This means that billing
is calculated based on the size of the uploaded artifact and not the
size of the zip file."
2023-06-12 14:03:22 -07:00
Max Schmitt b21c81d344
devops: add bot for Chromium --headless=new (#23440)
https://github.com/microsoft/playwright/issues/23389
2023-06-02 20:40:12 +02:00
Yury Semikhatsky 2719f408b2
fix(devops): pass pr number in a file (#23456)
For some reason pull_requests field on workflow_run is empty for pull
requests created from branches in forked repositories, see
https://github.com/orgs/community/discussions/25220. As a workaround we
store triggering pull request number in a file.
2023-06-01 17:12:58 -07:00
Yury Semikhatsky 6cbcf18c48
fix(devops): path to blob report (#23444) 2023-06-01 10:17:34 -07:00
Yury Semikhatsky 972ad89af2
feat(devops): publish merged reports to pull requests (#23419)
* Moved report merging and publishing logic into create_test_report.yml
shared between all workflows
* Merged reports are now published for try jobs on pull requests too. In
order to achieve that the logic had to be extracted into a separate
workflow triggered by
[workflow_run](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_run),
this way it can access secrets even if the original workflow was not
able to.
* The blob report data flow is different depending on whether the
workflow is triggered by a pull request or a push:
- For `pull_request` the workflow doesn't have access to the secrets it
uploads the blob report to the GitHub
artifact storage. Later on the merge workflow uploads that blob report
to Azure blob storage.
- Workflows triggered by `push` event can read secrets. They upload blob
report directly to Azure blob storage
and the merge workflow downloads the report from there rather than from
GitHub artifacts.
2023-06-01 08:40:43 -07:00
Yury Semikhatsky ce9892c694
fix(devops): azure conection string var name (#23416) 2023-05-31 16:54:44 -07:00
Max Schmitt 92d650c317
devops: add missing workflow permission (#23413) 2023-06-01 01:19:23 +02:00
Yury Semikhatsky 90aa378e8f
chore: extract batch download action (#23410) 2023-05-31 16:07:29 -07:00
Max Schmitt 40581ed906
devops: post merged test results comment as GitHub Status Check (#23409) 2023-06-01 00:29:57 +02:00
Yury Semikhatsky 38c89df330
chore: replace "blob download-batch" with a bash script (#23381)
`az storage blob download-batch` has been timing out over the last few
days, see upstream issue
https://github.com/Azure/azure-cli/issues/26567. Replacing it with a
simple bash script that discovers blobs with a given prefix and then
downloads one-by-one.
2023-05-30 16:46:37 -07:00
Max Schmitt 1f7223eb21
chore: test on Node.js 20 (#22651)
Fixes https://github.com/microsoft/playwright/issues/22582
2023-05-30 18:16:34 +02:00
Yury Semikhatsky 1e64fa4f7c
chore: no empty dash suffix in blob report (#23285) 2023-05-25 13:18:57 -07:00
Yury Semikhatsky 0a65a0b8e8
devops: post merged report URL to the PR (#23266)
Posting a comment to the PR using GITHUB_TOKEN requires elevated
permissions, see
https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
2023-05-24 14:43:36 -07:00
Yury Semikhatsky 10b7cb3979
chore: blob report project suffix (#23212) 2023-05-22 17:54:37 -07:00
Yury Semikhatsky c8a42af11f
devops: merged report for tracing tests (#23167) 2023-05-19 13:31:37 -07:00
Yury Semikhatsky 5004f908d9
devops: do not upload test-results artifacts (#23168) 2023-05-19 13:06:42 -07:00
Yury Semikhatsky 27403dcbb4
devops: remove explicit npm@8 installation (#23096)
Node 16+ comes with npm 8+ by default.
2023-05-17 14:32:38 -07:00
Max Schmitt 89aa02228f
devops: add macOS 13 coverage (#23065)
Coverage https://github.com/microsoft/playwright/issues/22226
2023-05-17 01:13:09 +02:00
Max Schmitt 44b63b042e
devops: do not download electron on non-electron workflows (#22232) 2023-04-06 19:49:29 +02:00
Max Schmitt 6682fb6075
devops: bump Node.js version 14 -> 16 (#20800)
Fixes https://github.com/microsoft/playwright/issues/20788
2023-02-10 00:53:32 +01:00
Dmitry Gozman edfff8cd8c
test: run more tests in 'service' mode (#20734) 2023-02-07 18:25:33 -08:00
Andrey Lushnikov 0a752f3fd6
chore(docker): remove experimental docker integration (#20676)
This removes everything related to docker integration experiments that
we conducted over the last 6 months.

I'll send a follow-up with an alternative suggestion that was demo'ed on
a team meeting in the end of December.
2023-02-06 10:50:45 -08:00
Max Schmitt 3acf3f5cae
devops: fix service tests 2
Signed-off-by: Max Schmitt <max@schmitt.mx>
2022-12-21 21:10:44 +01:00
Max Schmitt 2c801f9309
test: fix service tests (#19592) 2022-12-20 15:37:30 +01:00
Max Schmitt 6a5552ba24
chore: drop Ubuntu 18.04 support (#18924) 2022-11-21 11:23:45 -10:00
Andrey Lushnikov db826c9c8c
chore: drive-by fixes for the snapshotPathTemplate (#18715)
- Use `snapshotPathTemplate` for docker screenshots in html-reporter
- Mark the snapshot path template test as slow since it re-spawns
  worker for each project.
- Fix docker smoke tests
2022-11-10 15:50:13 -08:00
Max Schmitt d1d24bb490
Revert "devops: use debug Node.js build on CI for browser downloads (#17504)" (#17711)
This reverts commit 56df171d17.
2022-09-29 21:08:28 +02:00
Max Schmitt 56df171d17
devops: use debug Node.js build on CI for browser downloads (#17504)
https://github.com/microsoft/playwright/issues/17394
2022-09-28 10:52:30 +02:00
Andrey Lushnikov a12112c24d
devops(docker): add docker integration smoke tests (#17267) 2022-09-14 15:05:18 -07:00
Max Schmitt e8b9d4e758
devops: bump github actions (#16952) 2022-08-31 17:23:42 +02:00
Andrey Lushnikov f3269c6891
tests: do not test on MacOS 10.15 any more (#16465)
Playwright v1.25 was the last release to support macOS 10.15, so we
no longer need to test on macOS 10.15.
2022-08-11 09:50:44 -07:00
Ross Wollman a3836de182
chore: relax npm version constraint in installation tests (#16427)
Resolves #16281.
2022-08-10 15:33:47 -07:00
Ross Wollman 7c4099a011
tests(installation): pin npm version as workaround (#16323)
Relates https://github.com/microsoft/playwright/issues/16281.
2022-08-05 15:24:47 -07:00
Max Schmitt 93a96347c2
devops: use bash always for upload_flakiness_dashboard (#16298) 2022-08-05 10:47:44 +02:00
Max Schmitt 57637d4a4a
devops: upload edge stable,beta,dev to flakiness dashboard (#16297) 2022-08-05 09:51:45 +02:00
Yury Semikhatsky 37f9500efa
devops: add macos-12 bots, rearrange others (#16218)
* Added macos-12 bots to the secondary workflow
* Moved ubuntu 20.04 from primary to secondary workflow
* For the bots where we don't care about macos version (Chrome Stable, Edge Dev etc.) switched to macos-latest

References #16180
2022-08-03 12:34:41 -07:00
Max Schmitt deaef17e70
devops: run tests on ubuntu 22 (#14741) 2022-06-09 14:30:55 +02:00
Dmitry Gozman a5506a80ee
test: normalize node versions for installation tests (#14610)
We should cover node 14 on all platforms (similarly to
other types of tests), and node 16/18 on linux.
2022-06-07 18:11:58 -07:00
Dmitry Gozman c5face5ecc
test: add more chromium-tip-of-tree bots (#14120) 2022-05-12 15:44:13 +01:00
Andrey Lushnikov 3efd0e15e5
devops: upload all headed tests to flakiness (#14056) 2022-05-10 06:49:28 -07:00
Pavel Feldman 1658172b2c
chore: run headed on all OSs (#14042) 2022-05-09 12:55:59 -07:00
Andrey Lushnikov bc9ef1c76e
devops: follow-up with installation tests on flakienss dashbaord (#14015)
- upload more installation tests to the dashboard
- use json reporter on CI
2022-05-07 09:17:07 -07:00
Pavel Feldman f4933e6d8b
chore(electron): add Win+Mac bots (#13943) 2022-05-04 12:29:22 -07:00
Ross Wollman c4af7f2fc8
fix: unbreak installation tests, include in primary CI runs (#13903) 2022-05-03 12:48:35 -07:00
Ross Wollman 3e84ab4701
test: run installation tests on all major platforms (#13742) 2022-04-26 11:09:49 -07:00
Max Schmitt f7af79137b
devops: unpin NPM on CI and use again version 8 (#13765) 2022-04-26 17:52:10 +02:00