docs(ci): elaborate ci caching docs (#2192)

docs/ci.md

@@ -6,14 +6,14 @@ Playwright tests can be executed to run on your CI environments. To simplify thi
 - [CI configurations](#ci-configurations)
   * [GitHub Actions](#github-actions)
   * [Docker](#docker)
-    - [Tips](#tips)
   * [Azure Pipelines](#azure-pipelines)
   * [Travis CI](#travis-ci)
-    - [Tips](#tips-1)
   * [CircleCI](#circleci)
   * [AppVeyor](#appveyor)
-- [Debugging browser launches](#debugging-browser-launches)
 - [Caching browsers](#caching-browsers)
+    - [Exception: `node_modules` are cached](#exception-nodemodules-are-cached)
+    - [Directories to cache](#directories-to-cache)
+- [Debugging browser launches](#debugging-browser-launches)
 <!-- GEN:stop -->
 
 Broadly, configuration on CI involves **ensuring system dependencies** are in place, **installing Playwright and browsers** (typically with `npm install`), and **running tests** (typically with `npm test`). Windows and macOS build agents do not require any additional system dependencies. Linux build agents can require additional dependencies, depending on the Linux distribution.
@@ -37,7 +37,7 @@ We run [our tests](/.github/workflows/tests.yml) on GitHub Actions, across a mat
 
 We have a [pre-built Docker image](docker/README.md) which can either be used directly, or as a reference to update your existing Docker definitions.
 
-#### Tips
+Suggested configuration
 1. By default, Docker runs a container with a `/dev/shm` shared memory space 64MB.
    This is [typically too small](https://github.com/c0b/chrome-in-docker/issues/1) for Chromium
    and will cause Chromium to crash when rendering large pages. To fix, run the container with
@@ -72,9 +72,14 @@ For Linux agents, refer to [our Docker setup](docker/README.md) to see additiona
 
 We run our tests on Travis CI over a Linux agent (Ubuntu 18.04). Use our [Travis configuration](/.travis.yml) to see list of additional dependencies to be installed.
 
-#### Tips
+Suggested configuration
-- [User namespace cloning](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) should be enabled to support proper sandboxing
+1. [User namespace cloning](http://man7.org/linux/man-pages/man7/user_namespaces.7.html)
-- [xvfb](https://en.wikipedia.org/wiki/Xvfb) should be launched in order to run Chromium in non-headless mode (e.g. to test Chrome Extensions)
+   should be enabled to support proper sandboxing
+1. [xvfb](https://en.wikipedia.org/wiki/Xvfb) should be launched in order to run
+   Chromium in non-headless mode (e.g. to test Chrome Extensions)
+1. If your project does not have `package-lock.json`, Travis would be auto-caching
+   `node_modules` directory. If you run `npm install` (instead of `npm ci`), it is
+   possible that the browser binaries are not downloaded. Fix this with [these steps](#exception-nodemodules-are-cached) outlined below.
 
 To sum up, your `.travis.yml` might look like this:
 
@@ -84,6 +89,8 @@ dist: bionic
 addons:
   apt:
     packages:
+    # This is required to run chromium
+    - libgbm1
     # These are required to run webkit
     - libwoff1
     - libopus0
@@ -100,8 +107,8 @@ addons:
     - libnotify4
     - libxslt1.1
     - libvpx5
-    # This is required to run chromium
+    # For headful execution
-    - libgbm1
+    - xvfb
 
 # allow headful tests
 before_install:
@@ -139,6 +146,46 @@ We run our tests on CircleCI, with our [pre-built Docker image](docker/README.md
 
 We run our tests on Windows agents in AppVeyor. Use our [AppVeyor configuration](/.appveyor.yml) to create your own.
 
+## Caching browsers
+
+By default, Playwright downloads browser binaries when the Playwright NPM package
+is installed. The NPM packages have a `postinstall` hook that downloads the browser
+binaries. This behavior can be [customized with environment variables](installation.md).
+
+Caching browsers on CI is **strictly optional**: The `postinstall` hooks should
+execute and download the browser binaries on every run.
+
+#### Exception: `node_modules` are cached
+
+Most CI providers cache the [npm-cache](https://docs.npmjs.com/cli-commands/cache.html)
+directory (located at `$HOME/.npm`). If your CI pipelines caches the `node_modules`
+directory and you run `npm install` (instead of `npm ci`), the default configuration 
+**will not work**. This is because the `npm install` step will find the NPM
+package on disk, and not execute the `postinstall` step.
+
+> Travis CI automatically caches `node_modules` if your repo does not have a
+  `package-lock.json` file.
+
+This behavior can be fixed with one of the following approaches:
+1. Move to caching `$HOME/.npm` or the npm-cache directory. (This is the default
+   behavior in most CI providers.)
+1. Set `PLAYWRIGHT_BROWSERS_PATH=0` as the environment variable before running
+   `npm install`. This will download the browser binaries in the `node_modules`
+   directory and cache them with the package code. See [installation docs](installation.md).
+1. Cache the browser binaries, with the steps below.
+
+#### Directories to cache
+
+With the default behavior, Playwright downloads the browser binaries in the following
+directories:
+
+- `%USERPROFILE%\AppData\Local\ms-playwright` on Windows
+- `~/Library/Caches/ms-playwright` on MacOS
+- `~/.cache/ms-playwright` on Linux
+
+To cache the browser downloads between CI runs, cache this location in your CI
+configuration, against a hash of the Playwright version.
+
 ## Debugging browser launches
 
 Playwright supports the `DEBUG` environment variable to output debug logs during execution. Setting it to `pw:browser*` is helpful while debugging `Error: Failed to launch browser` errors.
@@ -146,13 +193,3 @@ Playwright supports the `DEBUG` environment variable to output debug logs during
 ```
 DEBUG=pw:browser* npm run test
 ```
-
-## Caching browsers
-
-By default, Playwright installs browser binaries in the following directories. This behavior can be [customized with environment variables](installation.md).
-
-- `%USERPROFILE%\AppData\Local\ms-playwright` on Windows
-- `~/Library/Caches/ms-playwright` on MacOS
-- `~/.cache/ms-playwright` on Linux
-
-These locations are not covered by typical CI configurations, which cache the project `node_modules` or the [npm-cache directory](https://docs.npmjs.com/cli-commands/cache.html). To cache the browser binaries between CI runs, cache this location in your CI configuration, against a hash of the Playwright version.