Merge 110ae95ce8 into fe46e0c363

Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>

.github/workflows/main.yml

@@ -2,6 +2,7 @@ name: "Spec"
 
 env:
   HUGO_VERSION: 0.139.0
+  PYTHON_VERSION: 3.13
 
 on:
   push:
@@ -40,7 +41,7 @@ jobs:
       - name: "➕ Setup Python"
         uses: actions/setup-python@v5
         with:
-          python-version: '3.9'
+          python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip'
           cache-dependency-path: scripts/requirements.txt
       - name: "➕ Install dependencies"
@@ -59,7 +60,7 @@ jobs:
       - name: "➕ Setup Python"
         uses: actions/setup-python@v5
         with:
-          python-version: '3.9'
+          python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip'
           cache-dependency-path: scripts/requirements.txt
       - name: "➕ Install dependencies"
@@ -78,7 +79,7 @@ jobs:
       - name: "➕ Setup Python"
         uses: actions/setup-python@v5
         with:
-          python-version: '3.9'
+          python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip'
           cache-dependency-path: scripts/requirements.txt
       - name: "➕ Install dependencies"
@@ -120,7 +121,7 @@ jobs:
       - name: "➕ Setup Python"
         uses: actions/setup-python@v5
         with:
-          python-version: '3.9'
+          python-version: ${{ env.PYTHON_VERSION }}
           cache: 'pip'
           cache-dependency-path: scripts/requirements.txt
       - name: "➕ Install dependencies"
@@ -172,7 +173,7 @@ jobs:
       - name: "➕ Setup Python"
         uses: actions/setup-python@v5
         with:
-          python-version: '3.9'
+          python-version: ${{ env.PYTHON_VERSION }}
       - name: "➕ Install towncrier"
         run: "pip install 'towncrier'"
       - name: "Generate changelog"

changelogs/client_server/newsfragments/2104.clarification

@@ -0,0 +1 @@
+Rooms published in `/publicRooms` don't necessarily have `public` join rules or `world_readable` history visibility.

changelogs/internal/newsfragments/2088.clarification

@@ -0,0 +1 @@
+Replace Hugo shortcodes in OpenAPI output.

changelogs/server_server/newsfragments/2067.clarification

@@ -0,0 +1 @@
+Add a note to the invite endpoints that invites to local users may be received twice over federation if the homeserver is already in the room.

changelogs/server_server/newsfragments/2104.clarification

@@ -0,0 +1 @@
+Rooms published in `/publicRooms` don't necessarily have `public` join rules or `world_readable` history visibility.

content/client-server-api/_index.md

@@ -2831,7 +2831,38 @@ re-invited.
 
 {{% http-api spec="client-server" api="banning" %}}
 
-### Listing rooms
+### Room directory
+
+Homeservers MAY publish a room directory to allow users to discover rooms. A room
+can have one of two visibility settings in the directory:
+
+`public`
+The room will be shown in the published room directory.
+
+`private`
+The room will be hidden from the published room directory.
+
+Clients can define a room's initial visibility in the directory via the `visibility`
+parameter in [`/createRoom`](#post_matrixclientv3createroom). Irrespective of room
+creation, clients can query and change a room's visibility in the directory through
+the endpoints listed below, provided that the server permits this.
+
+{{% boxes/warning %}}
+The visibility setting merely defines whether a room is included in the published
+room directory or not. It doesn't make any guarantees about the room's
+[join rule](#mroomjoin_rules) or [history visibility](#room-history-visibility).
+
+In particular, a visibility setting of `public` should not be confused with a `public`
+join rule. Rooms with a join rule of `knock`, for instance, could reasonably be published
+in the directory, too.
+
+Similarly, a visibility setting of `public` does not necessarily imply a `world_readable`
+history visibility.
+
+To increase performance, servers MAY apply additional filters when listing the
+directory, for instance, by automatically excluding rooms with `invite` join rules
+that are not `world_readable` regardless of their visibility.
+{{% /boxes/warning %}}
 
 {{% http-api spec="client-server" api="list_public_rooms" %}}
 

content/server-server-api.md

@@ -1047,11 +1047,10 @@ user's Matrix ID and the token delivered when the invite was stored,
 this verification will prove that the `m.room.member` invite event comes
 from the user owning the invited third-party identifier.
 
-## Public Room Directory
+## Room Directory
 
-To complement the [Client-Server
+To complement the [room directory in the Client-Server API](/client-server-api#room-directory), 
-API](/client-server-api)'s room directory,
+homeservers need a way to query the published rooms of another server.
-homeservers need a way to query the public rooms for another server.
 This can be done by making a request to the `/publicRooms` endpoint for
 the server the room directory should be retrieved for.
 

data/api/client-server/create_room.yaml

@@ -87,12 +87,8 @@ paths:
                     - public
                     - private
                   description: |-
-                    A `public` visibility indicates that the room will be shown
+                    The room's visibility in the server's [room directory](#room-directory).
-                    in the published room list. A `private` visibility will hide
+                    Defaults to `private`.
-                    the room from the published room list. Rooms default to
-                    `private` visibility if this key is not included. NB: This
-                    should not be confused with `join_rules` which also uses the
-                    word `public`.
                 room_alias_name:
                   type: string
                   description: |-

data/api/client-server/definitions/public_rooms_chunk.yaml

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 type: object
-title: "PublicRoomsChunk"
+title: "PublishedRoomsChunk"
 properties:
   canonical_alias:
     type: string

data/api/client-server/definitions/public_rooms_response.yaml

@@ -13,28 +13,15 @@
 # limitations under the License.
 
 type: object
-description: A list of the rooms on the server.
+description: A list of the published rooms on the server.
 required: ["chunk"]
 properties:
   chunk:
     type: array
     description: |-
-      A paginated chunk of public rooms.
+      A paginated chunk of published rooms.
     items:
-      allOf:
+      $ref: "public_rooms_chunk.yaml"
-        - $ref: "public_rooms_chunk.yaml"
-        - type: object
-          title: PublicRoomsChunk
-          properties:
-            # Override description of join_rule
-            join_rule:
-              type: string
-              description: |-
-                The room's join rule. When not present, the room is assumed to
-                be `public`. Note that rooms with `invite` join rules are not
-                expected here, but rooms with `knock` rules are given their
-                near-public nature.
-              example: "public"
   next_batch:
     type: string
     description: |-
@@ -50,7 +37,7 @@ properties:
   total_room_count_estimate:
     type: integer
     description: |-
-        An estimate on the total number of public rooms, if the
+        An estimate on the total number of published rooms, if the
         server has an estimate.
 example: {
   "chunk": [

data/api/client-server/list_public_rooms.yaml

@@ -19,8 +19,7 @@ paths:
   "/directory/list/room/{roomId}":
     get:
       summary: Gets the visibility of a room in the directory
-      description: Gets the visibility of a given room on the server's public room
+      description: Gets the visibility of a given room in the server's room directory.
-        directory.
       operationId: getRoomVisibilityOnDirectory
       parameters:
         - in: path
@@ -32,7 +31,7 @@ paths:
             type: string
       responses:
         "200":
-          description: The visibility of the room in the directory
+          description: The visibility of the room in the directory.
           content:
             application/json:
               schema:
@@ -50,7 +49,7 @@ paths:
                     "visibility": "public"
                   }
         "404":
-          description: The room is not known to the server
+          description: The room is not known to the server.
           content:
             application/json:
               schema:
@@ -66,12 +65,11 @@ paths:
     put:
       summary: Sets the visibility of a room in the room directory
       description: |-
-        Sets the visibility of a given room in the server's public room
+        Sets the visibility of a given room in the server's room directory.
-        directory.
 
-        Servers may choose to implement additional access control checks
+        Servers MAY implement additional access control checks, for instance,
-        here, for instance that room visibility can only be changed by
+        to ensure that a room's visibility can only be changed by the room creator
-        the room creator or a server administrator.
+        or a server administrator.
       operationId: setRoomVisibilityOnDirectory
       security:
         - accessTokenQuery: []
@@ -97,7 +95,7 @@ paths:
                     - public
                   description: |-
                     The new visibility setting for the room.
-                    Defaults to 'public'.
+                    Defaults to `public`.
               example: {
                 "visibility": "public"
               }
@@ -114,7 +112,7 @@ paths:
                 response:
                   value: {}
         "404":
-          description: The room is not known to the server
+          description: The room is not known to the server.
           content:
             application/json:
               schema:
@@ -129,9 +127,9 @@ paths:
         - Room discovery
   /publicRooms:
     get:
-      summary: Lists the public rooms on the server.
+      summary: Lists a server's published room directory
       description: |-
-        Lists the public rooms on the server.
+        Lists a server's published room directory.
 
         This API returns paginated responses. The rooms are ordered by the number
         of joined members, with the largest rooms first.
@@ -154,13 +152,13 @@ paths:
         - in: query
           name: server
           description: |-
-            The server to fetch the public room lists from. Defaults to the
+            The server to fetch the room directory from. Defaults to the
             local server. Case sensitive.
           schema:
             type: string
       responses:
         "200":
-          description: A list of the rooms on the server.
+          description: A list of the published rooms on the server.
           content:
             application/json:
               schema:
@@ -168,9 +166,9 @@ paths:
       tags:
         - Room discovery
     post:
-      summary: Lists the public rooms on the server with optional filter.
+      summary: Lists a server's published room directory with an optional filter
       description: |-
-        Lists the public rooms on the server, with optional filter.
+        Lists a server's published room directory with an optional filter.
 
         This API returns paginated responses. The rooms are ordered by the number
         of joined members, with the largest rooms first.
@@ -182,7 +180,7 @@ paths:
         - in: query
           name: server
           description: |-
-            The server to fetch the public room lists from. Defaults to the
+            The server to fetch the room directory from. Defaults to the
             local server. Case sensitive.
           schema:
             type: string
@@ -253,7 +251,7 @@ paths:
         required: true
       responses:
         "200":
-          description: A list of the rooms on the server.
+          description: A filtered list of the published rooms on the server.
           content:
             application/json:
               schema:

data/api/server-server/invites-v1.yaml

@@ -20,7 +20,7 @@ paths:
     put:
       summary: Invites a remote user to a room
       description: |-
-        Invites a remote user to a room. Once the event has been  signed by both the inviting
+        Invites a remote user to a room. Once the event has been signed by both the inviting
         homeserver and the invited homeserver, it can be sent to all of the servers in the
         room by the inviting homeserver.
 
@@ -32,6 +32,10 @@ paths:
         [room version specification](/rooms) for precise event formats. **The request and response
         bodies here describe the common event fields in more detail and may be missing other
         required fields for a PDU.**
+
+        Also note that if the remote homeserver is already in the room, it will receive the
+        invite event twice; once through this endpoint, and again through a [federation
+        transaction](/server-server-api/#transactions).
       operationId: sendInviteV1
       security:
         - signedRequest: []

data/api/server-server/invites-v2.yaml

@@ -24,7 +24,7 @@ paths:
         This API is nearly identical to the v1 API with the exception of the request
         body being different, and the response format fixed.
 
-        Invites a remote user to a room. Once the event has been  signed by both the inviting
+        Invites a remote user to a room. Once the event has been signed by both the inviting
         homeserver and the invited homeserver, it can be sent to all of the servers in the
         room by the inviting homeserver.
 
@@ -36,6 +36,10 @@ paths:
         [room version specification](/rooms) for precise event formats. **The request and response
         bodies here describe the common event fields in more detail and may be missing other
         required fields for a PDU.**
+
+        Also note that if the remote homeserver is already in the room, it will receive the
+        invite event twice; once through this endpoint, and again through a [federation
+        transaction](/server-server-api/#transactions).
       operationId: sendInviteV2
       security:
         - signedRequest: []

data/api/server-server/public_rooms.yaml

@@ -13,16 +13,20 @@
 # limitations under the License.
 openapi: 3.1.0
 info:
-  title: Matrix Federation Public Rooms API
+  title: Matrix Federation Room Directory API
   version: 1.0.0
 paths:
   /publicRooms:
     get:
-      summary: Get all the public rooms for a homeserver
+      summary: Lists the server's published room directory.
       description: |-
-        Gets all the public rooms for the homeserver. This should not return
+        Lists the server's published room directory.
-        rooms that are listed on another homeserver's directory, just those
+
-        listed on the receiving homeserver's directory.
+        This API returns paginated responses. The rooms are ordered by the number
+        of joined members, with the largest rooms first.
+
+        This SHOULD not return rooms that are listed on another homeserver's directory,
+        just those listed on the receiving homeserver's directory.
       operationId: getPublicRooms
       security:
         - signedRequest: []
@@ -62,21 +66,18 @@ paths:
             type: string
       responses:
         "200":
-          description: The public room list for the homeserver.
+          description: A list of the published rooms on the server.
           content:
             application/json:
               schema:
                 $ref: ../client-server/definitions/public_rooms_response.yaml
     post:
-      summary: Gets the public rooms on the server with optional filter.
+      summary: Lists the server's published room directory with an optional filter
       description: |-
-        Lists the public rooms on the server, with optional filter.
+        Lists the server's published room directory with an optional filter.
 
         This API returns paginated responses. The rooms are ordered by the number
         of joined members, with the largest rooms first.
-
-        Note that this endpoint receives and returns the same format that is seen
-        in the Client-Server API's `POST /publicRooms` endpoint.
       operationId: queryPublicRooms
       security:
         - signedRequest: []
@@ -147,69 +148,11 @@ paths:
         required: true
       responses:
         "200":
-          description: A list of the rooms on the server.
+          description: A filtered list of the published rooms on the server.
           content:
             application/json:
               schema:
-                type: object
+                $ref: ../client-server/definitions/public_rooms_response.yaml
-                description: A list of the rooms on the server.
-                required:
-                  - chunk
-                properties:
-                  chunk:
-                    title: PublicRoomsChunks
-                    type: array
-                    description: A paginated chunk of public rooms.
-                    items:
-                      allOf:
-                        - $ref: ../client-server/definitions/public_rooms_chunk.yaml
-                        - type: object
-                          properties:
-                            # Override description of join_rule
-                            join_rule:
-                              type: string
-                              description: |-
-                                The room's join rule. When not present, the room is assumed to
-                                be `public`. Note that rooms with `invite` join rules are not
-                                expected here, but rooms with `knock` rules are given their
-                                near-public nature.
-                  next_batch:
-                    type: string
-                    description: |-
-                      A pagination token for the response. The absence of this token
-                      means there are no more results to fetch and the client should
-                      stop paginating.
-                  prev_batch:
-                    type: string
-                    description: |-
-                      A pagination token that allows fetching previous results. The
-                      absence of this token means there are no results before this
-                      batch, i.e. this is the first batch.
-                  total_room_count_estimate:
-                    type: integer
-                    description: |-
-                      An estimate on the total number of public rooms, if the
-                      server has an estimate.
-              examples:
-                response:
-                  value: {
-                    "chunk": [
-                      {
-                        "avatar_url": "mxc://bleecker.street/CHEDDARandBRIE",
-                        "guest_can_join": false,
-                        "name": "CHEESE",
-                        "num_joined_members": 37,
-                        "room_id": "!ol19s:bleecker.street",
-                        "topic": "Tasty tasty cheese",
-                        "world_readable": true,
-                        "join_rule": "public",
-                        "room_type": "m.space"
-                      }
-                    ],
-                    "next_batch": "p190q",
-                    "prev_batch": "p1902",
-                    "total_room_count_estimate": 115
-                  }
 servers:
   - url: "{protocol}://{hostname}{basePath}"
     variables:

scripts/dump-openapi.py

@@ -32,6 +32,35 @@ import yaml
 scripts_dir = os.path.dirname(os.path.abspath(__file__))
 api_dir = os.path.join(os.path.dirname(scripts_dir), "data", "api")
 
+# Finds a Hugo shortcode in a string.
+#
+# A shortcode is defined as (newlines and whitespaces for presentation purpose):
+#
+# {{%
+#     <zero or more whitespaces>
+#     <name of shortcode>
+#     (optional <one or more whitespaces><list of parameters>)
+#     <zero or more whitespaces>
+# %}}
+#
+# With:
+#
+# * <name of shortcode>: any word character and `-` and `/`. `re.ASCII` is used to only match
+#   ASCII characters in the name.
+# * <list of parameters>: any character except `}`, must not start or end with a
+#   whitespace.
+shortcode_regex = re.compile(r"""\{\{\%                                   # {{%
+                                 \s*                                      # zero or more whitespaces
+                                 (?P<name>[\w/-]+)                        # name of shortcode
+                                 (?:\s+(?P<params>[^\s\}][^\}]+[^\s\}]))? # optional list of parameters
+                                 \s*                                      # zero or more whitespaces
+                                 \%\}\}                                   # %}}""", re.ASCII | re.VERBOSE)
+
+# Parses the parameters of a Hugo shortcode.
+#
+# For simplicity, this currently only supports the `key="value"` format.
+shortcode_params_regex = re.compile(r"(?P<key>\w+)=\"(?P<value>[^\"]+)\"", re.ASCII)
+
 def prefix_absolute_path_references(text, base_url):
     """Adds base_url to absolute-path references.
 
@@ -44,17 +73,90 @@ def prefix_absolute_path_references(text, base_url):
     """
     return text.replace("](/", "]({}/".format(base_url))
 
-def edit_links(node, base_url):
+def replace_match(match, replacement):
-    """Finds description nodes and makes any links in them absolute."""
+    """Replaces the regex match by the replacement in the text."""
+    return match.string[:match.start()] + replacement + match.string[match.end():]
+
+def replace_shortcode(shortcode):
+    """Replaces the shortcode by a Markdown fallback in the text.
+
+    The supported shortcodes are:
+
+    * boxes/note, boxes/rationale, boxes/warning
+    * added-in, changed-in
+    
+    All closing tags (`{{ /shortcode }}`) are replaced with the empty string.
+    """
+
+    if shortcode['name'].startswith("/"):
+        # This is the end of the shortcode, just remove it.
+        return replace_match(shortcode, "")
+
+    # Parse the parameters of the shortcode
+    params = {}
+    if shortcode['params']:
+        for param in shortcode_params_regex.finditer(shortcode['params']):
+            if param['key']:
+                params[param['key']] = param['value']
+
+    match shortcode['name']:
+        case "boxes/note":
+            return replace_match(shortcode, "**NOTE:** ")
+        case "boxes/rationale":
+            return replace_match(shortcode, "**RATIONALE:** ")
+        case "boxes/warning":
+            return replace_match(shortcode, "**WARNING:** ")
+        case "added-in":
+            version = params['v']
+            if not version:
+                raise ValueError("Missing parameter `v` for `added-in` shortcode")
+
+            return replace_match(shortcode, f"**[Added in `v{version}`]** ")
+        case "changed-in":
+            version = params['v']
+            if not version:
+                raise ValueError("Missing parameter `v` for `changed-in` shortcode")
+
+            return replace_match(shortcode, f"**[Changed in `v{version}`]** ")
+        case _:
+            raise ValueError("Unknown shortcode", shortcode['name'])
+
+
+def find_and_replace_shortcodes(text):
+    """Finds Hugo shortcodes and replaces them by a Markdown fallback.
+
+    The supported shortcodes are:
+
+    * boxes/note, boxes/rationale, boxes/warning
+    * added-in, changed-in
+    """
+    # We use a `while` loop with `search` instead of a `for` loop with
+    # `finditer`, because as soon as we start replacing text, the
+    # indices of the match are invalid.
+    while shortcode := shortcode_regex.search(text):
+        text = replace_shortcode(shortcode)
+
+    return text
+
+def edit_descriptions(node, base_url):
+    """Finds description nodes and apply fixes to them.
+
+    The fixes that are applied are:
+
+    * Make links absolute
+    * Replace Hugo shortcodes
+    """
     if isinstance(node, dict):
         for key in node:
             if isinstance(node[key], str):
                 node[key] = prefix_absolute_path_references(node[key], base_url)
+                node[key] = find_and_replace_shortcodes(node[key])
             else:
-                edit_links(node[key], base_url)
+                edit_descriptions(node[key], base_url)
     elif isinstance(node, list):
         for item in node:
-            edit_links(item, base_url)
+            edit_descriptions(item, base_url)
+
 
 parser = argparse.ArgumentParser(
     "dump-openapi.py - assemble the OpenAPI specs into a single JSON file"
@@ -164,7 +266,7 @@ for filename in os.listdir(selected_api_dir):
 if untagged != 0:
     print("{} untagged operations, you may want to look into fixing that.".format(untagged))
 
-edit_links(output, base_url)
+edit_descriptions(output, base_url)
 
 print("Generating %s" % output_file)