From 8521c2d6963e878c60a91ffa5ab3d03c245e1cfa Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 13 Nov 2018 11:31:41 -0500 Subject: [PATCH 001/128] initial proposal for key verification methods --- proposals/xxxx-key_verification.md | 126 +++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 proposals/xxxx-key_verification.md diff --git a/proposals/xxxx-key_verification.md b/proposals/xxxx-key_verification.md new file mode 100644 index 00000000..b71fcf34 --- /dev/null +++ b/proposals/xxxx-key_verification.md @@ -0,0 +1,126 @@ +# Key verification mechanisms + +Key verification is an essential part of ensuring that end-to-end encrypted +messages are secure. Matrix may support multiple verification methods that +require sending events; in fact, two such methods have already been proposed. + +This proposal tries to present a common framework for verification methods to +use, and presents a way to request key verification. + +## Proposal + +Each key verification method is identified by a name. Verification method +names defined in the Matrix spec will begin with `m.`, and verification method +names that are not defined in the Matrix spec must be namespaced following the +Java package naming convention. + +If Alice wants to verify keys with Bob, Alice's device may send `to_device` +events to Bob's devices with the `type` set to `m.key.verification.request`, as +described below. The event lists the verification methods that Alice's device +supports. Upon receipt of this message, Bob's client should prompt him to +verify keys with Alice using one of the applicable methods. In order to avoid +displaying stale key verification prompts, if Bob does not interact with the +prompt, it should be automatically hidden 10 minutes after the message is sent +(according to the `timestamp` field), or 2 minutes after the client receives +the message, whichever comes first. The prompt should also be hidden if an +appropriate `m.key.verification.cancel` message is received. If Bob chooses to +reject the key verification request, Bob's client should send a +`m.key.verification.cancel` message to Alice's device. If Bob's client does +not understand any of the methods offered, it should display a message to Bob +saying so. + +To initiate a key verification process, Bob's device sends a `to_device` event +to one of Alice's devices with the `type` set to `m.key.verification.start`. +This may either be done in response to an `m.key.verification.request` message, +or can be done independently. If Alice's device receives an +`m.key.verification.start` message in response to an +`m.key.verification.request` message, it should send an +`m.key.verification.cancel` message to Bob's other devices that it had +originally sent an `m.key.verification.request` to, in order to cancel the key +verification request. + +Verification methods can define other events required to complete the +verification. Event types for verification methods defined in the Matrix spec +should be in the `m.key.verification` namespace. Event types that are not +defined in the Matrix spec must be namespaced following the Java package naming +convention. + +Alice's or Bob's devices can cancel a key verification process or a key +verification request by sending a `to_device` event with `type` set to +`m.key.verification.cancel`. + +### Event Definitions + +#### `m.key.verification.request` + +Requests a key verification. + +Properties: + +- `from_device` (string): the device ID of the device requesting verification. +- `transaction_id` (string): an identifier for the verification request. Must + be unique with respect to the pair of devices. +- `methods` ([string]): the verification methods supported by the sender. +- `timestamp` (integer): the time when the request was made. If the timestamp + is in the future (by more than 5 minutes, to allow for clock skew), or more + than 10 minutes in the past, then the message must be ignored. + +#### `m.key.verification.start` + +Begins a key verification process. + +Properties: + +- `method` (string): the verification method to use. +- `from_device` (string): The device ID of the device starting the verification + process. +- `transaction_id` (string): an identifier for the verification process. If + this message is sent in reponse to an `m.key.verification.request` event, then + it must use the same `transaction_id` as the one given in the + `m.key.verification.request`. + +Key verification methods can define additional properties to be included. + +#### `m.key.verification.cancel` + +Cancels a key verification process or a key verification request. Upon +receiving an `m.key.verification.cancel` message, the receiving device must +cancel the verification or the request. If it is a verification process that +is cancelled, or a verification request initiated by the recipient of the +cancellation message, the device should inform the user of the reason. + +Properties: + +- `transaction_id` (string): the identifier for the request or key verification + to cancel. +- `code` (string): machine-readable reason for cancelling. Possible reasons + are: + - `m.user`: the user cancelled the verification. + - `m.timeout`: the verification process has timed out. Different verification + methods may define their own timeouts. + - `m.unknown_transaction`: the device does not know about the given transaction + ID. + - `m.unknown_method`: the device does not know how to handle the given method. + This can be sent in response to an `m.key.verification.start` message, or + can be sent in response to other verification method-specific messages. + - `m.unexpected_message`: the device received an unexpected message. For + example, a message for a verification method may have been received when it + was not expected. +- `reason` (string): human-readable reason for cancelling. This should only be + used if the recieving client does not understand the code given. + +Verification methods may define their own additional cancellation codes. +Cancellation codes defined in the Matrix spec will begin with `m.`; other +cancellation codes must be namespaced following the Java package naming +convention. + +## Tradeoffs + +## Potential issues + +## Security considerations + +## Conclusion + +This proposal presents common event definitions for use by key verification +methods and defines a way for users to request key verification. From ec2e02e8b5479b4734b0d8a6ed5aa8953f495a67 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 13 Nov 2018 11:45:18 -0500 Subject: [PATCH 002/128] rename to match MSC number --- proposals/{xxxx-key_verification.md => 1717-key_verification.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{xxxx-key_verification.md => 1717-key_verification.md} (100%) diff --git a/proposals/xxxx-key_verification.md b/proposals/1717-key_verification.md similarity index 100% rename from proposals/xxxx-key_verification.md rename to proposals/1717-key_verification.md From b535226d407b28709c8d126ef6852f3bb9344c10 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 13 Nov 2018 22:00:42 -0500 Subject: [PATCH 003/128] initial work on documenting olm unwedging --- proposals/xxxx-olm_unwedging.md | 45 +++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 proposals/xxxx-olm_unwedging.md diff --git a/proposals/xxxx-olm_unwedging.md b/proposals/xxxx-olm_unwedging.md new file mode 100644 index 00000000..60f45fe1 --- /dev/null +++ b/proposals/xxxx-olm_unwedging.md @@ -0,0 +1,45 @@ +# Olm unwedging + +Olm sessions sometimes get out of sync, resulting in undecryptable messages. +This proposal documents a method for devices to create a new session to replace +the broken session. + +## Proposal + +When a device receives an olm-encrypted message that it cannot decrypt, it +should assume that the olm session has become corrupted and create a new olm +session to replace it. It should then send a dummy message, using that +session, to the other party in order to inform them of the new session. To +send a dummy message, clients may send an event with type `m.dummy`, and with +empty contents. + +If the corrupted session has already been replaced, the receiving device should +do nothing, under the assumption that the message from the corrupted session +was sent before the sender was informed of the replacement session, in order to +avoid creating too many extra sessions. + +The spec currently says, "If a client has multiple sessions established with +another device, it should use the session from which it last received a +message." (the last paragraph of the [`m.olm.v1.curve25519-aes-sha2` +section](https://matrix.org/docs/spec/client_server/r0.4.0.html#m-olm-v1-curve25519-aes-sha2)). +When comparing the time of the last received message for each session, the +client should consider only consider messages that were successfully decrypted, +and for sessions that have never received a message, it should use the creation +time of the session. The spec will be changed to read: + +> If a client has multiple sessions established with another device, it should +> use the session from which it last received and successfully decrypted a +> message. For these purposes, a session that has not received any messages +> should consider its creation time to be the time that it last received a +> message. + +## Tradeoffs + +## Potential issues + +## Security considerations + +## Conclusion + +This proposal outlines how wedged olm sessions can be replaced by a new +session. From d0bfdc13af8b4b8e5bba695c6f6bb3a2f6d7d275 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 13 Nov 2018 22:03:07 -0500 Subject: [PATCH 004/128] rename to match MSC number --- proposals/{xxxx-olm_unwedging.md => 1719-olm_unwedging.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{xxxx-olm_unwedging.md => 1719-olm_unwedging.md} (100%) diff --git a/proposals/xxxx-olm_unwedging.md b/proposals/1719-olm_unwedging.md similarity index 100% rename from proposals/xxxx-olm_unwedging.md rename to proposals/1719-olm_unwedging.md From 495df02da6efb5d821fad69abb4fb66600db20b4 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 14 Nov 2018 10:19:11 -0500 Subject: [PATCH 005/128] we don't actually know which session got stuck, so rate-limit by device --- proposals/1719-olm_unwedging.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index 60f45fe1..13c4ea4b 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -13,10 +13,11 @@ session, to the other party in order to inform them of the new session. To send a dummy message, clients may send an event with type `m.dummy`, and with empty contents. -If the corrupted session has already been replaced, the receiving device should -do nothing, under the assumption that the message from the corrupted session -was sent before the sender was informed of the replacement session, in order to -avoid creating too many extra sessions. +In order to avoid creating too many extra sessions, a client should rate-limit +the number of new sessions it creates per device that it receives a message +from; the client should not create a new session with another device if it has +already created one for that given device in the past 1 hour. (TODO: is 1 hour +the right amount of time?) The spec currently says, "If a client has multiple sessions established with another device, it should use the session from which it last received a From 2b5805255564bb80a63c73dd977b487c67c9007c Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 14 Nov 2018 10:28:26 -0500 Subject: [PATCH 006/128] add note about re-requesting megolm keys --- proposals/1719-olm_unwedging.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index 13c4ea4b..e696c1e2 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -19,6 +19,12 @@ from; the client should not create a new session with another device if it has already created one for that given device in the past 1 hour. (TODO: is 1 hour the right amount of time?) +Clients may wish to ask the sender of the undecryptable messages to re-send the +message. For exampe, if the undecryptable message was a megolm session, then +the client can send an +[`m.room_key_request`](https://matrix.org/docs/spec/client_server/r0.4.0.html#m-room-key-request) +message to request that the sender re-send the key. + The spec currently says, "If a client has multiple sessions established with another device, it should use the session from which it last received a message." (the last paragraph of the [`m.olm.v1.curve25519-aes-sha2` From c02ecb58aec48778d9ca4531bb84f5e875d71210 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 8 Jan 2019 16:11:00 -0500 Subject: [PATCH 007/128] mark which fields are required --- proposals/1717-key_verification.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index b71fcf34..fa1d7373 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -57,13 +57,15 @@ Requests a key verification. Properties: -- `from_device` (string): the device ID of the device requesting verification. -- `transaction_id` (string): an identifier for the verification request. Must - be unique with respect to the pair of devices. -- `methods` ([string]): the verification methods supported by the sender. -- `timestamp` (integer): the time when the request was made. If the timestamp - is in the future (by more than 5 minutes, to allow for clock skew), or more - than 10 minutes in the past, then the message must be ignored. +- `from_device` (string): Required. The device ID of the device requesting + verification. +- `transaction_id` (string): Required. An identifier for the verification + request. Must be unique with respect to the pair of devices. +- `methods` ([string]): Required. The verification methods supported by the + sender. +- `timestamp` (integer): Required. The time when the request was made. If the + timestamp is in the future (by more than 5 minutes, to allow for clock skew), + or more than 10 minutes in the past, then the message must be ignored. #### `m.key.verification.start` @@ -71,13 +73,13 @@ Begins a key verification process. Properties: -- `method` (string): the verification method to use. -- `from_device` (string): The device ID of the device starting the verification - process. -- `transaction_id` (string): an identifier for the verification process. If - this message is sent in reponse to an `m.key.verification.request` event, then - it must use the same `transaction_id` as the one given in the - `m.key.verification.request`. +- `method` (string): Required. The verification method to use. +- `from_device` (string): Required. The device ID of the device starting the + verification process. +- `transaction_id` (string): Required. An identifier for the verification + process. If this message is sent in reponse to an + `m.key.verification.request` event, then it must use the same + `transaction_id` as the one given in the `m.key.verification.request`. Key verification methods can define additional properties to be included. From ff0b9eac76cfa65222dd82d45ff3358570ab1fd1 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 8 Jan 2019 16:11:15 -0500 Subject: [PATCH 008/128] add ability to start verifications that happen in two stages --- proposals/1717-key_verification.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index fa1d7373..8acc4654 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -80,6 +80,10 @@ Properties: process. If this message is sent in reponse to an `m.key.verification.request` event, then it must use the same `transaction_id` as the one given in the `m.key.verification.request`. +- `next_method` (string): Optional. If the selected verification method only + verifies one user's key, then this property can be used to indicate the + method to use to verify the other user's key, which will be started + immediately after after the current key verification is complete. Key verification methods can define additional properties to be included. From 09a547d67e0a4b835405af0be9ce739f9dfe400a Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 8 Jan 2019 17:45:13 -0500 Subject: [PATCH 009/128] add some cancellation codes, and mention existing verification MSCs --- proposals/1717-key_verification.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index 8acc4654..b536be0e 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -2,7 +2,8 @@ Key verification is an essential part of ensuring that end-to-end encrypted messages are secure. Matrix may support multiple verification methods that -require sending events; in fact, two such methods have already been proposed. +require sending events; in fact, two such methods (such as MSC1267 and MSC1543) +have already been proposed. This proposal tries to present a common framework for verification methods to use, and presents a way to request key verification. @@ -112,6 +113,12 @@ Properties: - `m.unexpected_message`: the device received an unexpected message. For example, a message for a verification method may have been received when it was not expected. + - `m.key_mismatch`: the key was not verified. + - `m.user_mismatch`: the expected user did not match the user verified. + - `m.invalid_message`: an invalid message was received. + - `m.accepted`: when an `m.key.verification.request` is accepted by one + device, an `m.key.verification.cancel` message with `code` set to + `m.accepted` is sent to the other devices - `reason` (string): human-readable reason for cancelling. This should only be used if the recieving client does not understand the code given. From 82258fc0fc90c8f2cc5d96967229a029335b1b7e Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 13 Feb 2019 23:11:14 +0000 Subject: [PATCH 010/128] Proposal for changing event ids. Again. --- .../1884-replace-slashes-in-event_ids.md | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 proposals/1884-replace-slashes-in-event_ids.md diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md new file mode 100644 index 00000000..5d9d0bc2 --- /dev/null +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -0,0 +1,91 @@ +# MSC1884: Proposal to replace slashes in event IDs + +[MSC1659](https://github.com/matrix-org/matrix-doc/pull/1659) mandated that, +starting in version 3 rooms, event IDs must be calculated as a base64-encoding +of a hash. This implies that event IDs may contain any character in the +standard Base64 alphabet, which notably includes the slash character, `/`. + +Event IDs are often embedded in URI paths, and since the slash character is +used as a separator in URI paths, this presents a problem. The immediate +solution is to ensure that event IDs are URL-encoded, so that `/` is instead +represented as `%2F`. However, this is not entirely satisfactory for a number +of reasons: + + * There exist a number of client (and possibly server) implementations which + do not currently URL-encode such parameters; these are therefore broken by + such event IDs and must be updated. Furthermore, all future client + implementers must remember to do the encoding correctly. + + * Even if client implementations do rembember to URL-encode their parameters, + they may not do it correctly: many URL-encoding implementations may be + intended to encode parameters in the query-string (which can of course + contain literal slashes) rather tha the path component. + + * Some proxy software may treat `%2F` specially: for instance, Apache, when + configured as a reverse-proxy, will reject requests for a path containing + `%2F` unless it is also configured with `nocanon`. Again this means that + existing setups will be broken by this change, and it is a trap for new + users of the software. + +## Proposal + +This MSC proposes that we should introduce a new room version, in which event +IDs are encoded using the [URL-safe Base64 +encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and +`_` as the 62nd and 63rd characters instead of `+` and `/`). + +## Counterarguments + +1. Inconsistency. Base64 encoding is used heavily elsewhere in the matrix + protocol and in all cases the standard encoding is used (though with some + variation as to the inclusion of padding characters). Further, SHA256 hashes + are used in a number of places and are universally included with standard, + unpadded Base64. + + Changing event IDs alone would therefore leave us with a confusing mix of + encodings. + + A potential extension would be to change *all* Base64 encodings to be + URL-safe. This would address the inconsistency. However, it feels like a + large job which would span the entire matrix ecosystem (far larger than + updating clients to URL-encode their URL prarameters), and again the + situation would be confusing while the transition was in progress. + +2. Incompleteness. Event IDs are certainly not the only identifier which can + contain slashes - Room aliases, Room IDs, Group IDs, User IDs [1], and state + keys can all contain slashes, as well as a number of identifiers whose + grammars are currently underspecified (eg transaction ids, event types, + device IDs). (Indeed, there was nothing preventing Event IDs from containing + slashes before room v3 - it just happened that Synapse used an algorithm + which didn't generate them). + + All of these other identifiers can appear in URLs in either or both the + client-server or server-server APIs, and all have the potential to cause + misbehaviour if software does not correctly URL-encode them. + + It can be argued that it is better for software to fail 50% of the time [2] + so that it can be fixed than it is to fail only on edge-cases or, worse, + when deliberately provoked by a malicious or "curious" actor. + + Of course, an alternative is to modify the grammars of all of these + identifiers to forbid slashes. + +[1] Discussion remains open as to whether allowing slashes in User IDs was a +good idea. + +[2] 48% of random 32-byte sequences will contain a slash when Base64-encoded. + +## Alternatives + +An alternative would be to modify all REST endpoints to use query or body +parameters instead of path parameters. This would of course be a significant +and incompatible change, but it would also bring the benefit of solving a +common problem where forgetting to use `nocanon` in a reverse-proxy +configuration [breaks +federation](https://github.com/matrix-org/synapse/issues/3294) (though other +solutions to that are also possible). + +## Conclusion + +It's unclear to me that changing the format of event IDs alone solves any +problems. From f9a00fc943d9ae49ae32845a7a971578ff84c4a0 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 13 Mar 2019 23:25:11 -0400 Subject: [PATCH 011/128] clarify that not understanding a verification method should not auto-cancel --- proposals/1717-key_verification.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index b536be0e..10ccf785 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -28,7 +28,10 @@ appropriate `m.key.verification.cancel` message is received. If Bob chooses to reject the key verification request, Bob's client should send a `m.key.verification.cancel` message to Alice's device. If Bob's client does not understand any of the methods offered, it should display a message to Bob -saying so. +saying so. However, it should not send a `m.key.verification.cancel` message +to Alice's device unless Bob chooses to reject the verification request, as Bob +may have another device that is capable of verifying using one of the given +methods. To initiate a key verification process, Bob's device sends a `to_device` event to one of Alice's devices with the `type` set to `m.key.verification.start`. From 4842a718d8ab3c4000e86270d3ef3aea6cf42154 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 13 Mar 2019 23:39:00 -0400 Subject: [PATCH 012/128] fill in some of the other sections --- proposals/1717-key_verification.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index 10ccf785..9296cf40 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -132,10 +132,18 @@ convention. ## Tradeoffs -## Potential issues +Rather than broadcasting verification requests to Bob's devices, Alice could +simply send an `m.key.verification.start` request to a single device. However, +this would require Alice to choose the right device to send to, which may be +hard for Alice to do if, for example, Bob has many devices, or if his devices +have similar names. ## Security considerations +An attacker could try to spam a user with verification requests. Clients +should take care that such requests do not interfere with a user's use of the +client. + ## Conclusion This proposal presents common event definitions for use by key verification From 1749a91344786815ecbae43a53e19fb53157c307 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 13 Mar 2019 23:41:43 -0400 Subject: [PATCH 013/128] make MSC mentions into links --- proposals/1717-key_verification.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index 9296cf40..02018ef9 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -2,8 +2,10 @@ Key verification is an essential part of ensuring that end-to-end encrypted messages are secure. Matrix may support multiple verification methods that -require sending events; in fact, two such methods (such as MSC1267 and MSC1543) -have already been proposed. +require sending events; in fact, two such methods (such as [MSC +1267](https://github.com/matrix-org/matrix-doc/issues/1267) and [MSC +1543](https://github.com/matrix-org/matrix-doc/issues/1543)) have already been +proposed. This proposal tries to present a common framework for verification methods to use, and presents a way to request key verification. From 0e07a6d243f2fc11c1ce95504e842d2f96f24c00 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 Mar 2019 13:32:38 -0600 Subject: [PATCH 014/128] Proposal to have a push rule for m.room.tombstone events --- proposals/1930-tombstone-notifications.md | 40 +++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 proposals/1930-tombstone-notifications.md diff --git a/proposals/1930-tombstone-notifications.md b/proposals/1930-tombstone-notifications.md new file mode 100644 index 00000000..715921aa --- /dev/null +++ b/proposals/1930-tombstone-notifications.md @@ -0,0 +1,40 @@ +# Proposal to add a default push rule for m.room.tombstone events + +Currently users are unaware of when a room becomes upgraded, leaving them potentially in the old room +without knowing until they visit the room again. By having a notification for when the room is upgraded, +users are able to ensure they are able to stay relevant in rooms by joining the upgraded room. + + +## Proposal + +A new default override rule is to be added which is similar to `@room` notifications: + +```json +{ + "rule_id": ".m.rule.tombstone", + "default": true, + "enabled": true, + "conditions": [ + { + "kind": "event_match", + "key": "type", + "pattern": "m.room.tombstone" + } + ], + "actions": [ + "notify", + { + "set_tweak": "highlight", + "value": true + } + ] +} +``` + + +## Tradeoffs + +Clients could calculate this on their own and show some sort of "room upgraded" notification instead, +however by doing it this way it means that all clients would need to be aware of room upgrades. Having +a default push rule means that clients get this notification for free. Clients which want a more diverse +UX can still do so by ignoring this push rule locally. From 696e568fb2101480826d873fac83897ae904472a Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 26 Mar 2019 15:10:14 -0400 Subject: [PATCH 015/128] add some clarifications --- proposals/1717-key_verification.md | 46 +++++++++++++++++++----------- 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index 02018ef9..6c9e2341 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -19,26 +19,40 @@ Java package naming convention. If Alice wants to verify keys with Bob, Alice's device may send `to_device` events to Bob's devices with the `type` set to `m.key.verification.request`, as -described below. The event lists the verification methods that Alice's device -supports. Upon receipt of this message, Bob's client should prompt him to -verify keys with Alice using one of the applicable methods. In order to avoid -displaying stale key verification prompts, if Bob does not interact with the -prompt, it should be automatically hidden 10 minutes after the message is sent -(according to the `timestamp` field), or 2 minutes after the client receives -the message, whichever comes first. The prompt should also be hidden if an -appropriate `m.key.verification.cancel` message is received. If Bob chooses to -reject the key verification request, Bob's client should send a -`m.key.verification.cancel` message to Alice's device. If Bob's client does -not understand any of the methods offered, it should display a message to Bob -saying so. However, it should not send a `m.key.verification.cancel` message -to Alice's device unless Bob chooses to reject the verification request, as Bob -may have another device that is capable of verifying using one of the given -methods. +described below. The `m.key.verification.request` messages should all have the +same `transaction_id`, and are considered to be a single request. Thus, for +example, if Bob rejects the request on one device, then the entire request +should be considered as rejected across all of his devices. Similarly, if Bob +accepts the request on one device, that device is now in charge of completing +the key verification, and Bob's other devices no longer need to be involved. + +The `m.key.verification.request` event lists the verification methods that +Alice's device supports, and upon receipt of this message, Bob's client should +prompt him to verify keys with Alice using one of the applicable methods. In +order to avoid displaying stale key verification prompts, if Bob does not +interact with the prompt, it should be automatically hidden 10 minutes after +the message is sent (according to the `timestamp` field), or 2 minutes after +the client receives the message, whichever comes first. The prompt should also +be hidden if an appropriate `m.key.verification.cancel` message is received. + +If Bob chooses to reject the key verification request, Bob's client should send +a `m.key.verification.cancel` message to Alice's device. This indicates to +Alice that Bob does not wish to verify keys with her. In this case, Alice's +device should send an `m.key.verification.cancel` message to all of Bob's +devices to notify them that the request has been rejected. + +If one of Bob's clients does not understand any of the methods offered, it +should display a message to Bob saying so. However, it should not send a +`m.key.verification.cancel` message to Alice's device unless Bob chooses to +reject the verification request, as Bob may have another device that is capable +of verifying using one of the given methods. To initiate a key verification process, Bob's device sends a `to_device` event to one of Alice's devices with the `type` set to `m.key.verification.start`. This may either be done in response to an `m.key.verification.request` message, -or can be done independently. If Alice's device receives an +or can be done independently. If it is done in response to an +`m.key.verification.request` messsage, it should use the same `transaction_id` +as the `m.key.verification.request` message. If Alice's device receives an `m.key.verification.start` message in response to an `m.key.verification.request` message, it should send an `m.key.verification.cancel` message to Bob's other devices that it had From 207d6cf8512231eb121d9696369b1d1d01449673 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sat, 30 Mar 2019 13:12:18 +0000 Subject: [PATCH 016/128] update MSC1884 to reflect new conclusions following discussion on the PR --- .../1884-replace-slashes-in-event_ids.md | 80 ++++++++++++++++++- 1 file changed, 77 insertions(+), 3 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index 5d9d0bc2..9c3b7ea7 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -11,12 +11,20 @@ solution is to ensure that event IDs are URL-encoded, so that `/` is instead represented as `%2F`. However, this is not entirely satisfactory for a number of reasons: + * The act of escaping and unescaping slash characters when doing casual + development and ops work becomes an constant and annoying chore which + is entirely avoidable. Whenever using tools like `curl` and `grep` or + manipulating SQL, developers will have to constantly keep in mind whether + they are dealing with escaped or unescaped IDs, and manually convert between + the two as needed. This will only get worse with further keys-as-IDs + landing with MSC1228. + * There exist a number of client (and possibly server) implementations which do not currently URL-encode such parameters; these are therefore broken by such event IDs and must be updated. Furthermore, all future client implementers must remember to do the encoding correctly. - * Even if client implementations do rembember to URL-encode their parameters, + * Even if client implementations do remember to URL-encode their parameters, they may not do it correctly: many URL-encoding implementations may be intended to encode parameters in the query-string (which can of course contain literal slashes) rather tha the path component. @@ -27,6 +35,14 @@ of reasons: existing setups will be broken by this change, and it is a trap for new users of the software. + * Cosmetically, URL-escaping base64 in otherwise-constant-length IDs results + in variable length IDs, making it harder to visually scan lists of IDs and + manipulate them in columnar form when doing devops work. + + * Those developing against the CS API might reasonably expect us to use + URL-safe identifiers in URLs where available, rather than deliberately + choosing non-URL-safe IDs, which could be seen as developer-unfriendly. + ## Proposal This MSC proposes that we should introduce a new room version, in which event @@ -34,6 +50,22 @@ IDs are encoded using the [URL-safe Base64 encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and `_` as the 62nd and 63rd characters instead of `+` and `/`). +URL-safe Base64 encoding is then used consistently for encoding binary +identifiers in the CS API - particularly in upcoming MSC1228 IDs for rooms and +users, such that typical CS API developers should be able to safely assume +that for all common cases they should use URL-safe Base64 when decoding base64 +strings. + +The exception would be for E2EE data (device keys and signatures etc) which +currently use normal Base64 with no easy mechanism to migrate to a new encoding. +Given E2EE development is rare and requires expert skills, it seems acceptable +to expect E2EE developers to be able to use the right encoding without tripping +up significantly. + +Similarly, the S2S API could continue to use standard base64-encoded hashes and +signatures, given they are only exposed to S2S API developers who are necessarily +expert and should be able to correctly pick the right encoding. + ## Counterarguments 1. Inconsistency. Base64 encoding is used heavily elsewhere in the matrix @@ -45,6 +77,14 @@ encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and Changing event IDs alone would therefore leave us with a confusing mix of encodings. + However, the current uses of standard Base64 encodings are not exposed to + common CS API developers, and so whilst this might be slightly confusing + for the minority of expert homeserver developers, the confusion does not + exist today for client developers. Therefore it seems safe to standardise + on URL-safe Base64 for identifiers exposed to the client developers, who + form by far the majority of the Matrix ecosystem today, and expect as + simple an API as possible. + A potential extension would be to change *all* Base64 encodings to be URL-safe. This would address the inconsistency. However, it feels like a large job which would span the entire matrix ecosystem (far larger than @@ -70,6 +110,16 @@ encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and Of course, an alternative is to modify the grammars of all of these identifiers to forbid slashes. + The counter-counterargument to this is that it is of course best practice + for implementations is to URL-escape any IDs used in URLs, and human-selected + IDs such as Room aliases, Group IDs, Matrix user IDs etc apply an adequate + forcing function already to remind developers to do this. However, + it doesn't follow that we should then also deliberately pick URL-unsafe + encodings for machine-selected IDs - the argument that it is better for software + to fail 50% of the time to force a fix is irrelevant when the possibility + exists for the software to fail 0% of the time in the first place by picking + an identifier format which cannot fail. + [1] Discussion remains open as to whether allowing slashes in User IDs was a good idea. @@ -87,5 +137,29 @@ solutions to that are also possible). ## Conclusion -It's unclear to me that changing the format of event IDs alone solves any -problems. +There are two main questions here: + + 1. Whether it's worth forcing casual CS API developers to juggle escaping of + machine-selected IDs in order to remind them to escape all variables in + their URIs correctly. + + 2. Whether it's a significant problem for E2EE & SS API developers to have to + handle strings which are a mix of standard Base64 and URL-safe Base64 + encodings. + +Both of these are a subjective judgement call. + +Given we wish the CS API particularly to be as easy for casual developers to +use as possible, it feels that we should find another way to encourage +developers to escape variables in their URLs in general - e.g. by recommending +that developers test their clients against a 'torture room' full of exotic IDs +and data, or by improving warnings in the spec... rather than (ab)using +machine-selected IDs as a reminder. + +Meanwhile, given we have many more CS API developers than SS or E2EE developers, +and we wish to make the CS API particularly easy for casual developers to use, +it feels we should not prioritise consistency of encodings for SS/E2EE developers +over the usability of the CS API. + +Therefore, on balance, it seems plausible that changing the format of event IDs +does solve sufficient problems to make it desirable. \ No newline at end of file From 9dcf2d6a2805e4c81b5c04af71034a10cb68282a Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 1 Apr 2019 00:43:31 +0100 Subject: [PATCH 017/128] Update proposals/1884-replace-slashes-in-event_ids.md Co-Authored-By: ara4n --- proposals/1884-replace-slashes-in-event_ids.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index 9c3b7ea7..ae2c0e94 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -27,7 +27,7 @@ of reasons: * Even if client implementations do remember to URL-encode their parameters, they may not do it correctly: many URL-encoding implementations may be intended to encode parameters in the query-string (which can of course - contain literal slashes) rather tha the path component. + contain literal slashes) rather than the path component. * Some proxy software may treat `%2F` specially: for instance, Apache, when configured as a reverse-proxy, will reject requests for a path containing @@ -162,4 +162,4 @@ it feels we should not prioritise consistency of encodings for SS/E2EE developer over the usability of the CS API. Therefore, on balance, it seems plausible that changing the format of event IDs -does solve sufficient problems to make it desirable. \ No newline at end of file +does solve sufficient problems to make it desirable. From 417f3a3e8b1b0510513306139ba1fd05f39e295e Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 1 Apr 2019 00:58:10 +0100 Subject: [PATCH 018/128] incorporate further feedback from vdh --- .../1884-replace-slashes-in-event_ids.md | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index ae2c0e94..9f1a2ad0 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -11,8 +11,8 @@ solution is to ensure that event IDs are URL-encoded, so that `/` is instead represented as `%2F`. However, this is not entirely satisfactory for a number of reasons: - * The act of escaping and unescaping slash characters when doing casual - development and ops work becomes an constant and annoying chore which + * The act of escaping and unescaping slash characters when manually calling + the API during devops work becomes an constant and annoying chore which is entirely avoidable. Whenever using tools like `curl` and `grep` or manipulating SQL, developers will have to constantly keep in mind whether they are dealing with escaped or unescaped IDs, and manually convert between @@ -50,11 +50,10 @@ IDs are encoded using the [URL-safe Base64 encoding](https://tools.ietf.org/html/rfc4648#section-5) (which uses `-` and `_` as the 62nd and 63rd characters instead of `+` and `/`). -URL-safe Base64 encoding is then used consistently for encoding binary -identifiers in the CS API - particularly in upcoming MSC1228 IDs for rooms and -users, such that typical CS API developers should be able to safely assume -that for all common cases they should use URL-safe Base64 when decoding base64 -strings. +We will then aim to use URL-safe Base64 encoding across Matrix in future, +such that typical CS API developers should be able to safely assume +that for all common cases (including upcoming MSC1228 identifiers) they should +use URL-safe Base64 when decoding base64 strings. The exception would be for E2EE data (device keys and signatures etc) which currently use normal Base64 with no easy mechanism to migrate to a new encoding. @@ -63,8 +62,9 @@ to expect E2EE developers to be able to use the right encoding without tripping up significantly. Similarly, the S2S API could continue to use standard base64-encoded hashes and -signatures, given they are only exposed to S2S API developers who are necessarily -expert and should be able to correctly pick the right encoding. +signatures in the places it does today, given they are only exposed to S2S API +developers who are necessarily expert and should be able to correctly pick the +right encoding. ## Counterarguments @@ -81,9 +81,9 @@ expert and should be able to correctly pick the right encoding. common CS API developers, and so whilst this might be slightly confusing for the minority of expert homeserver developers, the confusion does not exist today for client developers. Therefore it seems safe to standardise - on URL-safe Base64 for identifiers exposed to the client developers, who - form by far the majority of the Matrix ecosystem today, and expect as - simple an API as possible. + (except those implementing E2EE) on URL-safe Base64 for identifiers exposed + to the client developers, who form by far the majority of the Matrix + ecosystem today, and expect as simple an API as possible. A potential extension would be to change *all* Base64 encodings to be URL-safe. This would address the inconsistency. However, it feels like a @@ -139,7 +139,7 @@ solutions to that are also possible). There are two main questions here: - 1. Whether it's worth forcing casual CS API developers to juggle escaping of + 1. Whether it's worth forcing manual CS API users to juggle escaping of machine-selected IDs in order to remind them to escape all variables in their URIs correctly. @@ -149,17 +149,17 @@ There are two main questions here: Both of these are a subjective judgement call. -Given we wish the CS API particularly to be as easy for casual developers to -use as possible, it feels that we should find another way to encourage -developers to escape variables in their URLs in general - e.g. by recommending -that developers test their clients against a 'torture room' full of exotic IDs -and data, or by improving warnings in the spec... rather than (ab)using +Given we wish the CS API particularly to be as easy as possible for manual +use, it feels that we should find another way to encourage developers to +escape variables in their URLs in general - e.g. by recommending that +developers test their clients against a 'torture room' full of exotic IDs and +data, or by improving warnings in the spec... rather than (ab)using machine-selected IDs as a reminder. -Meanwhile, given we have many more CS API developers than SS or E2EE developers, -and we wish to make the CS API particularly easy for casual developers to use, -it feels we should not prioritise consistency of encodings for SS/E2EE developers -over the usability of the CS API. +Meanwhile, given we have many more CS API developers than SS or E2EE +developers, and we wish to make the CS API particularly easy for developers to +manually invoke, it feels we should not prioritise consistency of encodings +for SS/E2EE developers over the usability of the CS API. Therefore, on balance, it seems plausible that changing the format of event IDs does solve sufficient problems to make it desirable. From 88f533f0db33518e48c489872d8cbc68a204ab2c Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 1 Apr 2019 10:27:30 +0100 Subject: [PATCH 019/128] incorporate further feedback --- .../1884-replace-slashes-in-event_ids.md | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/proposals/1884-replace-slashes-in-event_ids.md b/proposals/1884-replace-slashes-in-event_ids.md index 9f1a2ad0..bec8d7ad 100644 --- a/proposals/1884-replace-slashes-in-event_ids.md +++ b/proposals/1884-replace-slashes-in-event_ids.md @@ -80,10 +80,10 @@ right encoding. However, the current uses of standard Base64 encodings are not exposed to common CS API developers, and so whilst this might be slightly confusing for the minority of expert homeserver developers, the confusion does not - exist today for client developers. Therefore it seems safe to standardise - (except those implementing E2EE) on URL-safe Base64 for identifiers exposed - to the client developers, who form by far the majority of the Matrix - ecosystem today, and expect as simple an API as possible. + exist today for client developers (except those implementing E2EE). + Therefore it seems safe to standardise on URL-safe Base64 for identifiers + exposed to the client developers, who form by far the majority of the + Matrix ecosystem today, and expect as simple an API as possible. A potential extension would be to change *all* Base64 encodings to be URL-safe. This would address the inconsistency. However, it feels like a @@ -139,9 +139,9 @@ solutions to that are also possible). There are two main questions here: - 1. Whether it's worth forcing manual CS API users to juggle escaping of - machine-selected IDs in order to remind them to escape all variables in - their URIs correctly. + 1. Whether it's worth forcing CS API developers to juggle escaping of + machine-selected IDs during manual use of the API in order to remind them + to escape all variables in their URIs correctly when writing code. 2. Whether it's a significant problem for E2EE & SS API developers to have to handle strings which are a mix of standard Base64 and URL-safe Base64 @@ -156,10 +156,11 @@ developers test their clients against a 'torture room' full of exotic IDs and data, or by improving warnings in the spec... rather than (ab)using machine-selected IDs as a reminder. -Meanwhile, given we have many more CS API developers than SS or E2EE -developers, and we wish to make the CS API particularly easy for developers to -manually invoke, it feels we should not prioritise consistency of encodings -for SS/E2EE developers over the usability of the CS API. +Meanwhile, given we have many more people manually invoking the CS API than +developing on the SS or E2EE APIs, and we wish to make the CS API particularly +easy for developers to manually invoke, it feels we should not prioritise +consistency of encodings for SS/E2EE developers over the usability of the CS +API. Therefore, on balance, it seems plausible that changing the format of event IDs does solve sufficient problems to make it desirable. From 743eeca27a821b0939780d4fd3713c3f514b3660 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Fri, 5 Apr 2019 15:19:22 +0100 Subject: [PATCH 020/128] MSC to remove prev_content from the essential keys list --- ...ove-prev_event-from-essential-keys-list.md | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 proposals/1954-remove-prev_event-from-essential-keys-list.md diff --git a/proposals/1954-remove-prev_event-from-essential-keys-list.md b/proposals/1954-remove-prev_event-from-essential-keys-list.md new file mode 100644 index 00000000..992b3e80 --- /dev/null +++ b/proposals/1954-remove-prev_event-from-essential-keys-list.md @@ -0,0 +1,64 @@ +# Remove prev_content from the essential keys list + +Matrix supports the concept of event redaction. The ability to redact rather +than delete is necessary because some events e.g. membership events are +essential to the protocol and _cannot_ be deleted. Therefore we do not delete +events outright and instead redact them. This involves removing all keys from +an event that are not required by the protocol. The stripped down event is +thereafter returned anytime a client or remote server requests it. + + +## Proposal + +[The redaction algorithm](https://matrix.org/docs/spec/client_server/r0.4.0.html#redactions) +defines which keys must be retained through a redaction. Currently it lists +```prev_content``` as a key to retain, though in practice there is no need to +do so at the protocol level. + +The proposal is simply to remove ```prev_content``` from the essential keys +list. + +Note: the inclusion of ```prev_content``` in the essential keys list was +unintentional and should be considered a spec bug. Synapse (and other server +implementations) have not implemented the bug and already omit +```prev_content``` from redacted events. + + +## Tradeoffs + +When sending events over federation the events are [hashed and +signed](https://matrix.org/docs/spec/server_server/unstable.html#adding-hashes-and-signatures-to-outgoing-events), +this involves operating not only on the original event but also the redacted +form of the event. The redacted hash and redacted signed event are necessary if +the event is ever redacted in future. As a result, any change of the essential +keys list must be managed carefully. If disparate servers implement different +versions of the redaction algorithm (for a given event) attempts to send the +event over federation will fail. + +We _could_ manage this change via room versioning and create a new room +version that implements this MSC. However, because the federation already +omits the ```prev_content``` key by convention, implementing this MSC only in +the new room version would mean that the entire existing federation would not +be spec compliant. + +As a result it seems pragmatic to have the spec reflect reality, acknowledge +that the spec and federation have deviated and instead update the spec +retrospectively to describe the de-facto redaction algorithm. + +## Potential issues + +It is theoretically possible that a closed federation could exist whose servers +do follow the spec as is. This MSC would render those servers uncompliant with +the spec. On balance this seems unlikely and in the worst case those +implementors could add the change to a subsequent room version, eventually +reaching spec consistency as older room versions are deprecated. + +## Security considerations + +I am unaware of any security issues related to this proposal, but can certainly +see issues with a precedent that the federation deviates from the spec. + +## Conclusions +Removing ```prev_content``` is pragmatic response to the current situation. It +alligns the federation and the spec, and does so in a way that removes +unecessary overhead. From 8ad82ce3c0783cacb29f0730546f738acaacd92a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 5 Apr 2019 16:24:51 -0600 Subject: [PATCH 021/128] Add permalink routing through ?via parameters on matrix.to URIs Spec for [MSC1704](https://github.com/matrix-org/matrix-doc/pull/1704) Reference implementations: * Original: https://github.com/matrix-org/matrix-react-sdk/pull/2250 * Modern recommendations: https://github.com/matrix-org/matrix-react-sdk/blob/2ca281f6b7b735bc96945370584c5cb1b5f7e1f1/src/matrix-to.js#L29-L70 The only deviation from the original MSC is the recommendation for which servers to pick. The original MSC failed to consider server ACLs and IP addresses correctly, and during implementation it was realized that both of these cases should be handled. The core principles of the original MSC are left unaltered. --- .../appendices/identifier_grammar.rst | 61 +++++++++++++++++-- 1 file changed, 55 insertions(+), 6 deletions(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index 729246d0..ea805955 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -311,13 +311,16 @@ in the room's history (a permalink). A matrix.to URI has the following format, based upon the specification defined in RFC 3986: - https://matrix.to/#// + https://matrix.to/#//? The identifier may be a room ID, room alias, user ID, or group ID. The extra parameter is only used in the case of permalinks where an event ID is referenced. The matrix.to URI, when referenced, must always start with ``https://matrix.to/#/`` followed by the identifier. +The ```` and the preceeding question mark are optional and +only apply in certain circumstances, documented below. + Clients should not rely on matrix.to URIs falling back to a web server if accessed and instead should perform some sort of action within the client. For example, if the user were to click on a matrix.to URI for a room alias, the client may open @@ -331,7 +334,7 @@ Examples of matrix.to URIs are: * Room alias: ``https://matrix.to/#/%23somewhere%3Aexample.org`` * Room: ``https://matrix.to/#/!somewhere%3Aexample.org`` * Permalink by room: ``https://matrix.to/#/!somewhere%3Aexample.org/%24event%3Aexample.org`` -* Permalink by room alias: ``https://matrix.to/#/#somewhere:example.org/%24event%3Aexample.org`` +* Permalink by room alias: ``https://matrix.to/#/%23somewhere:example.org/%24event%3Aexample.org`` * User: ``https://matrix.to/#/%40alice%3Aexample.org`` * Group: ``https://matrix.to/#/%2Bexample%3Aexample.org`` @@ -345,7 +348,53 @@ Examples of matrix.to URIs are: appearing due to some `room versions `_. These slashes should normally be encoded when producing matrix.to URIs, however. -.. Note:: - Room ID permalinks are unroutable as there is no reliable domain to send requests - to upon receipt of the permalink. Clients should do their best route Room IDs to - where they need to go, however they should also be aware of `issue #1579 `_. +Routing +<<<<<<< + +Room IDs are not routable on their own as there is no reliable domain to send requests +to. This is partially mitigated with the addition of a ``via`` argument on a matrix.to +URI, however the problem of routability is still present. Clients should do their best +to route Room IDs to where they need to go, however they should also be aware of +`issue #1579 `_. + +A room (or room permalink) which isn't using a room alias should supply at least one +server using ``via`` in the ````, like so: +``https://matrix.to/!somewhere%3Aexample.org?via=example.org&via=alt.example.org``. The +parameter can be supplied multiple times to specify multiple servers to try. + +The values of ``via`` are intended to be passed along as the ``server_name`` parameters +on the Client Server ``/join`` API. + +When generating room links and permalinks, the application should pick servers which +have a high probability of being in the room in the distant future. How these servers +are picked is left as an implementation detail, however the current recommendation is +to pick 3 unique servers based on the following criteria: + +* The first server should be the server of the highest power level user in the room, + provided they are at least power level 50. If no user meets this criteria, pick the + most popular server in the room (most joined users). The rationale for not picking + users with power levels under 50 is that they are unlikely to be around into the + distant future while higher ranking users (and therefore servers) are less likely + to give up their power and move somewhere else. Most rooms in the public federation + have a power level 100 user and have not deviated from the default structure where + power level 50 users have moderator-style privileges. + +* The second server should be the next highest server by population, or the first + highest by population if the first server was based on a user's power level. The + rationale for picking popular servers is that the server is unlikely to be removed + as the room naturally grows in membership due to that server joining users. The + server could be refused participation in the future due to server ACLs or similar, + however the chance of that happening to a server which is organically joining the + room is unlikely. + +* The third server should be the next highest server by population. + +* Servers which are blocked due to server ACLs should never be chosen. + +* Servers which are IP addresses should never be chosen. Servers which use a domain + name are less likely to be unroutable in the future whereas IP addresses cannot be + pointed to a different location and therefore higher risk options. + +* All 3 servers should be unique from each other. If the room does not have enough users + to supply 3 servers, the application should only specify the servers it can. For example, + a room with only 2 users in it would result in maximum 2 ``via`` parameters. From 99cd064f2623ce762319f571c10b79912db3bc68 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 5 Apr 2019 16:37:27 -0600 Subject: [PATCH 022/128] Update original MSC as per proposal guidelines --- proposals/1704-matrix.to-permalinks.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/1704-matrix.to-permalinks.md b/proposals/1704-matrix.to-permalinks.md index 1430565c..7e1a2b28 100644 --- a/proposals/1704-matrix.to-permalinks.md +++ b/proposals/1704-matrix.to-permalinks.md @@ -33,4 +33,7 @@ is to pick up to 3 unique servers where the first one is that of the user with t highest power level in the room, provided that power level is 50 or higher. The other 2 servers should be the most popular servers in the room based on the number of joined users. This same heuristic should apply to the first server if no user meets the power -level requirements. +level requirements. Servers blocked by server ACLs should not be picked because they +are unlikely to continue being residents of the room. Similarly, IP addresses should +not be picked because they cannot be redirected to another location like domain names +can, making them a higher risk option. From b41fbc86b6d07a93c10f074c97befe9fe7e4733e Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Tue, 9 Apr 2019 13:56:45 +0100 Subject: [PATCH 023/128] add further potential issues and security concerns --- ...emove-prev_event-from-essential-keys-list.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/proposals/1954-remove-prev_event-from-essential-keys-list.md b/proposals/1954-remove-prev_event-from-essential-keys-list.md index 992b3e80..1cd0fd36 100644 --- a/proposals/1954-remove-prev_event-from-essential-keys-list.md +++ b/proposals/1954-remove-prev_event-from-essential-keys-list.md @@ -53,10 +53,23 @@ the spec. On balance this seems unlikely and in the worst case those implementors could add the change to a subsequent room version, eventually reaching spec consistency as older room versions are deprecated. +Another scenario is that a client may redact events according to the spec as is +and persist prev_content through the redaction, thereby diverting from that on +the server(s). Client authors will have to update their code to drop +```prev_content``` - however, given that prev_content should not be used in +important calculations and/or visualisations, this ought to be relatively +uninvaisive change. + + ## Security considerations -I am unaware of any security issues related to this proposal, but can certainly -see issues with a precedent that the federation deviates from the spec. +A further reason to support removal of ```prev_content``` is the case where a +malicious user adds illegal or abusive content into a state event and then +overwrites that state event. The content would then be preserved through the +redaction. + +Additionally, there are plenty of reasons to have security concerns over a +precedent that the federation can deviate from the spec. ## Conclusions Removing ```prev_content``` is pragmatic response to the current situation. It From 24e0ec4bcecee55e34d54bd8a747f619b57384b5 Mon Sep 17 00:00:00 2001 From: aqtusia <47819902+aqtusia@users.noreply.github.com> Date: Sun, 14 Apr 2019 18:17:44 +0200 Subject: [PATCH 024/128] Replace /bind with /3pid/bind --- api/identity/associations.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/identity/associations.yaml b/api/identity/associations.yaml index edd43f5d..152a0a9b 100644 --- a/api/identity/associations.yaml +++ b/api/identity/associations.yaml @@ -90,7 +90,7 @@ paths: } schema: $ref: "../client-server/definitions/errors/error.yaml" - "/bind": + "/3pid/bind": post: summary: Publish an association between a session and a Matrix user ID. description: |- From 2eb9708f7fcdead5e21ad22bd8a054e1ae4bfd31 Mon Sep 17 00:00:00 2001 From: aqtusia <47819902+aqtusia@users.noreply.github.com> Date: Sun, 14 Apr 2019 18:21:59 +0200 Subject: [PATCH 025/128] Replace /unbind with /3pid/unbind --- proposals/1915-unbind-identity-server-param.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index 053552f6..6817ece3 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -55,12 +55,12 @@ from the given identity server. ### Identity Server 3PID Unbind API -Add `POST /_matrix/identity/api/v1/unbind` with `mxid` and `threepid` fields. +Add `POST /_matrix/identity/api/v1/3pid/unbind` with `mxid` and `threepid` fields. The `mxid` is the user's `user_id` and `threepid` is a dict with the usual `medium` and `address` fields. If the server returns a 400, 404 or 501 HTTP error code then the homeserver -should assume that the identity server doesn't support the `/unbind` API, unless +should assume that the identity server doesn't support the `/3pid/unbind` API, unless it returns a specific matrix error response (i.e. the body is a JSON object with `error` and `errcode` fields). @@ -73,7 +73,7 @@ The identity server should authenticate the request in one of two ways: Example: ``` -POST /_matrix/identity/api/v1/unbind HTTP/1.1 +POST /_matrix/identity/api/v1/3pid/unbind HTTP/1.1 { "mxid": "@foobar:example.com", From 911fb94ea0d7213ea363293c1db39ca4391669f7 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Mon, 15 Apr 2019 17:08:09 +0100 Subject: [PATCH 026/128] typos --- .../1954-remove-prev_event-from-essential-keys-list.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1954-remove-prev_event-from-essential-keys-list.md b/proposals/1954-remove-prev_event-from-essential-keys-list.md index 1cd0fd36..735fac01 100644 --- a/proposals/1954-remove-prev_event-from-essential-keys-list.md +++ b/proposals/1954-remove-prev_event-from-essential-keys-list.md @@ -48,7 +48,7 @@ retrospectively to describe the de-facto redaction algorithm. ## Potential issues It is theoretically possible that a closed federation could exist whose servers -do follow the spec as is. This MSC would render those servers uncompliant with +do follow the spec as is. This MSC would render those servers non-compliant with the spec. On balance this seems unlikely and in the worst case those implementors could add the change to a subsequent room version, eventually reaching spec consistency as older room versions are deprecated. @@ -57,8 +57,8 @@ Another scenario is that a client may redact events according to the spec as is and persist prev_content through the redaction, thereby diverting from that on the server(s). Client authors will have to update their code to drop ```prev_content``` - however, given that prev_content should not be used in -important calculations and/or visualisations, this ought to be relatively -uninvaisive change. +important calculations and/or visualisations, this ought to be a relatively +non-invasive change. ## Security considerations From 043dddc49061f4a8837a0b9d5dc0fb0c0d79f85d Mon Sep 17 00:00:00 2001 From: Jonas Platte Date: Sat, 20 Apr 2019 21:56:43 +0200 Subject: [PATCH 027/128] Fix a typo in m.call.invite --- event-schemas/schema/m.call.invite | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/event-schemas/schema/m.call.invite b/event-schemas/schema/m.call.invite index ebf09267..65796e1e 100644 --- a/event-schemas/schema/m.call.invite +++ b/event-schemas/schema/m.call.invite @@ -10,7 +10,7 @@ "properties": { "call_id": { "type": "string", - "description": "A unique identifer for the call." + "description": "A unique identifier for the call." }, "offer": { "type": "object", From 14715468bb6aa157601fa271bd3a57507f90b8dd Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Fri, 26 Apr 2019 14:41:19 +0100 Subject: [PATCH 028/128] Make thumbnail dimensions mandatory Fixes #1883 --- api/client-server/content-repo.yaml | 2 ++ changelogs/client_server/newsfragments/1975.clarification | 1 + 2 files changed, 3 insertions(+) create mode 100644 changelogs/client_server/newsfragments/1975.clarification diff --git a/api/client-server/content-repo.yaml b/api/client-server/content-repo.yaml index 5f4e9111..07df18fe 100644 --- a/api/client-server/content-repo.yaml +++ b/api/client-server/content-repo.yaml @@ -202,6 +202,7 @@ paths: type: integer x-example: 64 name: width + required: true description: |- The *desired* width of the thumbnail. The actual thumbnail may not match the size specified. @@ -209,6 +210,7 @@ paths: type: integer x-example: 64 name: height + required: true description: |- The *desired* height of the thumbnail. The actual thumbnail may not match the size specified. diff --git a/changelogs/client_server/newsfragments/1975.clarification b/changelogs/client_server/newsfragments/1975.clarification new file mode 100644 index 00000000..03a9b754 --- /dev/null +++ b/changelogs/client_server/newsfragments/1975.clarification @@ -0,0 +1 @@ +Clarify that ``width`` and ``height`` are required parameters on ``/_matrix/media/r0/thumbnail/{serverName}/{mediaId}``. It is somewhat meaningless to request a thumbnail without specifying a desired size, and Synapse has never permitted such requests. \ No newline at end of file From 6cdc8982fa84fc25b6ea9d20d241df8e2795b643 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Fri, 26 Apr 2019 15:58:31 +0100 Subject: [PATCH 029/128] jenkins is dead, long live buildkite --- .buildkite/pipeline.yaml | 11 +++++++++++ jenkins.sh | 3 --- scripts/requirements.txt | 8 ++++++-- 3 files changed, 17 insertions(+), 5 deletions(-) create mode 100644 .buildkite/pipeline.yaml delete mode 100755 jenkins.sh diff --git a/.buildkite/pipeline.yaml b/.buildkite/pipeline.yaml new file mode 100644 index 00000000..e98d7026 --- /dev/null +++ b/.buildkite/pipeline.yaml @@ -0,0 +1,11 @@ +steps: + - label: ":books: Build spec" + command: + - python3 -m venv env + - env/bin/pip install -r scripts/requirements.txt + - ". env/bin/activate; scripts/generate-matrix-org-assets" + artifact_paths: + - assets.tar.gz + plugins: + - docker#v3.0.1: + image: "python:3.6" diff --git a/jenkins.sh b/jenkins.sh deleted file mode 100755 index 79b77acb..00000000 --- a/jenkins.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/sh - -exec ./scripts/test-and-build.sh diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 2a7d7ff8..66027f91 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -4,8 +4,12 @@ docutils >= 0.14 pygments >= 2.2.0 Jinja2 >= 2.9.6 -jsonschema >= 2.6.0 + +# jsonschema 3.0.0 objects to the $refs in our schema file. TODO: figure out +# why. +jsonschema >= 2.6.0, < 3.0.0 + PyYAML >= 3.12 requests >= 2.18.4 towncrier == 18.6.0 -six >= 1.11.0 \ No newline at end of file +six >= 1.11.0 From 4e9dc2098ff65acda435797b8d824653289f0e8b Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Tue, 30 Apr 2019 17:43:21 +0100 Subject: [PATCH 030/128] Fix comments which refer to jenkins. (#1981) * Fix comments which refer to jenkins. * Spelling Co-Authored-By: turt2live --- scripts/test-and-build.sh | 2 +- specification/proposals.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/test-and-build.sh b/scripts/test-and-build.sh index 710b03dd..f45e2da6 100755 --- a/scripts/test-and-build.sh +++ b/scripts/test-and-build.sh @@ -30,5 +30,5 @@ go get gopkg.in/fsnotify/fsnotify.v1 # build the spec for matrix.org. # (we don't actually use it on travis, but it's still useful to check we -# can build it. On Jenkins, this is then used to deploy to matrix.org). +# can build it. On Buildkite, this is then used to deploy to matrix.org). ./scripts/generate-matrix-org-assets diff --git a/specification/proposals.rst b/specification/proposals.rst index 371850ab..94878f80 100644 --- a/specification/proposals.rst +++ b/specification/proposals.rst @@ -1,6 +1,6 @@ Tables of Tracked Proposals --------------------------- -This file is autogenerated by a jenkins build process +This file is generated by an automated process on our build server. -View the current live version `at https://matrix.org/docs/spec/proposals `_ +View the current live version `at https://matrix.org/docs/spec/proposals `_. From 7c7bc677fb485d3df0e2dcb52554eb1bf2bd7f28 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Tue, 7 May 2019 00:11:14 +0100 Subject: [PATCH 031/128] Trigger matrix.org rebuild --- .buildkite/pipeline.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.buildkite/pipeline.yaml b/.buildkite/pipeline.yaml index e98d7026..6e595120 100644 --- a/.buildkite/pipeline.yaml +++ b/.buildkite/pipeline.yaml @@ -9,3 +9,8 @@ steps: plugins: - docker#v3.0.1: image: "python:3.6" + + - label: "rebuild matrix.org" + trigger: "matrix-dot-org" + async: true + branches: "master" From da82a423084d4bd5a209e73548001fb89024aa35 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 15:40:39 -0700 Subject: [PATCH 032/128] fix grammatical error --- specification/modules/end_to_end_encryption.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 59d241b6..8d8a2d06 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -182,7 +182,7 @@ process: field as the ``from`` parameter. If the client is tracking the device list of any of the users listed in the response, it marks them as outdated. It combines this list with those already flagged as outdated, and initiates a - |/keys/query|_ requests for all of them. + |/keys/query|_ request for all of them. .. Warning:: From c233535de00e8c66d80fa807535b68f3e9ec291b Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 15:51:32 -0700 Subject: [PATCH 033/128] fix typo --- specification/modules/end_to_end_encryption.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 59d241b6..9ad80a12 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -460,7 +460,7 @@ Keys can be manually exported from one device to an encrypted file, copied to another device, and imported. The file is encrypted using a user-supplied passphrase, and is created as follows: -1. Encode the sessions a JSON object, formatted as described in `Key export +1. Encode the sessions as a JSON object, formatted as described in `Key export format`_. 2. Generate a 512-bit key from the user-entered passphrase by computing `PBKDF2`_\(HMAC-SHA-512, passphrase, S, N, 512), where S is a 128-bit From 40482f76163951aaf95a2ccef848fc43789ec3f9 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 15:55:07 -0700 Subject: [PATCH 034/128] Add missing period --- specification/appendices/signing_json.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/appendices/signing_json.rst b/specification/appendices/signing_json.rst index 795d6669..8036950e 100644 --- a/specification/appendices/signing_json.rst +++ b/specification/appendices/signing_json.rst @@ -59,7 +59,7 @@ Grammar +++++++ Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing -insignificant whitespace, fractions, exponents and redundant character escapes +insignificant whitespace, fractions, exponents and redundant character escapes. .. code:: From 3c62b90dfbf58a4ff99c4e7fc403d8737816dc2b Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 16:59:10 -0700 Subject: [PATCH 035/128] Add missing punctuation --- api/client-server/keys.yaml | 2 +- api/server-server/user_keys.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/api/client-server/keys.yaml b/api/client-server/keys.yaml index 55f8a5a5..718703fd 100644 --- a/api/client-server/keys.yaml +++ b/api/client-server/keys.yaml @@ -253,7 +253,7 @@ paths: responses: 200: description: - The claimed keys + The claimed keys. schema: type: object properties: diff --git a/api/server-server/user_keys.yaml b/api/server-server/user_keys.yaml index 63c74d20..3c59cf81 100644 --- a/api/server-server/user_keys.yaml +++ b/api/server-server/user_keys.yaml @@ -63,7 +63,7 @@ paths: - one_time_keys responses: 200: - description: The claimed keys + description: The claimed keys. schema: type: object properties: From 57cf1aaa96f9767444f42a32f0f7c7d21b92062e Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 19:37:25 -0700 Subject: [PATCH 036/128] Add missing period --- specification/modules/send_to_device.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/send_to_device.rst b/specification/modules/send_to_device.rst index 86288546..cca0338a 100644 --- a/specification/modules/send_to_device.rst +++ b/specification/modules/send_to_device.rst @@ -108,7 +108,7 @@ to_device ToDevice Optional. Information on the send-to-device messages ========= ========= ============================================= Parameter Type Description ========= ========= ============================================= -events [Event] List of send-to-device messages +events [Event] List of send-to-device messages. ========= ========= ============================================= ``Event`` From 20d2fdc288e3e79909db449f99895f1a2ad77764 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 19:51:11 -0700 Subject: [PATCH 037/128] Add changelog --- changelogs/client_server/newsfragments/1988.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1988.clarification diff --git a/changelogs/client_server/newsfragments/1988.clarification b/changelogs/client_server/newsfragments/1988.clarification new file mode 100644 index 00000000..b0f05203 --- /dev/null +++ b/changelogs/client_server/newsfragments/1988.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. From abd770419b6169510293e963c373625bfefd0646 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 19:52:38 -0700 Subject: [PATCH 038/128] Add changelog --- changelogs/client_server/newsfragments/1989.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1989.clarification diff --git a/changelogs/client_server/newsfragments/1989.clarification b/changelogs/client_server/newsfragments/1989.clarification new file mode 100644 index 00000000..b0f05203 --- /dev/null +++ b/changelogs/client_server/newsfragments/1989.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. From 713e4401b47ff71da6aad21ce2bd781281f48007 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 7 May 2019 19:54:38 -0700 Subject: [PATCH 039/128] Add changelogs --- changelogs/client_server/newsfragments/1991.clarification | 1 + changelogs/server_server/newsfragments/1991.clarification | 1 + 2 files changed, 2 insertions(+) create mode 100644 changelogs/client_server/newsfragments/1991.clarification create mode 100644 changelogs/server_server/newsfragments/1991.clarification diff --git a/changelogs/client_server/newsfragments/1991.clarification b/changelogs/client_server/newsfragments/1991.clarification new file mode 100644 index 00000000..b0f05203 --- /dev/null +++ b/changelogs/client_server/newsfragments/1991.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. diff --git a/changelogs/server_server/newsfragments/1991.clarification b/changelogs/server_server/newsfragments/1991.clarification new file mode 100644 index 00000000..b0f05203 --- /dev/null +++ b/changelogs/server_server/newsfragments/1991.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. From 00b7b70c062e78c8bdbce060a56d099540654f9e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 7 May 2019 23:09:39 -0600 Subject: [PATCH 040/128] Create 1992.clarification --- changelogs/client_server/newsfragments/1992.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/1992.clarification diff --git a/changelogs/client_server/newsfragments/1992.clarification b/changelogs/client_server/newsfragments/1992.clarification new file mode 100644 index 00000000..b0f05203 --- /dev/null +++ b/changelogs/client_server/newsfragments/1992.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. From 383e02835e5c9e60d0f574e69cf7cd6fa2cae262 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 May 2019 18:07:58 +0100 Subject: [PATCH 041/128] Words on using m.login.dummy for disambiguation Add some text on how m.login.dummy can be used to distinguish a flow that would otherwise be a subset of other flows. --- specification/client_server_api.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index f9f815f7..b2ff3dbd 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -789,7 +789,14 @@ Dummy Auth :Description: Dummy authentication always succeeds and requires no extra parameters. Its purpose is to allow servers to not require any form of User-Interactive - Authentication to perform a request. + Authentication to perform a request. It can also be used to differentiate + flows where otherwise one flow would be a subset of another flow. eg. if + a server offers flows ``m.login.recaptcha`` and ``m.login.recaptcha, + m.login.email.identity`` and the client completes the recaptcha stage first, + the auth would succeed with the former flow, even if the client was intending + to then complete the email auth stage. A server can instead send flows + ``m.login.recaptcha, m.login.dummy`` and ``m.login.recaptcha, + m.login.email.identity`` to fix the ambiguity. To use this authentication type, clients should submit an auth dict with just the type and session, if provided: From c38581fb868c0d198297d82932d2b8117264bad3 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 May 2019 19:20:01 +0100 Subject: [PATCH 042/128] Too many spaces Co-Authored-By: Travis Ralston --- specification/client_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index b2ff3dbd..6f65ebb1 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -790,7 +790,7 @@ Dummy Auth Dummy authentication always succeeds and requires no extra parameters. Its purpose is to allow servers to not require any form of User-Interactive Authentication to perform a request. It can also be used to differentiate - flows where otherwise one flow would be a subset of another flow. eg. if + flows where otherwise one flow would be a subset of another flow. eg. if a server offers flows ``m.login.recaptcha`` and ``m.login.recaptcha, m.login.email.identity`` and the client completes the recaptcha stage first, the auth would succeed with the former flow, even if the client was intending From 37871106c6fc6013c17b5711fb93853fff140267 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 21 May 2019 16:21:47 +0100 Subject: [PATCH 043/128] MSC2002: Proposal for adopting MSC1884 as v4 rooms (#2002) --- proposals/2002-rooms-v4.md | 54 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 proposals/2002-rooms-v4.md diff --git a/proposals/2002-rooms-v4.md b/proposals/2002-rooms-v4.md new file mode 100644 index 00000000..defacee1 --- /dev/null +++ b/proposals/2002-rooms-v4.md @@ -0,0 +1,54 @@ +# MSC 2002 - Rooms V4 + +This MSC proposes creating a new room version named v4 to allow servers to switch +event ID grammar to that proposed in MSC1884. + +## Proposal + +The new room version is called "4". The only difference between v4 and v3 is +that v4 rooms use the grammar for defining event IDs as proposed in MSC1884 - +expressing event IDs as url-safe base64 rather than plain base64 for the +convenience of client implementors. + +It is not proposed that servers change the default room version used when +creating new rooms, and it is not proposed that servers recommend upgrading +existing rooms to v4. + +## Rationale and Context + +We would like to default to creating rooms with a reasonably secure room +algorithm in upcoming Matrix 1.0. We do not want to default to creating v3 +rooms due to the inconvenience the v3 event ID grammar might cause, so instead +we are proposing favouring v4. + +Ideally we would also include other room algorithm changes in v4 (e.g. +honouring server signing key validity periods, as per +https://github.com/matrix-org/synapse/issues/4364), but as spec & +implementation work is still ongoing there, we are proposing using v4 as a +room version which can be supported in the wild before Matrix 1.0 and then +used as the initial default for creating rooms. The expectation is for the +versions of the spec which coincide with 1.0 to also support v5 rooms, but in +practice v5 will not be marked as default until it has sufficient adoption on +the public network. + +The expectation is never to recommend upgrading existing +rooms to v4, but instead v5 (once ready), to avoid overburdening room admins +with upgrade notifications. + +To conclude, the proposed plan is: + 1. Define room v4 as MSC1884 + 2. Introduce servers with v4 support into the public network as rapidly as possible + 3. Wait for enough servers to speak v4. Meanwhile: + 1. Define an MSC for honouring server signing key validity periods + 2. Implement this MSC + 3. Define room v5 as this MSC + 4. Release Matrix 1.0, defining room v4 as the new default for creating rooms, + but also shipping support for room v5 for the first time. + 5. Wait for enough servers to speak v5 rooms. + 6. Define room v5 as the new default for creating rooms. + 7. Define room versions prior to v5 as unsafe, thus prompting users to upgrade their + rooms to v5. + +The reason we don't wait for v5 to be widespread before releasing 1.0 is to avoid +delaying the 1.0 yet further. It is good enough for 1.0 to support v5 without it +also being the default for creating rooms. From 77050062601cad901623660de151081608fb5ca3 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 22 May 2019 12:43:43 -0400 Subject: [PATCH 044/128] 1 hour seems to be fine --- proposals/1719-olm_unwedging.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index e696c1e2..50b6f0e6 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -16,8 +16,7 @@ empty contents. In order to avoid creating too many extra sessions, a client should rate-limit the number of new sessions it creates per device that it receives a message from; the client should not create a new session with another device if it has -already created one for that given device in the past 1 hour. (TODO: is 1 hour -the right amount of time?) +already created one for that given device in the past 1 hour. Clients may wish to ask the sender of the undecryptable messages to re-send the message. For exampe, if the undecryptable message was a megolm session, then From d39baba21b64c0dfe3b57e2bbbe8b9609fb69f2d Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 22 May 2019 12:44:04 -0400 Subject: [PATCH 045/128] add a security note --- proposals/1719-olm_unwedging.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index 50b6f0e6..3e0ce35d 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -45,6 +45,10 @@ time of the session. The spec will be changed to read: ## Security considerations +An attacker could use this to create a new session on a device that they are +able to read. However, this would require the attacker to have compromised the +device's keys. + ## Conclusion This proposal outlines how wedged olm sessions can be replaced by a new From dd74baa5d0fb6c2f15878a929c1a70e2edf69e62 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 22 May 2019 12:54:01 -0400 Subject: [PATCH 046/128] clarify wording --- proposals/1719-olm_unwedging.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index 3e0ce35d..67bd6010 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -36,8 +36,7 @@ time of the session. The spec will be changed to read: > If a client has multiple sessions established with another device, it should > use the session from which it last received and successfully decrypted a > message. For these purposes, a session that has not received any messages -> should consider its creation time to be the time that it last received a -> message. +> should use its creation time as the time that it last received a message. ## Tradeoffs From ac08c8461240e2fc9f88bc59c34263d8add4115f Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 23 May 2019 14:39:46 -0400 Subject: [PATCH 047/128] remove duplicate consideration Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- proposals/1719-olm_unwedging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index 67bd6010..c181e3f7 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -29,7 +29,7 @@ another device, it should use the session from which it last received a message." (the last paragraph of the [`m.olm.v1.curve25519-aes-sha2` section](https://matrix.org/docs/spec/client_server/r0.4.0.html#m-olm-v1-curve25519-aes-sha2)). When comparing the time of the last received message for each session, the -client should consider only consider messages that were successfully decrypted, +client should only consider messages that were successfully decrypted, and for sessions that have never received a message, it should use the creation time of the session. The spec will be changed to read: From 9c2a789d34da6a465ace23875d9d9e89ff7ced48 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 23 May 2019 22:13:57 -0600 Subject: [PATCH 048/128] Add missing changelogs and make existing ones match conventions The conventions are not set in stone, however the changelog should not be a mixed bag of voices. --- changelogs/application_service/newsfragments/1650.clarification | 1 + changelogs/client_server/newsfragments/1650.clarification | 1 + changelogs/client_server/newsfragments/1656.clarification | 1 + changelogs/client_server/newsfragments/1701.feature | 2 +- changelogs/client_server/newsfragments/1744.clarification | 2 +- changelogs/client_server/newsfragments/1889.clarification | 2 +- changelogs/client_server/newsfragments/1891.clarification | 1 + changelogs/client_server/newsfragments/1969.clarification | 1 + changelogs/client_server/newsfragments/1975.clarification | 2 +- changelogs/client_server/newsfragments/1999.clarification | 1 + changelogs/identity_service/newsfragments/1967.clarification | 1 + changelogs/server_server/newsfragments/1650.clarification | 1 + 12 files changed, 12 insertions(+), 4 deletions(-) create mode 100644 changelogs/application_service/newsfragments/1650.clarification create mode 100644 changelogs/client_server/newsfragments/1650.clarification create mode 100644 changelogs/client_server/newsfragments/1656.clarification create mode 100644 changelogs/client_server/newsfragments/1891.clarification create mode 100644 changelogs/client_server/newsfragments/1969.clarification create mode 100644 changelogs/client_server/newsfragments/1999.clarification create mode 100644 changelogs/identity_service/newsfragments/1967.clarification create mode 100644 changelogs/server_server/newsfragments/1650.clarification diff --git a/changelogs/application_service/newsfragments/1650.clarification b/changelogs/application_service/newsfragments/1650.clarification new file mode 100644 index 00000000..617b7ab6 --- /dev/null +++ b/changelogs/application_service/newsfragments/1650.clarification @@ -0,0 +1 @@ +Change examples to use example.org instead of a real domain. diff --git a/changelogs/client_server/newsfragments/1650.clarification b/changelogs/client_server/newsfragments/1650.clarification new file mode 100644 index 00000000..617b7ab6 --- /dev/null +++ b/changelogs/client_server/newsfragments/1650.clarification @@ -0,0 +1 @@ +Change examples to use example.org instead of a real domain. diff --git a/changelogs/client_server/newsfragments/1656.clarification b/changelogs/client_server/newsfragments/1656.clarification new file mode 100644 index 00000000..0c8f4ad0 --- /dev/null +++ b/changelogs/client_server/newsfragments/1656.clarification @@ -0,0 +1 @@ +Clarify that ``state_default`` in ``m.room.power_levels`` always defaults to 50. diff --git a/changelogs/client_server/newsfragments/1701.feature b/changelogs/client_server/newsfragments/1701.feature index 39c22dd7..cf6084ae 100644 --- a/changelogs/client_server/newsfragments/1701.feature +++ b/changelogs/client_server/newsfragments/1701.feature @@ -1 +1 @@ -Documented megolm session export format. \ No newline at end of file +Add megolm session export format. diff --git a/changelogs/client_server/newsfragments/1744.clarification b/changelogs/client_server/newsfragments/1744.clarification index dc103920..dfe838f1 100644 --- a/changelogs/client_server/newsfragments/1744.clarification +++ b/changelogs/client_server/newsfragments/1744.clarification @@ -1 +1 @@ -Add missing status_msg to m.presence schema. +Add missing ``status_msg`` to ``m.presence`` schema. diff --git a/changelogs/client_server/newsfragments/1889.clarification b/changelogs/client_server/newsfragments/1889.clarification index 5026dab3..2737a7ee 100644 --- a/changelogs/client_server/newsfragments/1889.clarification +++ b/changelogs/client_server/newsfragments/1889.clarification @@ -1 +1 @@ -Add the missing `m.push_rules` event schema. +Add the missing ``m.push_rules`` event schema. diff --git a/changelogs/client_server/newsfragments/1891.clarification b/changelogs/client_server/newsfragments/1891.clarification new file mode 100644 index 00000000..ef4edfb4 --- /dev/null +++ b/changelogs/client_server/newsfragments/1891.clarification @@ -0,0 +1 @@ +Clarify how modern day local echo is meant to be solved by clients. diff --git a/changelogs/client_server/newsfragments/1969.clarification b/changelogs/client_server/newsfragments/1969.clarification new file mode 100644 index 00000000..b0f05203 --- /dev/null +++ b/changelogs/client_server/newsfragments/1969.clarification @@ -0,0 +1 @@ +Fix various spelling mistakes throughout the specification. diff --git a/changelogs/client_server/newsfragments/1975.clarification b/changelogs/client_server/newsfragments/1975.clarification index 03a9b754..ac118bfd 100644 --- a/changelogs/client_server/newsfragments/1975.clarification +++ b/changelogs/client_server/newsfragments/1975.clarification @@ -1 +1 @@ -Clarify that ``width`` and ``height`` are required parameters on ``/_matrix/media/r0/thumbnail/{serverName}/{mediaId}``. It is somewhat meaningless to request a thumbnail without specifying a desired size, and Synapse has never permitted such requests. \ No newline at end of file +Clarify that ``width`` and ``height`` are required parameters on ``/_matrix/media/r0/thumbnail/{serverName}/{mediaId}``. diff --git a/changelogs/client_server/newsfragments/1999.clarification b/changelogs/client_server/newsfragments/1999.clarification new file mode 100644 index 00000000..748c55f2 --- /dev/null +++ b/changelogs/client_server/newsfragments/1999.clarification @@ -0,0 +1 @@ +Clarify how ``m.login.dummy`` can be used to disambiguate login flows. diff --git a/changelogs/identity_service/newsfragments/1967.clarification b/changelogs/identity_service/newsfragments/1967.clarification new file mode 100644 index 00000000..b080caeb --- /dev/null +++ b/changelogs/identity_service/newsfragments/1967.clarification @@ -0,0 +1 @@ +Fix route for ``/3pid/bind``. diff --git a/changelogs/server_server/newsfragments/1650.clarification b/changelogs/server_server/newsfragments/1650.clarification new file mode 100644 index 00000000..617b7ab6 --- /dev/null +++ b/changelogs/server_server/newsfragments/1650.clarification @@ -0,0 +1 @@ +Change examples to use example.org instead of a real domain. From 3c38956510511d97dc770c0ec0be4d681b382af0 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 May 2019 11:57:22 -0600 Subject: [PATCH 049/128] Remove prev_content from the redaction essential keys list As per [MSC1954](https://github.com/matrix-org/matrix-doc/pull/1954) No known changes since the proposal was accepted. --- changelogs/client_server/newsfragments/2016.clarification | 1 + specification/client_server_api.rst | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/2016.clarification diff --git a/changelogs/client_server/newsfragments/2016.clarification b/changelogs/client_server/newsfragments/2016.clarification new file mode 100644 index 00000000..77ea0d4c --- /dev/null +++ b/changelogs/client_server/newsfragments/2016.clarification @@ -0,0 +1 @@ +Remove ``prev_content`` from the redaction algorithm's essential keys list. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 6f65ebb1..604c2b1c 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1500,7 +1500,6 @@ the following list: - ``room_id`` - ``sender`` - ``state_key`` -- ``prev_content`` - ``content`` - ``hashes`` - ``signatures`` From 3b0e194ff7a5a4b6e20bbf1572337738904feda5 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 May 2019 15:50:05 -0600 Subject: [PATCH 050/128] Add version 4 rooms to the spec As per [MSC1884](https://github.com/matrix-org/matrix-doc/pull/1884) and [MSC2002](https://github.com/matrix-org/matrix-doc/pull/2002). No known changes since the proposals were accepted. Due to being in the area: This fixes https://github.com/matrix-org/matrix-doc/issues/1863 --- api/server-server/definitions/pdu_v3.yaml | 4 ++ api/server-server/definitions/pdu_v4.yaml | 47 ++++++++++++++ api/server-server/examples/pdu_v3.json | 3 +- api/server-server/examples/pdu_v4.json | 12 ++++ specification/index.rst | 1 + specification/rooms/v4.rst | 76 +++++++++++++++++++++++ specification/targets.yaml | 4 ++ 7 files changed, 146 insertions(+), 1 deletion(-) create mode 100644 api/server-server/definitions/pdu_v4.yaml create mode 100644 api/server-server/examples/pdu_v4.json create mode 100644 specification/rooms/v4.rst diff --git a/api/server-server/definitions/pdu_v3.yaml b/api/server-server/definitions/pdu_v3.yaml index 8d41fbda..38105098 100644 --- a/api/server-server/definitions/pdu_v3.yaml +++ b/api/server-server/definitions/pdu_v3.yaml @@ -20,6 +20,10 @@ allOf: - $ref: "unsigned_pdu_base.yaml" - type: object properties: + redacts: + type: string + description: For redaction events, the ID of the event being redacted. + example: "$def/456+oldevent" auth_events: type: array items: diff --git a/api/server-server/definitions/pdu_v4.yaml b/api/server-server/definitions/pdu_v4.yaml new file mode 100644 index 00000000..55b1c26f --- /dev/null +++ b/api/server-server/definitions/pdu_v4.yaml @@ -0,0 +1,47 @@ +# Copyright 2019 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +type: object +title: Persistent Data Unit +description: A persistent data unit (event) for room version 3 and beyond. +example: + $ref: "../examples/pdu_v4.json" +allOf: + - $ref: "pdu_v3.yaml" + - type: object + properties: + redacts: + type: string + description: For redaction events, the ID of the event being redacted. + example: "$def_456-oldevent" + auth_events: + type: array + items: + type: string + description: Event ID. + description: |- + Event IDs for the authorization events that would + allow this event to be in the room. + example: ["$URLsafe-base64EncodedHash", "$Another_Event"] + prev_events: + type: array + items: + type: string + description: Event ID. + description: |- + Event IDs for the most recent events in the room + that the homeserver was aware of when it made this event. + example: ["$URLsafe-base64EncodedHash", "$Another_Event"] + required: + - auth_events + - prev_events diff --git a/api/server-server/examples/pdu_v3.json b/api/server-server/examples/pdu_v3.json index 6a454b4e..acffdf26 100644 --- a/api/server-server/examples/pdu_v3.json +++ b/api/server-server/examples/pdu_v3.json @@ -15,5 +15,6 @@ "prev_events": [ "$base64encodedeventid", "$adifferenteventid" - ] + ], + "redacts": "$some/old+event" } diff --git a/api/server-server/examples/pdu_v4.json b/api/server-server/examples/pdu_v4.json new file mode 100644 index 00000000..3c2f0e22 --- /dev/null +++ b/api/server-server/examples/pdu_v4.json @@ -0,0 +1,12 @@ +{ + "$ref": "pdu_v3.json", + "auth_events": [ + "$urlsafe_base64_encoded_eventid", + "$a-different-event-id" + ], + "prev_events": [ + "$urlsafe_base64_encoded_eventid", + "$a-different-event-id" + ], + "redacts": "$some-old_event" +} diff --git a/specification/index.rst b/specification/index.rst index 19c5ad22..33dff5a3 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -493,6 +493,7 @@ The available room versions are: * `Version 1 `_ - **Stable**. The current version of most rooms. * `Version 2 `_ - **Stable**. Implements State Resolution Version 2. * `Version 3 `_ - **Stable**. Introduces events whose IDs are the event's hash. +* `Version 4 `_ - **Stable**. Builds on v3 by using URL-safe base64 for event IDs. Specification Versions ---------------------- diff --git a/specification/rooms/v4.rst b/specification/rooms/v4.rst new file mode 100644 index 00000000..8d2cc59b --- /dev/null +++ b/specification/rooms/v4.rst @@ -0,0 +1,76 @@ +.. Copyright 2019 The Matrix.org Foundation C.I.C. +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Room Version 4 +============== + +This room version builds on `version 3 `_ using a different encoding for +event IDs. + +.. contents:: Table of Contents +.. sectnum:: + + +Client considerations +--------------------- + +This room version changes the format form event IDs sent to clients. Clients should +already be treating event IDs as opaque identifiers, and should not be concerned with +the format of them. Clients should still encode the event ID when including it in a +request path. + +Clients should expect to see event IDs changed from the format of ``$randomstring:example.org`` +to something like ``$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg`` (note the lack of domain). + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + intricacies contained here. The section above regarding client considerations + is the resource that Client-Server API use cases should reference. + + +Room version 4 uses the same algorithms defined in `room version 3 `_, however +using URL-safe base64 to generate the event ID. + +Event IDs +~~~~~~~~~ + +.. admonition:: Rationale + + Room version 3 generated event IDs that were difficult for client implementations + which were not encoding the event ID to function in those rooms. It additionally + raised concern due to the ``/`` character being interpretted differently by some + reverse proxy software, and generally made administration harder. + +The event ID is the `reference hash`_ of the event encoded using a variation of +`Unpadded Base64`_ which replaces the 62nd and 63rd characters with ``-`` and ``_`` +instead of using ``+`` and ``/``. This matches `RFC4648's definition of URL-safe base64 +`_. Event IDs are still prefixed +with ``$`` and may result in looking like ``$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg``. + +Just like in room version 3, event IDs should not be sent over federation to servers +when the room uses this room version. On the receiving end of an event, the server +should compute the relevant event ID for itself. Room version 3 also changes the format +of ``auth_events`` and ``prev_events`` in a PDU. + +{{definition_ss_pdu_v4}} + +.. _`Unpadded Base64`: ../appendices.html#unpadded-base64 +.. _`Canonical JSON`: ../appendices.html#canonical-json +.. _`Signing Events`: ../server_server/r0.1.1.html#signing-events +.. _`reference hash`: ../server_server/r0.1.1.html#reference-hashes diff --git a/specification/targets.yaml b/specification/targets.yaml index 830449ae..be4a063a 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -38,6 +38,10 @@ targets: files: - rooms/v3.rst version_label: v3 + rooms@v4: # this is translated to be rooms/v4.html + files: + - rooms/v4.rst + version_label: v4 appendices: files: - appendices.rst From ceaccffdf7244e79aa91e3916811f93881be9bca Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 May 2019 17:14:23 -0600 Subject: [PATCH 051/128] Add a .m.rule.tombstone default push rule for room upgrades As per [MSC1930](https://github.com/matrix-org/matrix-doc/pull/1930) There are no known changes to this proposal since it was accepted. --- .../client_server/newsfragments/2020.feature | 1 + specification/modules/push.rst | 35 +++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2020.feature diff --git a/changelogs/client_server/newsfragments/2020.feature b/changelogs/client_server/newsfragments/2020.feature new file mode 100644 index 00000000..0d7c7eb8 --- /dev/null +++ b/changelogs/client_server/newsfragments/2020.feature @@ -0,0 +1 @@ +Add a ``.m.rule.tombstone`` default push rule for room ugprade notifications. diff --git a/specification/modules/push.rst b/specification/modules/push.rst index 9830729a..1bac0c2e 100644 --- a/specification/modules/push.rst +++ b/specification/modules/push.rst @@ -177,7 +177,7 @@ notification is delivered for a matching event. The following actions are define This prevents each matching event from generating a notification ``coalesce`` This enables notifications for matching events but activates homeserver - specific behaviour to intelligently coalesce multiple events into a single + specific behaviour to intelligently coalesce multiple events into a single notification. Not all homeservers may support this. Those that do not support it should treat it as the ``notify`` action. ``set_tweak`` @@ -369,6 +369,37 @@ Definition: } +``.m.rule.tombstone`` +````````````````````` +Matches any event whose type is ``m.room.tombstone``. This is intended +to notify users of a room when it is upgraded, similar to what an +``@room`` notification would accomplish. + +Definition: + +.. code:: json + + { + "rule_id": ".m.rule.tombstone", + "default": true, + "enabled": true, + "conditions": [ + { + "kind": "event_match", + "key": "type", + "pattern": "m.room.tombstone" + } + ], + "actions": [ + "notify", + { + "set_tweak": "highlight", + "value": true + } + ] + } + + ``.m.rule.roomnotif`` ````````````````````` Matches any message whose content is unencrypted and contains the @@ -599,7 +630,7 @@ Definition: Conditions ++++++++++ -Override, Underride and Default Rules MAY have a list of 'conditions'. +Override, Underride and Default Rules MAY have a list of 'conditions'. All conditions must hold true for an event in order to apply the ``action`` for the event. A rule with no conditions always matches. Room, Sender, User and Content rules do not have conditions in the same way, but instead have From 2b96d73305328188f584070f550ce1591d08bbff Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 15:39:35 -0600 Subject: [PATCH 052/128] Fix third party signed definitions for join APIs Fixes https://github.com/matrix-org/matrix-doc/issues/1978 --- .../definitions/third_party_signed.yaml | 41 +++++++++++ api/client-server/joining.yaml | 69 +------------------ specification/modules/third_party_invites.rst | 3 +- 3 files changed, 45 insertions(+), 68 deletions(-) create mode 100644 api/client-server/definitions/third_party_signed.yaml diff --git a/api/client-server/definitions/third_party_signed.yaml b/api/client-server/definitions/third_party_signed.yaml new file mode 100644 index 00000000..c9c761a1 --- /dev/null +++ b/api/client-server/definitions/third_party_signed.yaml @@ -0,0 +1,41 @@ +# Copyright 2019 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +type: object +title: Third Party Signed +description: |- + A signature of an ``m.third_party_invite`` token to prove that this user + owns a third party identity which has been invited to the room. +properties: + sender: + type: string + description: The Matrix ID of the user who issued the invite. + example: "@alice:example.org" + mxid: + type: string + description: The Matrix ID of the invitee. + example: "@bob:example.org" + token: + type: string + description: The state key of the m.third_party_invite event. + example: "random8nonce" + signatures: + type: object + description: A signatures object containing a signature of the entire signed object. + title: Signatures + example: { + "example.org": { + "ed25519:0": "some9signature" + } + } +required: ["sender", "mxid", "token", "signatures"] diff --git a/api/client-server/joining.yaml b/api/client-server/joining.yaml index 1dcf769f..af38d6f9 100644 --- a/api/client-server/joining.yaml +++ b/api/client-server/joining.yaml @@ -58,38 +58,9 @@ paths: name: third_party_signed schema: type: object - example: { - "third_party_signed": { - "sender": "@cat:the.hat", - "mxid": "@green:eggs.ham", - "token": "random8nonce", - "signatures": { - "horton.hears": { - "ed25519:0": "some9signature" - } - } - } - } properties: third_party_signed: - type: object - title: ThirdPartySigned - description: A signature of an ``m.third_party_invite`` token to prove that this user owns a third party identity which has been invited to the room. - properties: - sender: - type: string - description: The Matrix ID of the user who issued the invite. - mxid: - type: string - description: The Matrix ID of the invitee. - token: - type: string - description: The state key of the m.third_party_invite event. - signatures: - type: object - description: A signatures object containing a signature of the entire signed object. - title: Signatures - required: ["sender", "mxid", "token", "signatures"] + $ref: "definitions/third_party_signed.yaml" responses: 200: description: |- @@ -163,45 +134,9 @@ paths: name: third_party_signed schema: type: object - example: { - "third_party_signed": { - "signed": { - "sender": "@cat:the.hat", - "mxid": "@green:eggs.ham", - "token": "random8nonce", - "signatures": { - "horton.hears": { - "ed25519:0": "some9signature" - } - } - } - } - } properties: third_party_signed: - type: object - title: ThirdPartySigned - description: A signature of an ``m.third_party_invite`` token to prove that this user owns a third party identity which has been invited to the room. - properties: - signed: - type: object - title: Signed - properties: - sender: - type: string - description: The Matrix ID of the user who issued the invite. - mxid: - type: string - description: The Matrix ID of the invitee. - token: - type: string - description: The state key of the m.third_party_invite event. - signatures: - type: object - description: A signatures object containing a signature of the entire signed object. - title: Signatures - required: ["sender", "mxid", "token", "signatures"] - required: ["signed"] + $ref: "definitions/third_party_signed.yaml" responses: 200: description: |- diff --git a/specification/modules/third_party_invites.rst b/specification/modules/third_party_invites.rst index 3e11d929..b8ab9657 100644 --- a/specification/modules/third_party_invites.rst +++ b/specification/modules/third_party_invites.rst @@ -38,7 +38,8 @@ When the invitee's homeserver receives the notification of the binding, it should insert an ``m.room.member`` event into the room's graph for that user, with ``content.membership`` = ``invite``, as well as a ``content.third_party_invite`` property which contains proof that the invitee -does indeed own that third party identifier. +does indeed own that third party identifier. See the `m.room.member <#m-room-member>`_ +schema for more information. Events From 17a9524cad8774c9cfe12157742d0f196391de44 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 15:40:29 -0600 Subject: [PATCH 053/128] changelog --- changelogs/client_server/newsfragments/2025.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2025.clarification diff --git a/changelogs/client_server/newsfragments/2025.clarification b/changelogs/client_server/newsfragments/2025.clarification new file mode 100644 index 00000000..9e99b23d --- /dev/null +++ b/changelogs/client_server/newsfragments/2025.clarification @@ -0,0 +1 @@ +Fix the ``third_party_signed`` definitions for the join APIs. From 5eea4a477f98d79371791ab153bdaae91883f4b8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 20:42:21 -0600 Subject: [PATCH 054/128] Add server notices support As per [MSC1452](https://github.com/matrix-org/matrix-doc/issues/1452) Fixes https://github.com/matrix-org/matrix-doc/issues/1254 Although MSC1452 focuses on just the warnings part of the server notices, the base for notices has not been established in the spec. This commit adds the needed support to be able to handle notices. No intentional divergences from the proposal are included in this changeset. There are a few additions which are used in practice although not defined in the proposal, such as who is responsible for aesthetics, sending notices, and other misc rules. --- .../client_server/newsfragments/2026.feature | 1 + .../examples/m.room.message#m.server_notice | 11 +++ .../schema/m.room.message#m.server_notice | 39 ++++++++++ .../templating/matrix_templates/sections.py | 15 +++- .../matrix_templates/templates/events.tmpl | 7 ++ scripts/templating/matrix_templates/units.py | 2 + specification/client_server_api.rst | 5 +- specification/modules/server_notices.rst | 78 +++++++++++++++++++ specification/targets.yaml | 1 + 9 files changed, 154 insertions(+), 5 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2026.feature create mode 100644 event-schemas/examples/m.room.message#m.server_notice create mode 100644 event-schemas/schema/m.room.message#m.server_notice create mode 100644 specification/modules/server_notices.rst diff --git a/changelogs/client_server/newsfragments/2026.feature b/changelogs/client_server/newsfragments/2026.feature new file mode 100644 index 00000000..f82b9aea --- /dev/null +++ b/changelogs/client_server/newsfragments/2026.feature @@ -0,0 +1 @@ +Add support for sending server notices to clients. diff --git a/event-schemas/examples/m.room.message#m.server_notice b/event-schemas/examples/m.room.message#m.server_notice new file mode 100644 index 00000000..0eb44ea7 --- /dev/null +++ b/event-schemas/examples/m.room.message#m.server_notice @@ -0,0 +1,11 @@ +{ + "$ref": "core/room_event.json", + "type": "m.room.message", + "content": { + "body": "Human-readable message to explain the notice", + "msgtype": "m.server_notice", + "server_notice_type": "m.server_notice.usage_limit_reached", + "admin_contact": "mailto:server.admin@example.org", + "limit_type": "monthly_active_user" + } +} diff --git a/event-schemas/schema/m.room.message#m.server_notice b/event-schemas/schema/m.room.message#m.server_notice new file mode 100644 index 00000000..f1848821 --- /dev/null +++ b/event-schemas/schema/m.room.message#m.server_notice @@ -0,0 +1,39 @@ +--- +allOf: + - $ref: core-event-schema/room_event.yaml +description: Represents a server notice for a user. +properties: + content: + properties: + body: + description: A human-readable description of the notice. + type: string + msgtype: + enum: + - m.server_notice + type: string + server_notice_type: + description: |- + The type of notice being represented. + type: string + admin_contact: + description: |- + A URI giving a contact method for the server administrator. Required if the + notice type is ``m.server_notice.usage_limit_reached``. + type: string + limit_type: + description: |- + The kind of usage limit the server has exceeded. Required if the notice type is + ``m.server_notice.usage_limit_reached``. + type: string + required: + - msgtype + - body + - server_notice_type + type: object + type: + enum: + - m.room.message + type: string +title: ServerNoticeMessage +type: object diff --git a/scripts/templating/matrix_templates/sections.py b/scripts/templating/matrix_templates/sections.py index 5961aa24..92afa5ff 100644 --- a/scripts/templating/matrix_templates/sections.py +++ b/scripts/templating/matrix_templates/sections.py @@ -110,12 +110,13 @@ class MatrixSections(Sections): # Special function: Returning a dict will specify multiple sections where # the key is the section name and the value is the value of the section def render_group_events(self): - # map all event schemata to the form $EVENTTYPE_event with s/./_/g - # e.g. m_room_topic_event + # map all event schemata to the form $EVENTTYPE_event with s/.#/_/g + # e.g. m_room_topic_event or m_room_message_m_text_event schemas = self.units.get("event_schemas") renders = {} for event_type in schemas: - renders[event_type.replace(".", "_") + "_event"] = self._render_events( + underscored_event_type = event_type.replace(".", "_").replace("#", "_") + renders[underscored_event_type + "_event"] = self._render_events( lambda x: x == event_type, sorted ) return renders @@ -141,9 +142,15 @@ class MatrixSections(Sections): "m.room.message#m.notice", "m.room.message#m.image", "m.room.message#m.file" ] + excluded_types = [ + # We exclude server notices from here because we handle them in a + # dedicated module. We do not want to confuse developers this early + # in the spec. + "m.room.message#m.server_notice", + ] other_msgtypes = [ k for k in schemas.keys() if k.startswith("m.room.message#") and - k not in msgtype_order + k not in msgtype_order and k not in excluded_types ] for event_name in (msgtype_order + other_msgtypes): if not event_name.startswith("m.room.message#m."): diff --git a/scripts/templating/matrix_templates/templates/events.tmpl b/scripts/templating/matrix_templates/templates/events.tmpl index 0955cf4c..f55be73f 100644 --- a/scripts/templating/matrix_templates/templates/events.tmpl +++ b/scripts/templating/matrix_templates/templates/events.tmpl @@ -1,7 +1,14 @@ {% import 'tables.tmpl' as tables -%} +{% if (event.type_with_msgtype) %} +``{{event.type_with_msgtype}}`` +{{(4 + event.type_with_msgtype | length) * title_kind}} +{% endif -%} + +{% if (not event.type_with_msgtype) %} ``{{event.type}}`` {{(4 + event.type | length) * title_kind}} +{% endif -%} {% if (event.typeof | length) %} *{{event.typeof}}* diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index c1755119..466110f8 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -846,6 +846,7 @@ class MatrixUnits(Units): "title": None, "desc": None, "msgtype": None, + "type_with_msgtype": None, # for the template's sake "content_fields": [ # ] @@ -884,6 +885,7 @@ class MatrixUnits(Units): ) if msgtype: schema["msgtype"] = msgtype[0] # enum prop + schema["type_with_msgtype"] = schema["type"] + " (" + msgtype[0] + ")" # link to msgtypes for m.room.message if schema["type"] == "m.room.message" and not msgtype: diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 604c2b1c..a8bbfca0 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -219,9 +219,12 @@ Other error codes the client might encounter are: to modify state (eg: sending messages, account data, etc) and not routes which only read state (eg: ``/sync``, get account data, etc). +:``M_CANNOT_LEAVE_SERVER_NOTICE_ROOM``: + The user is unable to reject an invite to join the server notices room. See the + `Server Notices <#server-notices>`_ module for more information. + .. TODO: More error codes (covered by other issues) .. * M_CONSENT_NOT_GIVEN - GDPR: https://github.com/matrix-org/matrix-doc/issues/1512 -.. * M_CANNOT_LEAVE_SERVER_NOTICE_ROOM - GDPR: https://github.com/matrix-org/matrix-doc/issues/1254 .. _sect:txn_ids: diff --git a/specification/modules/server_notices.rst b/specification/modules/server_notices.rst new file mode 100644 index 00000000..63b7bfc5 --- /dev/null +++ b/specification/modules/server_notices.rst @@ -0,0 +1,78 @@ +.. Copyright 2019 The Matrix.org Foundation C.I.C. +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Server Notices +============== + +.. _module:server-notices: + +Homeserver hosts often want to send messages to users in an official capacity, +or have resource limits which affect a user's ability to use the homeserver. +For example, the homeserver may be limited to a certain number of active users +per month and has exceeded that limit. To communicate this failure to users, +the homeserver would use the Server Notices room. + +The aesthetics of the room (name, topic, avatar, etc) are left as an implementation +detail. It is recommended that the homeserver decorate the room such that it looks +like an official room to users. + +Events +------ +Notices are sent to the client as normal ``m.room.message`` events with a +``msgtype`` of ``m.server_notice`` in the server notices room. Events with +a ``m.server_notice`` ``msgtype`` outside of the server notice room must +be ignored by clients. + +The specified values for ``server_notice_type`` are: + +:``m.server_notice.usage_limit_reached``: + The server has exceeded some limit which requires the server administrator + to intervene. The ``limit_type`` describes the kind of limit reached. + The specified values for ``limit_type`` are: + + :``monthly_active_user``: + The server's number of active users in the last 30 days has exceeded the + maximum. New connections are being refused by the server. What defines + "active" is left as an implementation detail, however servers are encouraged + to treat syncing users as "active". + + +{{m_room_message_m_server_notice_event}} + +Client behaviour +---------------- +Clients can identify the server notices room by the ``m.server_notice`` tag +on the room. Active notices are represented by the `pinned events <#m-room-pinned-events>`_ +in the server notices room. Server notice events pinned in that room should +be shown to the user through special UI and not through the normal pinned +events interface in the client. For example, clients may show warning banners +or bring up dialogs to get the user's attention. Events which are not server +notice events and are pinned in the server notices room should be shown just +like any other pinned event in a room. + +The client must not expect to be able to reject an invite to join the server +notices room. Attempting to reject the invite must result in a +``M_CANNOT_LEAVE_SERVER_NOTICE_ROOM`` error. Servers should not prevent the user +leaving the room after joining the server notices room, however the same error +code must be used if the server will prevent leaving the room. + +Server behaviour +---------------- +Servers should manage exactly 1 server notices room per user. Servers must +identify this room to clients with the ``m.server_notice`` tag. Servers should +invite the target user rather than automatically join them to the server notice +room. + +How servers send notices to clients, and which user they use to send the events, +is left as an implementation detail for the server. diff --git a/specification/targets.yaml b/specification/targets.yaml index 830449ae..aa53e9c3 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -83,6 +83,7 @@ groups: # reusable blobs of files when prefixed with 'group:' - modules/server_acls.rst - modules/mentions.rst - modules/room_upgrades.rst + - modules/server_notices.rst title_styles: ["=", "-", "~", "+", "^", "`", "@", ":"] From 1d33adf62d8d56d1bfb88a0ff13da7e00fb0cc60 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 21:20:04 -0600 Subject: [PATCH 055/128] Add rationale for UIA on change password, and how access tokens behave Fixes https://github.com/matrix-org/matrix-doc/issues/680 --- api/client-server/registration.yaml | 8 ++++++-- changelogs/client_server/newsfragments/2027.clarification | 1 + 2 files changed, 7 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2027.clarification diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index e2d35d2c..3195ab41 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -326,13 +326,17 @@ paths: description: |- Changes the password for an account on this homeserver. - This API endpoint uses the `User-Interactive Authentication API`_. + This API endpoint uses the `User-Interactive Authentication API`_ to + ensure the user changing the password is actually the owner of the + account. An access token should be submitted to this endpoint if the client has an active session. The homeserver may change the flows available depending on whether a - valid access token is provided. + valid access token is provided. The homeserver SHOULD NOT revoke the + access token provided in the request, however all other access tokens + for the user should be revoked if the request succeeds. security: - accessToken: [] operationId: changePassword diff --git a/changelogs/client_server/newsfragments/2027.clarification b/changelogs/client_server/newsfragments/2027.clarification new file mode 100644 index 00000000..db74ea56 --- /dev/null +++ b/changelogs/client_server/newsfragments/2027.clarification @@ -0,0 +1 @@ +Clarify why User Interactive Auth is used on password changes and how access tokens are handled. From 221d9f24fd5699c78072feeab335c9a389a4de41 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 21:30:55 -0600 Subject: [PATCH 056/128] Clarify that logging out deletes devices too Fixes https://github.com/matrix-org/matrix-doc/issues/1651 --- api/client-server/logout.yaml | 5 +++-- changelogs/client_server/newsfragments/2028.clarification | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2028.clarification diff --git a/api/client-server/logout.yaml b/api/client-server/logout.yaml index 2dfd6d97..8451f739 100644 --- a/api/client-server/logout.yaml +++ b/api/client-server/logout.yaml @@ -32,7 +32,7 @@ paths: summary: Invalidates a user access token description: |- Invalidates an existing access token, so that it can no longer be used for - authorization. + authorization. The device associated with the access token is also deleted. operationId: logout security: - accessToken: [] @@ -49,7 +49,8 @@ paths: summary: Invalidates all access tokens for a user description: |- Invalidates all access tokens for a user, so that they can no longer be used for - authorization. This includes the access token that made this request. + authorization. This includes the access token that made this request. All devices + for the user are also deleted. This endpoint does not require UI authorization because UI authorization is designed to protect against attacks where the someone gets hold of a single access diff --git a/changelogs/client_server/newsfragments/2028.clarification b/changelogs/client_server/newsfragments/2028.clarification new file mode 100644 index 00000000..75e21e74 --- /dev/null +++ b/changelogs/client_server/newsfragments/2028.clarification @@ -0,0 +1 @@ +Clarify that devices are deleted upon logout. From d52fcdacfa7a98e6ab56916dd7d331e53d94ab77 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 21:36:31 -0600 Subject: [PATCH 057/128] Add M_NOT_FOUND definition for deleting non-existent aliases Fixes https://github.com/matrix-org/matrix-doc/issues/1675 --- api/client-server/directory.yaml | 9 +++++++++ .../client_server/newsfragments/2029.clarification | 1 + 2 files changed, 10 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2029.clarification diff --git a/api/client-server/directory.yaml b/api/client-server/directory.yaml index 78ddfa29..2c7c8386 100644 --- a/api/client-server/directory.yaml +++ b/api/client-server/directory.yaml @@ -148,5 +148,14 @@ paths: } schema: type: object + 404: + description: There is no mapped room ID for this room alias. + examples: + application/json: { + "errcode": "M_NOT_FOUND", + "error": "Room alias #monkeys:example.org not found." + } + schema: + "$ref": "definitions/errors/error.yaml" tags: - Room directory diff --git a/changelogs/client_server/newsfragments/2029.clarification b/changelogs/client_server/newsfragments/2029.clarification new file mode 100644 index 00000000..95b65481 --- /dev/null +++ b/changelogs/client_server/newsfragments/2029.clarification @@ -0,0 +1 @@ +Add ``M_NOT_FOUND`` error definition for deleting room aliases. From a30dbc590da00727e71920af0df7f5e140783aba Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 21:39:09 -0600 Subject: [PATCH 058/128] Clarify that e2e keys are also obliterated --- api/client-server/logout.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/api/client-server/logout.yaml b/api/client-server/logout.yaml index 8451f739..75a3ec87 100644 --- a/api/client-server/logout.yaml +++ b/api/client-server/logout.yaml @@ -33,6 +33,7 @@ paths: description: |- Invalidates an existing access token, so that it can no longer be used for authorization. The device associated with the access token is also deleted. + `Device keys <#device-keys>`_ for the device are deleted alongside the device. operationId: logout security: - accessToken: [] @@ -50,7 +51,8 @@ paths: description: |- Invalidates all access tokens for a user, so that they can no longer be used for authorization. This includes the access token that made this request. All devices - for the user are also deleted. + for the user are also deleted. `Device keys <#device-keys>`_ for the device are + deleted alongside the device. This endpoint does not require UI authorization because UI authorization is designed to protect against attacks where the someone gets hold of a single access From 1bda3fe2b20427be2e129ac2a91d7dc9bbd72215 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 21:49:46 -0600 Subject: [PATCH 059/128] Spec MSISDN UIA support Fixes https://github.com/matrix-org/matrix-doc/issues/1702 1702 describes the lack of `bind_msisdn` parameter, however the whole login type was missing from UIA. --- api/client-server/registration.yaml | 6 ++++ .../client_server/newsfragments/2030.feature | 1 + specification/client_server_api.rst | 29 +++++++++++++++++++ 3 files changed, 36 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2030.feature diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index e2d35d2c..741008bb 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -94,6 +94,12 @@ paths: If true, the server binds the email used for authentication to the Matrix ID with the identity server. example: false + bind_msisdn: + type: boolean + description: |- + If true, the server binds the phone number used for authentication + to the Matrix ID with the identity server. + example: false username: type: string description: |- diff --git a/changelogs/client_server/newsfragments/2030.feature b/changelogs/client_server/newsfragments/2030.feature new file mode 100644 index 00000000..b5975a73 --- /dev/null +++ b/changelogs/client_server/newsfragments/2030.feature @@ -0,0 +1 @@ +Add MSISDN (phone number) support to User-Interactive Authentication. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 604c2b1c..a40d463c 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -618,6 +618,7 @@ This specification defines the following auth types: - ``m.login.recaptcha`` - ``m.login.oauth2`` - ``m.login.email.identity`` + - ``m.login.msisdn`` - ``m.login.token`` - ``m.login.dummy`` @@ -782,6 +783,34 @@ To use this authentication type, clients should submit an auth dict as follows: "session": "" } +Phone number/MSISDN-based (identity server) +<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< +:Type: + ``m.login.msisdn`` +:Description: + Authentication is supported by authorising a phone number with an identity + server. + +Prior to submitting this, the client should authenticate with an identity +server. After authenticating, the session information should be submitted to +the homeserver. + +To use this authentication type, clients should submit an auth dict as follows: + +.. code:: json + + { + "type": "m.login.msisdn", + "threepidCreds": [ + { + "sid": "", + "client_secret": "", + "id_server": "" + } + ], + "session": "" + } + Dummy Auth <<<<<<<<<< :Type: From 00f97636a22b28de26ffaafd674e1014374210e0 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 22:08:04 -0600 Subject: [PATCH 060/128] Add missing reason property to m.call.hangup Fixes https://github.com/matrix-org/matrix-doc/issues/1689 --- .../client_server/newsfragments/2031.clarification | 1 + event-schemas/schema/m.call.hangup | 10 +++++++++- 2 files changed, 10 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/2031.clarification diff --git a/changelogs/client_server/newsfragments/2031.clarification b/changelogs/client_server/newsfragments/2031.clarification new file mode 100644 index 00000000..9bed3bcc --- /dev/null +++ b/changelogs/client_server/newsfragments/2031.clarification @@ -0,0 +1 @@ +Add missing ``reason`` to ``m.call.hangup``. diff --git a/event-schemas/schema/m.call.hangup b/event-schemas/schema/m.call.hangup index 9d45d179..0328e03a 100644 --- a/event-schemas/schema/m.call.hangup +++ b/event-schemas/schema/m.call.hangup @@ -1,6 +1,6 @@ { "type": "object", - "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.", + "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call. The ``reason`` for the hangup is expected to be provided when there was an error in the call negotiation, such as ``ice_failed`` for when ICE negotiation fails or ``invite_timeout`` for when the other party did not answer in time.", "allOf": [{ "$ref": "core-event-schema/room_event.yaml" }], @@ -15,6 +15,14 @@ "version": { "type": "integer", "description": "The version of the VoIP specification this message adheres to. This specification is version 0." + }, + "reason": { + "type": "string", + "description": "Optional error reason for the hangup. This should not be provided when the user naturally ends or rejects the call.", + "enum": [ + "ice_failed", + "invite_timeout" + ] } }, "required": ["call_id", "version"] From 23ab1c527a4864f632ce07d80d80fa23eae81ed6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 26 May 2019 22:16:12 -0600 Subject: [PATCH 061/128] Clarify how redactions affect room state Fixes https://github.com/matrix-org/matrix-doc/issues/1726 --- .../client_server/newsfragments/2032.clarification | 1 + specification/client_server_api.rst | 9 +++++++++ 2 files changed, 10 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2032.clarification diff --git a/changelogs/client_server/newsfragments/2032.clarification b/changelogs/client_server/newsfragments/2032.clarification new file mode 100644 index 00000000..e497b8be --- /dev/null +++ b/changelogs/client_server/newsfragments/2032.clarification @@ -0,0 +1 @@ +Clarify how redactions affect room state. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 604c2b1c..a254a85d 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1531,6 +1531,15 @@ property of the redacted event, under the ``redacted_because`` key. When a client receives a redaction event it should change the redacted event in the same way a server does. +.. NOTE:: + + Redacted events can still affect the state of the room. For example, a + redacted ``join`` event will still result in the user being considered + joined. Similarly, a redacted topic does not necessarily cause the topic + to revert to what is was prior to the event - it causes the topic to be + removed from the room. + + Events ++++++ From d14dc1d8e15a12b91cb20592d6e46c6b20ea678a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 21:15:24 -0600 Subject: [PATCH 062/128] Clarify that redacted state events affect the room with default values --- specification/client_server_api.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index a254a85d..22cc9425 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1533,11 +1533,12 @@ same way a server does. .. NOTE:: - Redacted events can still affect the state of the room. For example, a - redacted ``join`` event will still result in the user being considered - joined. Similarly, a redacted topic does not necessarily cause the topic - to revert to what is was prior to the event - it causes the topic to be - removed from the room. + Redacted events can still affect the state of the room. When redacted, + state events behave as though their properties were simply not specified + unless their properties are protected by the redaction algorithm. For example, + a redacted ``join`` event will still result in the user being considered joined. + Similarly, a redacted topic does not necessarily cause the topic to revert to + what is was prior to the event - it causes the topic to be removed from the room. Events From 229893589a30f21560cf55fac00d0112fa041413 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 21:18:04 -0600 Subject: [PATCH 063/128] Move wording for reason definitions --- event-schemas/schema/m.call.hangup | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/event-schemas/schema/m.call.hangup b/event-schemas/schema/m.call.hangup index 0328e03a..c0478f5a 100644 --- a/event-schemas/schema/m.call.hangup +++ b/event-schemas/schema/m.call.hangup @@ -1,6 +1,6 @@ { "type": "object", - "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call. The ``reason`` for the hangup is expected to be provided when there was an error in the call negotiation, such as ``ice_failed`` for when ICE negotiation fails or ``invite_timeout`` for when the other party did not answer in time.", + "description": "Sent by either party to signal their termination of the call. This can be sent either once the call has has been established or before to abort the call.", "allOf": [{ "$ref": "core-event-schema/room_event.yaml" }], @@ -18,7 +18,7 @@ }, "reason": { "type": "string", - "description": "Optional error reason for the hangup. This should not be provided when the user naturally ends or rejects the call.", + "description": "Optional error reason for the hangup. This should not be provided when the user naturally ends or rejects the call. When there was an error in the call negotiation, this should be ``ice_failed`` for when ICE negotiation fails or ``invite_timeout`` for when the other party did not answer in time.", "enum": [ "ice_failed", "invite_timeout" From ade346f8cc173a532a0e991b61ed153a3ea93e9c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 21:19:58 -0600 Subject: [PATCH 064/128] Add m.server_notice to reserved tags --- specification/modules/tags.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/specification/modules/tags.rst b/specification/modules/tags.rst index a4b0becf..9d0cd554 100644 --- a/specification/modules/tags.rst +++ b/specification/modules/tags.rst @@ -34,9 +34,9 @@ The JSON object associated with each tag gives information about the tag, e.g ho to order the rooms with a given tag. Ordering information is given under the ``order`` key as a number between 0 and -1. The numbers are compared such that 0 is displayed first. Therefore a room -with an ``order`` of ``0.2`` would be displayed before a room with an ``order`` -of ``0.7``. If a room has a tag without an ``order`` key then it should appear +1. The numbers are compared such that 0 is displayed first. Therefore a room +with an ``order`` of ``0.2`` would be displayed before a room with an ``order`` +of ``0.7``. If a room has a tag without an ``order`` key then it should appear after the rooms with that tag that have an ``order`` key. The name of a tag MUST NOT exceed 255 bytes. @@ -60,6 +60,7 @@ The following tags are defined in the ``m.*`` namespace: * ``m.favourite``: The user's favourite rooms. These should be shown with higher precedence than other rooms. * ``m.lowpriority``: These should be shown with lower precedence than others. +* ``m.server_notice``: Used to identify `Server Notice Rooms <#module-server-notices>`_. {{m_tag_event}} From 9acd960cf6afc8ddb011f7db5ca00fbf57cba5a5 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 21:22:07 -0600 Subject: [PATCH 065/128] Update specification/client_server_api.rst Co-Authored-By: Kitsune Ral --- specification/client_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 22cc9425..c18f4d03 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1535,7 +1535,7 @@ same way a server does. Redacted events can still affect the state of the room. When redacted, state events behave as though their properties were simply not specified - unless their properties are protected by the redaction algorithm. For example, + except those protected by the redaction algorithm. For example, a redacted ``join`` event will still result in the user being considered joined. Similarly, a redacted topic does not necessarily cause the topic to revert to what is was prior to the event - it causes the topic to be removed from the room. From 699cafe6701d2206160170c6eaafeb789405507a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 21:27:54 -0600 Subject: [PATCH 066/128] v4 is v4, not v3 --- api/server-server/definitions/pdu_v4.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/server-server/definitions/pdu_v4.yaml b/api/server-server/definitions/pdu_v4.yaml index 55b1c26f..a045e657 100644 --- a/api/server-server/definitions/pdu_v4.yaml +++ b/api/server-server/definitions/pdu_v4.yaml @@ -13,7 +13,7 @@ # limitations under the License. type: object title: Persistent Data Unit -description: A persistent data unit (event) for room version 3 and beyond. +description: A persistent data unit (event) for room version 4 and beyond. example: $ref: "../examples/pdu_v4.json" allOf: From 10648aa9e8ce9556d3b881539acca7c9a7962156 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 22:30:37 -0600 Subject: [PATCH 067/128] Clarify that FAIL_ERROR is not limited to just homeservers. Fixes https://github.com/matrix-org/matrix-doc/issues/1735 --- changelogs/client_server/newsfragments/2036.clarification | 1 + specification/client_server_api.rst | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/2036.clarification diff --git a/changelogs/client_server/newsfragments/2036.clarification b/changelogs/client_server/newsfragments/2036.clarification new file mode 100644 index 00000000..96058b7b --- /dev/null +++ b/changelogs/client_server/newsfragments/2036.clarification @@ -0,0 +1 @@ +Clarify that ``FAIL_ERROR`` in autodiscovery is not limited to just homeservers. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index a8bbfca0..4158bad8 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -289,7 +289,7 @@ In this section, the following terms are used with specific meanings: ``FAIL_ERROR`` Inform the user that auto-discovery did not return any usable URLs. Do not continue further with the current login process. At this point, valid data - was obtained, but no homeserver is available to serve the client. No further + was obtained, but no server is available to serve the client. No further guess should be attempted and the user should make a conscientious decision what to do next. From d0fd20fdb4b412c9bb596cfc2b06915d46c631a3 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 22:40:07 -0600 Subject: [PATCH 068/128] Clarify how homeservers are meant to auth themselves to appservices Fixes https://github.com/matrix-org/matrix-doc/issues/1765 Note that the swagger definitions already say that authorization is required. It just wasn't mentioned in the spec. --- .../application_service/newsfragments/2037.clarification | 1 + specification/application_service_api.rst | 8 ++++++++ 2 files changed, 9 insertions(+) create mode 100644 changelogs/application_service/newsfragments/2037.clarification diff --git a/changelogs/application_service/newsfragments/2037.clarification b/changelogs/application_service/newsfragments/2037.clarification new file mode 100644 index 00000000..f425b1c1 --- /dev/null +++ b/changelogs/application_service/newsfragments/2037.clarification @@ -0,0 +1 @@ +Add missing definition for how appservices verify requests came from a homeserver. diff --git a/specification/application_service_api.rst b/specification/application_service_api.rst index 865544dd..81077bcf 100644 --- a/specification/application_service_api.rst +++ b/specification/application_service_api.rst @@ -187,6 +187,14 @@ An example registration file for an IRC-bridging application service is below: Homeserver -> Application Service API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Authorization ++++++++++++++ + +Homeservers MUST include a query parameter named ``access_token`` containing the +``hs_token`` from the application service's registration when making requests to +the application service. Application services MUST verify the provided ``access_token`` +matches their known ``hs_token``, failing the request with a ``M_FORBIDDEN`` error. + Legacy routes +++++++++++++ From d8eb2949066c9c53f7501da9fb86f76fa762e960 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 22:44:24 -0600 Subject: [PATCH 069/128] Fix s2s changelog entries --- changelogs/server_server/newsfragments/1904.clarification | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/server_server/newsfragments/1904.clarification b/changelogs/server_server/newsfragments/1904.clarification index 94174ebd..531fdb94 100644 --- a/changelogs/server_server/newsfragments/1904.clarification +++ b/changelogs/server_server/newsfragments/1904.clarification @@ -1 +1 @@ -Fix the `access_token` parameter in the open_id endpoint. +Fix the ``access_token`` parameter in the open_id endpoint. From 88d47fd57e423c3e2e958fc547f7ef4812eb3ca2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 22:51:52 -0600 Subject: [PATCH 070/128] Rename changelog entry to point to PR We use the PR for every other entry, so here is no exception. --- .../newsfragments/{1904.clarification => 1906.clarification} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changelogs/server_server/newsfragments/{1904.clarification => 1906.clarification} (100%) diff --git a/changelogs/server_server/newsfragments/1904.clarification b/changelogs/server_server/newsfragments/1906.clarification similarity index 100% rename from changelogs/server_server/newsfragments/1904.clarification rename to changelogs/server_server/newsfragments/1906.clarification From 62890d21b25016dc008a5cc2e2f993ee3a0873ab Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 23:08:02 -0600 Subject: [PATCH 071/128] Fix example Content-Type for /media/upload request Fixes https://github.com/matrix-org/matrix-doc/issues/1770 --- api/client-server/content-repo.yaml | 11 ++++++----- .../client_server/newsfragments/2041.clarification | 1 + scripts/templating/matrix_templates/units.py | 9 +++++++-- 3 files changed, 14 insertions(+), 7 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2041.clarification diff --git a/api/client-server/content-repo.yaml b/api/client-server/content-repo.yaml index 07df18fe..4460bb69 100644 --- a/api/client-server/content-repo.yaml +++ b/api/client-server/content-repo.yaml @@ -41,7 +41,7 @@ paths: name: Content-Type type: string description: The content type of the file being uploaded - x-example: "Content-Type: audio/mpeg" + x-example: "Content-Type: application/pdf" - in: query type: string x-example: "War and Peace.pdf" @@ -51,6 +51,7 @@ paths: name: "" description: The content to be uploaded. required: true + x-example: "" # so the spec shows "" without quotes. schema: type: string example: "" @@ -103,7 +104,7 @@ paths: default: true description: | Indicates to the server that it should not attempt to fetch the media if it is deemed - remote. This is to prevent routing loops where the server contacts itself. Defaults to + remote. This is to prevent routing loops where the server contacts itself. Defaults to true if not provided. responses: 200: @@ -158,7 +159,7 @@ paths: default: true description: | Indicates to the server that it should not attempt to fetch the media if it is deemed - remote. This is to prevent routing loops where the server contacts itself. Defaults to + remote. This is to prevent routing loops where the server contacts itself. Defaults to true if not provided. responses: 200: @@ -228,7 +229,7 @@ paths: default: true description: | Indicates to the server that it should not attempt to fetch the media if it is deemed - remote. This is to prevent routing loops where the server contacts itself. Defaults to + remote. This is to prevent routing loops where the server contacts itself. Defaults to true if not provided. responses: 200: @@ -330,7 +331,7 @@ paths: m.upload.size: type: integer format: int64 - description: |- + description: |- The maximum size an upload can be in bytes. Clients SHOULD use this as a guide when uploading content. If not listed or null, the size limit should be treated as unknown. diff --git a/changelogs/client_server/newsfragments/2041.clarification b/changelogs/client_server/newsfragments/2041.clarification new file mode 100644 index 00000000..39bbddb5 --- /dev/null +++ b/changelogs/client_server/newsfragments/2041.clarification @@ -0,0 +1 @@ +Fix example ``Content-Type`` for ``/media/upload`` request. diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 466110f8..d4745e4d 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -521,6 +521,7 @@ class MatrixUnits(Units): path_template = path example_query_params = [] example_body = "" + example_mime = "application/json" for param in endpoint_swagger.get("parameters", []): # even body params should have names, otherwise the active docs don't work. param_name = param["name"] @@ -533,6 +534,10 @@ class MatrixUnits(Units): example_body = get_example_for_param(param) continue + if param_loc == "header": + if param["name"] == "Content-Type" and param["x-example"]: + example_mime = param["x-example"] + # description desc = param.get("description", "") if param.get("required"): @@ -610,8 +615,8 @@ class MatrixUnits(Units): example_query_params) if example_body: endpoint["example"][ - "req"] = "%s %s%s HTTP/1.1\nContent-Type: application/json\n\n%s" % ( - method.upper(), path_template, query_string, example_body + "req"] = "%s %s%s HTTP/1.1\nContent-Type: %s\n\n%s" % ( + method.upper(), path_template, query_string, example_mime, example_body ) else: endpoint["example"]["req"] = "%s %s%s HTTP/1.1\n\n" % ( From 572a6056ad64c5f3f654f91a27cf8a42c19d6ee5 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 23:15:02 -0600 Subject: [PATCH 072/128] Clarify that login flows must be completed in order Fixes https://github.com/matrix-org/matrix-doc/issues/1134 Evidence of this being the case is shown here: https://github.com/matrix-org/synapse/pull/5174 --- changelogs/client_server/newsfragments/2042.clarification | 1 + specification/client_server_api.rst | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2042.clarification diff --git a/changelogs/client_server/newsfragments/2042.clarification b/changelogs/client_server/newsfragments/2042.clarification new file mode 100644 index 00000000..4e17b99f --- /dev/null +++ b/changelogs/client_server/newsfragments/2042.clarification @@ -0,0 +1 @@ +Clarify that login flows are meant to be completed in order. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index a8bbfca0..4b7065b3 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -406,8 +406,9 @@ an additional stage. This exchange continues until the final success. For each endpoint, a server offers one or more 'flows' that the client can use to authenticate itself. Each flow comprises a series of stages, as described -above. The client is free to choose which flow it follows. When all stages in a -flow are complete, authentication is complete and the API call succeeds. +above. The client is free to choose which flow it follows, however the flow's +stages must be completed in order. When all stages in a flow are complete, +authentication is complete and the API call succeeds. User-interactive API in the REST API <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< From 4e58414b26d9d2bc145393091b7e141574c36450 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 23:20:09 -0600 Subject: [PATCH 073/128] Clarify that clients should not ack their own messages Fixes https://github.com/matrix-org/matrix-doc/issues/567 --- changelogs/client_server/newsfragments/2043.clarification | 1 + specification/modules/receipts.rst | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2043.clarification diff --git a/changelogs/client_server/newsfragments/2043.clarification b/changelogs/client_server/newsfragments/2043.clarification new file mode 100644 index 00000000..9bb975fa --- /dev/null +++ b/changelogs/client_server/newsfragments/2043.clarification @@ -0,0 +1 @@ +Clarify that clients should not send read receipts for their own messages. diff --git a/specification/modules/receipts.rst b/specification/modules/receipts.rst index faba7b62..ee2b697a 100644 --- a/specification/modules/receipts.rst +++ b/specification/modules/receipts.rst @@ -60,7 +60,8 @@ Clients should send read receipts when there is some certainty that the event in question has been **displayed** to the user. Simply receiving an event does not provide enough certainty that the user has seen the event. The user SHOULD need to *take some action* such as viewing the room that the event was sent to or -dismissing a notification in order for the event to count as "read". +dismissing a notification in order for the event to count as "read". Clients +SHOULD NOT send read receipts for events sent by their own user. A client can update the markers for its user by interacting with the following HTTP APIs. @@ -94,4 +95,3 @@ Security considerations As receipts are sent outside the context of the event graph, there are no integrity checks performed on the contents of ``m.receipt`` events. - From bf86b4b83ca9ee97ef8bc7edcc17a663996bcffd Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 27 May 2019 23:42:08 -0600 Subject: [PATCH 074/128] Fix incorrect PR reference on changelog --- .../client_server/newsfragments/{1903.feature => 1908.feature} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changelogs/client_server/newsfragments/{1903.feature => 1908.feature} (100%) diff --git a/changelogs/client_server/newsfragments/1903.feature b/changelogs/client_server/newsfragments/1908.feature similarity index 100% rename from changelogs/client_server/newsfragments/1903.feature rename to changelogs/client_server/newsfragments/1908.feature From 0580f5120693e2535920f5e57e4218e9467b7d18 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 28 May 2019 12:49:50 -0600 Subject: [PATCH 075/128] Clarify that failing to follow the flows == 401 --- specification/client_server_api.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 4b7065b3..3ea1411e 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -407,8 +407,9 @@ an additional stage. This exchange continues until the final success. For each endpoint, a server offers one or more 'flows' that the client can use to authenticate itself. Each flow comprises a series of stages, as described above. The client is free to choose which flow it follows, however the flow's -stages must be completed in order. When all stages in a flow are complete, -authentication is complete and the API call succeeds. +stages must be completed in order. Failing to follow the flows in order must +result in an HTTP 401 response, as defined below. When all stages in a flow +are complete, authentication is complete and the API call succeeds. User-interactive API in the REST API <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< From 8151aa331f6b553ffa755f38aaffb5f5b466b07a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 28 May 2019 12:51:36 -0600 Subject: [PATCH 076/128] Update specification/client_server_api.rst Co-Authored-By: Hubert Chathi --- specification/client_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index c18f4d03..a8246578 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1534,7 +1534,7 @@ same way a server does. .. NOTE:: Redacted events can still affect the state of the room. When redacted, - state events behave as though their properties were simply not specified + state events behave as though their properties were simply not specified, except those protected by the redaction algorithm. For example, a redacted ``join`` event will still result in the user being considered joined. Similarly, a redacted topic does not necessarily cause the topic to revert to From e2da3728a0631f95bae08f7ceb4a18b695841d3b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 28 May 2019 12:53:48 -0600 Subject: [PATCH 077/128] Only error if the token doesn't match --- specification/application_service_api.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specification/application_service_api.rst b/specification/application_service_api.rst index 81077bcf..3220df2d 100644 --- a/specification/application_service_api.rst +++ b/specification/application_service_api.rst @@ -193,7 +193,8 @@ Authorization Homeservers MUST include a query parameter named ``access_token`` containing the ``hs_token`` from the application service's registration when making requests to the application service. Application services MUST verify the provided ``access_token`` -matches their known ``hs_token``, failing the request with a ``M_FORBIDDEN`` error. +matches their known ``hs_token``, failing the request with a ``M_FORBIDDEN`` error +if it does not match. Legacy routes +++++++++++++ From 0463084924543ee5b024adb62ac32ee5339855c5 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 28 May 2019 14:19:56 -0600 Subject: [PATCH 078/128] Spec 3PID unbind API As per [MSC1915](https://github.com/matrix-org/matrix-doc/pull/1915) Implementation proof: * https://github.com/matrix-org/synapse/pull/4982 * https://github.com/matrix-org/sydent/pull/160 The only alteration made which differs from the proposal is clarity on how to handle homeservers not knowing the `id_server`. All other differences are unintentional. --- api/client-server/administrative_contact.yaml | 27 +++++- api/client-server/registration.yaml | 30 ++++++- api/identity/associations.yaml | 83 +++++++++++++++++++ .../client_server/newsfragments/2046.feature | 1 + .../identity_service/newsfragments/2046.new | 1 + .../1915-unbind-identity-server-param.md | 4 +- 6 files changed, 142 insertions(+), 4 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2046.feature create mode 100644 changelogs/identity_service/newsfragments/2046.new diff --git a/api/client-server/administrative_contact.yaml b/api/client-server/administrative_contact.yaml index 33ea9786..5cf01805 100644 --- a/api/client-server/administrative_contact.yaml +++ b/api/client-server/administrative_contact.yaml @@ -163,6 +163,14 @@ paths: schema: type: object properties: + id_server: + type: string + description: |- + The identity server to unbind from. If not provided, the homeserver + MUST use the ``id_server`` the identifier was added through. If the + homeserver does not know the original ``id_server``, it MUST return + a ``id_server_unbind_result`` of ``no-support``. + example: "example.org" medium: type: string description: The medium of the third party identifier being removed. @@ -180,7 +188,24 @@ paths: user. schema: type: object - properties: {} + properties: + id_server_unbind_result: + type: string + enum: + # XXX: I don't know why, but the order matters here so that "no-support" + # doesn't become "no- support" by the renderer. + - "no-support" + - "success" + description: |- + An indicator as to whether or not the homeserver was able to unbind + the 3PID from the identity server. ``success`` indicates that the + indentity server has unbound the identifier whereas ``no-support`` + indicates that the identity server refuses to support the request + or the homeserver was not able to determine an identity server to + unbind from. + example: "success" + required: + - id_server_unbind_result tags: - User data "/account/3pid/email/requestToken": diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index 3195ab41..f003cf1f 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -524,13 +524,39 @@ paths: description: |- Additional authentication information for the user-interactive authentication API. "$ref": "definitions/auth_data.yaml" + id_server: + type: string + description: |- + The identity server to unbind all of the user's 3PIDs from. + If not provided, the homeserver MUST use the ``id_server`` + that was originally use to bind each identifier. If the + homeserver does not know which ``id_server`` that was, + it must return an ``id_server_unbind_result`` of + ``no-support``. + example: "example.org" responses: 200: description: The account has been deactivated. - examples: - application/json: {} schema: type: object + properties: + id_server_unbind_result: + type: string + enum: + - "success" + - "no-support" + description: |- + An indicator as to whether or not the homeserver was able to unbind + the user's 3PIDs from the identity server(s). ``success`` indicates + that all identifiers have been unbound from the identity server while + ``no-support`` indicates that one or more identifiers failed to unbind + due to the identity server refusing the request or the homeserver + being unable to determine an identity server to unbind from. This + must be ``success`` if the homeserver has no identifiers to unbind + for the user. + example: "success" + required: + - id_server_unbind_result 401: description: |- The homeserver requires additional authentication information. diff --git a/api/identity/associations.yaml b/api/identity/associations.yaml index 152a0a9b..f44fe3cc 100644 --- a/api/identity/associations.yaml +++ b/api/identity/associations.yaml @@ -201,3 +201,86 @@ paths: } schema: $ref: "../client-server/definitions/errors/error.yaml" + "/3pid/unbind": + post: + summary: Remove an association between a session and a Matrix user ID. + description: |- + Remove an association between a session and a Matrix user ID. + + Future calls to ``/lookup`` for any of the session's 3pids will not + return the removed association. + + The identity server should authenticate the request in one of two + ways: + + 1. The request is signed by the homeserver which controls the ``user_id``. + 2. The request includes the ``sid`` and ``client_secret`` parameters, + as per ``/3pid/bind``, which proves ownership of the 3PID. + + If this endpoint returns a JSON Matrix error, that error should be passed + through to the client requesting an unbind through a homeserver, if the + homeserver is acting on behalf of a client. + operationId: unbind + parameters: + - in: body + name: body + schema: + type: object + example: { + "sid": "1234", + "client_secret": "monkeys_are_GREAT", + "mxid": "@ears:example.org", + "threepid": { + "medium": "email", + "address": "monkeys_have_ears@example.org" + } + } + properties: + sid: + type: string + description: The Session ID generated by the ``requestToken`` call. + client_secret: + type: string + description: The client secret passed to the ``requestToken`` call. + mxid: + type: string + description: The Matrix user ID to remove from the 3pids. + threepid: + type: object + title: 3PID + description: |- + The 3PID to remove. Must match the 3PID used to generate the session + if using ``sid`` and ``client_secret`` to authenticate this request. + properties: + medium: + type: string + description: |- + A medium from the `3PID Types`_ Appendix, matching the medium + of the identifier to unbind. + address: + type: string + description: The 3PID address to remove. + required: ['medium', 'address'] + required: ["threepid", "mxid"] + responses: + 200: + description: The association was successfully removed. + examples: + application/json: {} + schema: + type: object + 400: + description: |- + If the response body is not a JSON Matrix error, the identity server + does not support unbinds. If a JSON Matrix error is in the response + body, the requesting party should respect the error. + 404: + description: |- + If the response body is not a JSON Matrix error, the identity server + does not support unbinds. If a JSON Matrix error is in the response + body, the requesting party should respect the error. + 501: + description: |- + If the response body is not a JSON Matrix error, the identity server + does not support unbinds. If a JSON Matrix error is in the response + body, the requesting party should respect the error. diff --git a/changelogs/client_server/newsfragments/2046.feature b/changelogs/client_server/newsfragments/2046.feature new file mode 100644 index 00000000..e54df535 --- /dev/null +++ b/changelogs/client_server/newsfragments/2046.feature @@ -0,0 +1 @@ +Add ``id_server`` to ``/deactivate`` and ``/3pid/delete`` endpoints to unbind from a specific identity server. diff --git a/changelogs/identity_service/newsfragments/2046.new b/changelogs/identity_service/newsfragments/2046.new new file mode 100644 index 00000000..7146799b --- /dev/null +++ b/changelogs/identity_service/newsfragments/2046.new @@ -0,0 +1 @@ +Add ``/3pid/unbind`` for removing 3PIDs. diff --git a/proposals/1915-unbind-identity-server-param.md b/proposals/1915-unbind-identity-server-param.md index 6817ece3..5b7a1a50 100644 --- a/proposals/1915-unbind-identity-server-param.md +++ b/proposals/1915-unbind-identity-server-param.md @@ -27,7 +27,9 @@ known by the homeserver). The 200 response is a JSON object with an `id_server_unbind_result` field whose value is either `success` or `no-support`, where the latter indicates that the identity server (IS) does not support unbinding 3PIDs directly. If the identity -server returns an error then that should be returned to the client. +server returns an error then that should be returned to the client. If the homeserver +is unable to determine an `id_server` to use, it should return `no-support` for +the `id_server_unbind_result`. Example: From 84f0d9d7e4467460a9caf9128cd739ddd2fb03b6 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 28 May 2019 20:07:45 -0400 Subject: [PATCH 079/128] add clarification --- proposals/1717-key_verification.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/1717-key_verification.md b/proposals/1717-key_verification.md index 6c9e2341..429e3a97 100644 --- a/proposals/1717-key_verification.md +++ b/proposals/1717-key_verification.md @@ -139,7 +139,8 @@ Properties: device, an `m.key.verification.cancel` message with `code` set to `m.accepted` is sent to the other devices - `reason` (string): human-readable reason for cancelling. This should only be - used if the recieving client does not understand the code given. + used if the recieving client does not understand the code given in the `code` + property. Verification methods may define their own additional cancellation codes. Cancellation codes defined in the Matrix spec will begin with `m.`; other From 6706d772c91be7c4ab5c8089f4579695e461ffd4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 00:46:00 -0600 Subject: [PATCH 080/128] Fix test vectors with invalid JSON and signature Fixes https://github.com/matrix-org/matrix-doc/issues/2023 The content hashes appear correct, however applying the algorithm defined in the spec never resulted in the signatures previously demonstrated. --- specification/appendices/test_vectors.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/appendices/test_vectors.rst b/specification/appendices/test_vectors.rst index e2b8fb58..7759fa88 100644 --- a/specification/appendices/test_vectors.rst +++ b/specification/appendices/test_vectors.rst @@ -114,7 +114,7 @@ The event signing algorithm should emit the following signed event: "origin_server_ts": 1000000, "signatures": { "domain": { - "ed25519:1": "2Wptgo4CwmLo/Y8B8qinxApKaCkBG2fjTWB7AbP5Uy+aIbygsSdLOFzvdDjww8zUVKCmI02eP9xtyJxc/cLiBA" + "ed25519:1": "JV2dlZUASAefSdywnyCxzykHlyr7xkKGK7IRir1cF8eYsnONrCSb+GRn7aXXstr1UHKvzYjRXPx0001+boD1Ag" } }, "type": "X", @@ -129,7 +129,7 @@ Given the following event containing redactable content: { "content": { - "body": "Here is the message content", + "body": "Here is the message content" }, "event_id": "$0:domain", "origin": "domain", @@ -149,7 +149,7 @@ The event signing algorithm should emit the following signed event: { "content": { - "body": "Here is the message content", + "body": "Here is the message content" }, "event_id": "$0:domain", "hashes": { @@ -162,7 +162,7 @@ The event signing algorithm should emit the following signed event: "sender": "@u:domain", "signatures": { "domain": { - "ed25519:1": "Wm+VzmOUOz08Ds+0NTWb1d4CZrVsJSikkeRxh6aCcUwu6pNC78FunoD7KNWzqFn241eYHYMGCA5McEiVPdhzBA" + "ed25519:1": "4zc79tH2cU6Y+eg4YbbF7KiDOrnwEDjlhTqIKiH4k7L9zD9XCiomD7x9odL9eEwnyy1144QyMBe8O3HK++GHBg" } }, "unsigned": { From ffb70a2fabd46f5197cf123d6ed8bcd5d613566d Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 29 May 2019 15:01:35 -0400 Subject: [PATCH 081/128] fix typo Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- proposals/1719-olm_unwedging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index c181e3f7..5de6463f 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -19,7 +19,7 @@ from; the client should not create a new session with another device if it has already created one for that given device in the past 1 hour. Clients may wish to ask the sender of the undecryptable messages to re-send the -message. For exampe, if the undecryptable message was a megolm session, then +message. For example, if the undecryptable message was a megolm session, then the client can send an [`m.room_key_request`](https://matrix.org/docs/spec/client_server/r0.4.0.html#m-room-key-request) message to request that the sender re-send the key. From 6929579360113e4e653e884f7e4cb5bfa68fbbd7 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 29 May 2019 15:18:54 -0400 Subject: [PATCH 082/128] add some clarifications --- proposals/1719-olm_unwedging.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/proposals/1719-olm_unwedging.md b/proposals/1719-olm_unwedging.md index 5de6463f..dffdc8b1 100644 --- a/proposals/1719-olm_unwedging.md +++ b/proposals/1719-olm_unwedging.md @@ -1,8 +1,13 @@ # Olm unwedging Olm sessions sometimes get out of sync, resulting in undecryptable messages. -This proposal documents a method for devices to create a new session to replace -the broken session. +This can happen for several reasons. For example, if a user restores their +client state from a backup, the client will be using an old ratchet state +([riot-web#3822](https://github.com/vector-im/riot-web/issues/3822)). Or a +client might expire a one-time key that another client is trying to use +([riot-web#3309](https://github.com/vector-im/riot-web/issues/3309)). This +proposal documents a method for devices to create a new session to replace the +broken session. ## Proposal @@ -18,11 +23,11 @@ the number of new sessions it creates per device that it receives a message from; the client should not create a new session with another device if it has already created one for that given device in the past 1 hour. -Clients may wish to ask the sender of the undecryptable messages to re-send the -message. For example, if the undecryptable message was a megolm session, then -the client can send an -[`m.room_key_request`](https://matrix.org/docs/spec/client_server/r0.4.0.html#m-room-key-request) -message to request that the sender re-send the key. +Clients may wish to take steps to mitigate the loss of the undecryptable +messages. For example, megolm sessions that were sent using the old session +would have been lost, so the client can send +[`m.room_key_request`](https://matrix.org/docs/spec/client_server/latest.html#m-room-key-request) +messages to re-request any megolm sessions that it is unable to decrypt. The spec currently says, "If a client has multiple sessions established with another device, it should use the session from which it last received a From 15b8011f63a687bc8961c881c1dcb73fd4318c6b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 13:37:58 -0600 Subject: [PATCH 083/128] Add missing aesthetic parameters to /store-invite Fixes https://github.com/matrix-org/matrix-doc/issues/2048 --- api/identity/store_invite.yaml | 47 ++++++++++++++++--- .../newsfragments/2049.clarification | 1 + 2 files changed, 42 insertions(+), 6 deletions(-) create mode 100644 changelogs/identity_service/newsfragments/2049.clarification diff --git a/api/identity/store_invite.yaml b/api/identity/store_invite.yaml index 69103294..bca78d7e 100644 --- a/api/identity/store_invite.yaml +++ b/api/identity/store_invite.yaml @@ -50,31 +50,66 @@ paths: requests to ``/_matrix/identity/api/v1/pubkey/ephemeral/isvalid``. Currently, invites may only be issued for 3pids of the ``email`` medium. + + Optional fields in the request should be populated to the best of the + server's ability. Identity servers may use these variables when notifying + the ``address`` of the pending invite for display purposes. operationId: storeInvite parameters: - in: body name: body schema: type: object - example: { - "medium": "email", - "address": "foo@bar.baz", - "room_id": "!something:example.tld", - "sender": "@bob:example.com" - } properties: medium: type: string description: The literal string ``email``. + example: "email" address: type: string description: The email address of the invited user. + example: "foo@example.com" room_id: type: string description: The Matrix room ID to which the user is invited + example: "!something:example.org" sender: type: string description: The Matrix user ID of the inviting user + example: "@bob:example.com" + room_alias: + type: string + description: |- + The Matrix room alias for the room to which the user is + invited. This should be retrieved from the ``m.room.canonical_alias`` + state event. + example: "#somewhere:exmaple.org" + room_avatar_url: + type: string + description: |- + The Content URI for the room to which the user is invited. This should + be retrieved from the ``m.room.avatar`` state event. + example: "mxc://example.org/s0meM3dia" + room_join_rules: + type: string + description: |- + The ``join_rule`` for the room to which the user is invited. This should + be retrieved from the ``m.room.join_rules`` state event. + example: "public" + room_name: + type: string + description: |- + The name of the room to which the user is invited. This should be retrieved + from the ``m.room.name`` state event. + example: "Bob's Emporium of Messages" + sender_display_name: + type: string + description: The display name of the user ID initiating the invite. + example: "Bob Smith" + sender_avatar_url: + type: string + description: The Content URI for the avatar of the user ID initiating the invite. + example: "mxc://example.org/an0th3rM3dia" required: ["medium", "address", "room_id", "sender"] responses: 200: diff --git a/changelogs/identity_service/newsfragments/2049.clarification b/changelogs/identity_service/newsfragments/2049.clarification new file mode 100644 index 00000000..403ac8d0 --- /dev/null +++ b/changelogs/identity_service/newsfragments/2049.clarification @@ -0,0 +1 @@ +Add missing aesthetic parameters to ``/store-invite``. From 78d93432f4cf3fde1cb11af37b95fe16b46bbe0c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 14:00:34 -0600 Subject: [PATCH 084/128] Synchronize proposals_intro.rst and CONTRIBUTING.rst --- CONTRIBUTING.rst | 28 +++++++++++++++------------- specification/proposals_intro.rst | 26 +++++++++++++++++--------- 2 files changed, 32 insertions(+), 22 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 7a6c6be2..0b814fb9 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -26,10 +26,11 @@ For this to be effective, the APIs need to be present and working correctly in a server before they can be documented in the specification. This process can take some time to complete. -For this reason, we have not found the github pull-request model effective for -discussing changes to the specification. Instead, we have adopted the workflow -as described at https://matrix.org/docs/spec/proposals - *please read this for -details on how to contribute spec changes*. +Changes to the protocol (new endpoints, ideas, etc) need to go through the +`proposals process `_. Other changes, +such as fixing bugs, typos, or clarifying existing behaviour do not need a proposal. +If you're not sure, visit us at `#matrix-spec:matrix.org`_ +and ask. Other changes @@ -51,8 +52,7 @@ following: `_ label. (If there is any doubt about whether it is the spec or the implementations - that need fixing, please discuss it with us first in `#matrix-dev:matrix.org - `_.) + that need fixing, please discuss it with us first in `#matrix-spec:matrix.org`_.) * Clarifications to the specification which do not change the behaviour of Matrix servers or clients in a way which might introduce compatibility @@ -60,14 +60,16 @@ following: `clarification `_ label. - For example, recommendations for UI behaviour do not require a proposal - document. On the other hand, changes to event contents would be best - discussed in a proposal document even though no changes would be necessary to - server implementations. + For example, areas where the specification is unclear do not require a proposal + to fix. On the other hand, introducing new behaviour is best represented by a + proposal. -For such changes, please do just open a `pull request`_. +For such changes, please do just open a `pull request`_. If you're not sure if +your change is covered by the above, please visit `#matrix-spec:matrix.org` and +ask. -.. _pull request: https://help.github.com/articles/about-pull-requests +.. _`pull request`: https://help.github.com/articles/about-pull-requests +.. _`#matrix-spec:matrix.org`: https://matrix.to/#/#matrix-spec:matrix.org Adding to the changelog @@ -96,7 +98,7 @@ the ``newsfragments`` directory. The ``type`` can be one of the following: * ``breaking`` - Used when the change is not backwards compatible. -* ``deprecation`` - Used when deprecating something +* ``deprecation`` - Used when deprecating something. All news fragments must have a brief summary explaining the change in the contents of the file. The summary must end in a full stop to be in line with diff --git a/specification/proposals_intro.rst b/specification/proposals_intro.rst index d95128b7..d492700e 100644 --- a/specification/proposals_intro.rst +++ b/specification/proposals_intro.rst @@ -13,12 +13,18 @@ Proposals for Spec Changes to Matrix If you are interested in submitting a change to the Matrix Specification, please take note of the following guidelines. -All changes to Specification content require a formal proposal process. This -involves writing a proposal, having it reviewed by everyone, having the -proposal being accepted, then actually having your ideas implemented as -committed changes to the `Specification repository +Most changes to the Specification require a formal proposal. Bug fixes, typos, +and clarifications to existing behaviour do not need proposals - see the +`contributing guide `_ +for more information on what does and does not need a proposal. + +The proposal process involves some technical writing, having it reviewed by +everyone, having the proposal being accepted, then actually having your ideas +implemented as committed changes to the `Specification repository `_. +.. TODO: Replace GH team link with https://matrix.org/foundation or something + Meet the `members of the Core Team `_, a group of individuals tasked with ensuring the spec process is as smooth and painless as @@ -33,14 +39,15 @@ Guiding Principles Proposals **must** act to the greater benefit of the entire Matrix ecosystem, rather than benefiting or privileging any single player or subset of players - -and must not contain any patent encumbered intellectual property. Members of the Core Team pledge to act as -a neutral custodian for Matrix on behalf of the whole ecosystem. +and must not contain any patent encumbered intellectual property. Members of +the Core Team pledge to act as a neutral custodian for Matrix on behalf of the +whole ecosystem. For clarity: the Matrix ecosystem is anyone who uses the Matrix protocol. That includes client users, server admins, client developers, bot developers, -bridge and application service developers, users and admins who are indirectly using Matrix via -3rd party networks which happen to be bridged, server developers, room -moderators and admins, companies/projects building products or services on +bridge and application service developers, users and admins who are indirectly +using Matrix via 3rd party networks which happen to be bridged, server developers, +room moderators and admins, companies/projects building products or services on Matrix, spec contributors, translators, and those who created it in the first place. @@ -242,6 +249,7 @@ Spec PR Merged merged A proposal with Postponed proposal-postponed A proposal that is temporarily blocked or a feature that may not be useful currently but perhaps sometime in the future Closed proposal-closed A proposal which has been reviewed and deemed unsuitable for acceptance +Obsolete obsolete A proposal which has been made obsolete by another proposal or decision elsewhere. =============================== ============================= ==================================== From c25afa663ef619a82b4d7f93f96118974a59d4a0 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 14:56:42 -0600 Subject: [PATCH 085/128] Use consistent event examples throughout the Client-Server API Fixes https://github.com/matrix-org/matrix-doc/issues/1968 --- api/client-server/event_context.yaml | 88 ++++-------- api/client-server/message_pagination.yaml | 39 +---- api/client-server/notifications.yaml | 11 +- api/client-server/old_sync.yaml | 136 ++---------------- api/client-server/peeking_events.yaml | 12 +- api/client-server/room_initial_sync.yaml | 89 +----------- api/client-server/rooms.yaml | 136 +++--------------- api/client-server/search.yaml | 13 +- api/client-server/sync.yaml | 44 ++---- .../newsfragments/2050.clarification | 1 + 10 files changed, 94 insertions(+), 475 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2050.clarification diff --git a/api/client-server/event_context.yaml b/api/client-server/event_context.yaml index 91da3cf4..549902a0 100644 --- a/api/client-server/event_context.yaml +++ b/api/client-server/event_context.yaml @@ -101,65 +101,35 @@ paths: - "$ref": "definitions/event-schemas/schema/core-event-schema/state_event.yaml" examples: application/json: { - "end": "t29-57_2_0_2", - "events_after": [ - { - "age": 91911336, - "content": { - "body": "7", - "msgtype": "m.text" - }, - "event_id": "$14460306086CiUaL:localhost:8480", - "origin_server_ts": 1446030608551, - "room_id": "!sCDvXTtzjpiPxaqkkt:localhost:8480", - "type": "m.room.message", - "sender": "@test:localhost:8480" - } - ], - "events_before": [ - { - "age": 91911903, - "content": { - "body": "5", - "msgtype": "m.text" - }, - "event_id": "$14460306074UYTlh:localhost:8480", - "origin_server_ts": 1446030607984, - "room_id": "!sCDvXTtzjpiPxaqkkt:localhost:8480", - "type": "m.room.message", - "sender": "@test:localhost:8480" - } - ], - "start": "t27-54_2_0_2", - "state": [ - { - "age": 3123715284, - "content": { - "creator": "@test:localhost:8480" - }, - "event_id": "$14429988040dgQAE:localhost:8480", - "origin_server_ts": 1442998804603, - "room_id": "!sCDvXTtzjpiPxaqkkt:localhost:8480", - "state_key": "", - "type": "m.room.create", - "sender": "@test:localhost:8480" - }, - { - "age": 2067105053, - "content": { - "avatar_url": "mxc://localhost:8480/tVWZTAIIfqtXMZZtmGCkVjTD#auto", - "displayname": "Bob2", - "membership": "join" - }, - "event_id": "$14440554144URDbf:localhost:8480", - "origin_server_ts": 1444055414834, - "replaces_state": "$14440552472PgiGk:localhost:8480", - "room_id": "!sCDvXTtzjpiPxaqkkt:localhost:8480", - "state_key": "@test:localhost:8480", - "type": "m.room.member", - "sender": "@test:localhost:8480" - } - ] + "end": "t29-57_2_0_2", + "events_after": [ + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" } + ], + "event": { + "event_id": "$f3h4d129462ha:example.com", + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.message#m.image" + }, + "events_before": [ + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.message#m.file" + } + ], + "start": "t27-54_2_0_2", + "state": [ + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.create" + }, + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.member" + } + ] + } tags: - Room participation diff --git a/api/client-server/message_pagination.yaml b/api/client-server/message_pagination.yaml index 941e61fb..f29f1f23 100644 --- a/api/client-server/message_pagination.yaml +++ b/api/client-server/message_pagination.yaml @@ -114,43 +114,16 @@ paths: "end": "t47409-4357353_219380_26003_2265", "chunk": [ { - "origin_server_ts": 1444812213737, - "sender": "@alice:example.com", - "event_id": "$1444812213350496Caaaa:example.com", - "content": { - "body": "hello world", - "msgtype":"m.text" - }, - "room_id":"!Xq3620DUiqCaoxq:example.com", - "type":"m.room.message", - "age": 1042 + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" }, { - "origin_server_ts": 1444812194656 , - "sender": "@bob:example.com", - "event_id": "$1444812213350496Cbbbb:example.com", - "content": { - "body": "the world is big", - "msgtype":"m.text" - }, - "room_id":"!Xq3620DUiqCaoxq:example.com", - "type":"m.room.message", - "age": 20123 + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.name" }, { - "origin_server_ts": 1444812163990, - "sender": "@bob:example.com", - "event_id": "$1444812213350496Ccccc:example.com", - "content": { - "name": "New room name" - }, - "prev_content": { - "name": "Old room name" - }, - "state_key": "", - "room_id":"!Xq3620DUiqCaoxq:example.com", - "type":"m.room.name", - "age": 50789 + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.message#m.video" } ] } diff --git a/api/client-server/notifications.yaml b/api/client-server/notifications.yaml index b450885b..6d9366be 100644 --- a/api/client-server/notifications.yaml +++ b/api/client-server/notifications.yaml @@ -75,16 +75,7 @@ paths: "room_id": "!abcdefg:example.com", "ts": 1475508881945, "event": { - "sender": "@alice:example.com", - "type": "m.room.message", - "age": 124524, - "txn_id": "1234", - "content": { - "body": "I am a fish", - "msgtype": "m.text" - }, - "origin_server_ts": 1417731086797, - "event_id": "$74686972643033:example.com" + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" } } ] diff --git a/api/client-server/old_sync.yaml b/api/client-server/old_sync.yaml index c502c239..c9b7586b 100644 --- a/api/client-server/old_sync.yaml +++ b/api/client-server/old_sync.yaml @@ -64,18 +64,7 @@ paths: "start": "s3456_9_0", "end": "s3457_9_0", "chunk": [ - { - "age": 32, - "content": { - "body": "incoming message", - "msgtype": "m.text" - }, - "event_id": "$14328055551tzaee:localhost", - "origin_server_ts": 1432804485886, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "type": "m.room.message", - "sender": "@bob:localhost" - } + {"$ref": "definitions/event-schemas/examples/m.room.message#m.text"} ] } schema: @@ -142,16 +131,7 @@ paths: application/json: { "end": "s3456_9_0", "presence": [ - { - "content": { - "avatar_url": "mxc://localhost/GCmhgzMPRjqgpODLsNQzVuHZ#auto", - "displayname": "Bob", - "last_active_ago": 31053, - "presence": "online", - "user_id": "@bob:localhost" - }, - "type": "m.presence" - } + {"$ref": "definitions/event-schemas/examples/m.presence"} ], "account_data": [ { @@ -167,28 +147,12 @@ paths: "messages": { "chunk": [ { - "age": 343513403, - "content": { - "body": "foo", - "msgtype": "m.text" - }, - "event_id": "$14328044851tzTJS:localhost", - "origin_server_ts": 1432804485886, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "type": "m.room.message", - "sender": "@alice:localhost" + "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" }, { - "age": 343511809, - "content": { - "body": "bar", - "msgtype": "m.text" - }, - "event_id": "$14328044872spjFg:localhost", - "origin_server_ts": 1432804487480, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "type": "m.room.message", - "sender": "@bob:localhost" + "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", + "$ref": "definitions/event-schemas/examples/m.room.message#m.video" } ], "end": "s3456_9_0", @@ -197,81 +161,20 @@ paths: "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", "state": [ { - "age": 7148266897, - "content": { - "join_rule": "public" - }, - "event_id": "$14259997323TLwtb:localhost", - "origin_server_ts": 1425999732392, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "state_key": "", - "type": "m.room.join_rules", - "sender": "@alice:localhost" + "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", + "$ref": "definitions/event-schemas/examples/m.room.join_rules" }, { - "age": 6547561012, - "content": { - "avatar_url": "mxc://localhost/fzysBrHpPEeTGANCVLXWXNMI#auto", - "membership": "join" - }, - "event_id": "$1426600438280zExKY:localhost", - "membership": "join", - "origin_server_ts": 1426600438277, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "state_key": "@alice:localhost", - "type": "m.room.member", - "sender": "@alice:localhost" + "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", + "$ref": "definitions/event-schemas/examples/m.room.member" }, { - "age": 7148267200, - "content": { - "creator": "@alice:localhost" - }, - "event_id": "$14259997320KhbwJ:localhost", - "origin_server_ts": 1425999732089, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "state_key": "", - "type": "m.room.create", - "sender": "@alice:localhost" + "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", + "$ref": "definitions/event-schemas/examples/m.room.create" }, { - "age": 1622568720, - "content": { - "avatar_url": "mxc://localhost/GCmhgzMPRjqgpODLsNQzVuHZ#auto", - "displayname": "Bob", - "membership": "join" - }, - "event_id": "$1431525430134MxlLX:localhost", - "origin_server_ts": 1431525430569, - "replaces_state": "$142652023736BSXcM:localhost", - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "state_key": "@bob:localhost", - "type": "m.room.member", - "sender": "@bob:localhost" - }, - { - "age": 7148267004, - "content": { - "ban": 50, - "events": { - "m.room.name": 100, - "m.room.power_levels": 100 - }, - "events_default": 0, - "kick": 50, - "redact": 50, - "state_default": 50, - "users": { - "@alice:localhost": 100 - }, - "users_default": 0 - }, - "event_id": "$14259997322mqfaq:localhost", - "origin_server_ts": 1425999732285, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "state_key": "", - "type": "m.room.power_levels", - "sender": "@alice:localhost" + "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", + "$ref": "definitions/event-schemas/examples/m.room.power_levels" } ], "visibility": "private", @@ -423,16 +326,7 @@ paths: 200: description: The full event. examples: - application/json: { - "content": { - "body": "Hello world!", - "msgtype": "m.text" - }, - "room_id": "!wfgy43Sg4a:matrix.org", - "sender": "@bob:matrix.org", - "event_id": "$asfDuShaf7Gafaw:matrix.org", - "type": "m.room.message" - } + application/json: {"$ref": "definitions/event-schemas/examples/m.room.message#m.text"} schema: allOf: - "$ref": "definitions/event-schemas/schema/core-event-schema/event.yaml" diff --git a/api/client-server/peeking_events.yaml b/api/client-server/peeking_events.yaml index 2f66bae7..e3dc5777 100644 --- a/api/client-server/peeking_events.yaml +++ b/api/client-server/peeking_events.yaml @@ -75,16 +75,8 @@ paths: "end": "s3457_9_0", "chunk": [ { - "age": 32, - "content": { - "body": "incoming message", - "msgtype": "m.text" - }, - "event_id": "$14328055551tzaee:localhost", - "origin_server_ts": 1432804485886, - "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "type": "m.room.message", - "sender": "@bob:localhost" + "room_id": "!somewhere:over.the.rainbow", + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" } ] } diff --git a/api/client-server/room_initial_sync.yaml b/api/client-server/room_initial_sync.yaml index c27f0f24..2a354fc0 100644 --- a/api/client-server/room_initial_sync.yaml +++ b/api/client-server/room_initial_sync.yaml @@ -43,28 +43,12 @@ paths: "messages": { "chunk": [ { - "age": 343513403, - "content": { - "body": "foo", - "msgtype": "m.text" - }, - "event_id": "$14328044851tzTJS:example.com", - "origin_server_ts": 1432804485886, "room_id": "!636q39766251:example.com", - "type": "m.room.message", - "sender": "@alice:example.com" + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" }, { - "age": 343511809, - "content": { - "body": "bar", - "msgtype": "m.text" - }, - "event_id": "$14328044872spjFg:example.com", - "origin_server_ts": 1432804487480, "room_id": "!636q39766251:example.com", - "type": "m.room.message", - "sender": "@bob:example.com" + "$ref": "definitions/event-schemas/examples/m.room.message#m.file" } ], "end": "s3456_9_0", @@ -73,81 +57,20 @@ paths: "room_id": "!636q39766251:example.com", "state": [ { - "age": 7148266897, - "content": { - "join_rule": "public" - }, - "event_id": "$14259997323TLwtb:example.com", - "origin_server_ts": 1425999732392, "room_id": "!636q39766251:example.com", - "state_key": "", - "type": "m.room.join_rules", - "sender": "@alice:example.com" + "$ref": "definitions/event-schemas/examples/m.room.join_rules" }, { - "age": 6547561012, - "content": { - "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto", - "membership": "join" - }, - "event_id": "$1426600438280zExKY:example.com", - "membership": "join", - "origin_server_ts": 1426600438277, "room_id": "!636q39766251:example.com", - "state_key": "@alice:example.com", - "type": "m.room.member", - "sender": "@alice:example.com" + "$ref": "definitions/event-schemas/examples/m.room.member" }, { - "age": 7148267200, - "content": { - "creator": "@alice:example.com" - }, - "event_id": "$14259997320KhbwJ:example.com", - "origin_server_ts": 1425999732089, "room_id": "!636q39766251:example.com", - "state_key": "", - "type": "m.room.create", - "sender": "@alice:example.com" + "$ref": "definitions/event-schemas/examples/m.room.create" }, { - "age": 1622568720, - "content": { - "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto", - "displayname": "Bob", - "membership": "join" - }, - "event_id": "$1431525430134MxlLX:example.com", - "origin_server_ts": 1431525430569, - "replaces_state": "$142652023736BSXcM:example.com", "room_id": "!636q39766251:example.com", - "state_key": "@bob:example.com", - "type": "m.room.member", - "sender": "@bob:example.com" - }, - { - "age": 7148267004, - "content": { - "ban": 50, - "events": { - "m.room.name": 100, - "m.room.power_levels": 100 - }, - "events_default": 0, - "kick": 50, - "redact": 50, - "state_default": 50, - "users": { - "@alice:example.com": 100 - }, - "users_default": 0 - }, - "event_id": "$14259997322mqfaq:example.com", - "origin_server_ts": 1425999732285, - "room_id": "!636q39766251:example.com", - "state_key": "", - "type": "m.room.power_levels", - "sender": "@alice:example.com" + "$ref": "definitions/event-schemas/examples/m.room.power_levels" } ], "visibility": "private", diff --git a/api/client-server/rooms.yaml b/api/client-server/rooms.yaml index cc1f2bf7..55456436 100644 --- a/api/client-server/rooms.yaml +++ b/api/client-server/rooms.yaml @@ -42,7 +42,7 @@ paths: name: roomId description: The ID of the room the event is in. required: true - x-example: "!asfDuShaf7Gafaw:matrix.org" + x-example: "!636q39766251:matrix.org" - in: path type: string name: eventId @@ -54,15 +54,9 @@ paths: description: The full event. examples: application/json: { - "content": { - "body": "Hello world!", - "msgtype": "m.text" - }, - "room_id": "!wfgy43Sg4a:matrix.org", - "sender": "@bob:matrix.org", - "event_id": "$asfDuShaf7Gafaw:matrix.org", - "type": "m.room.message" - } + "room_id": "!636q39766251:matrix.org", + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + } schema: allOf: - "$ref": "definitions/event-schemas/schema/core-event-schema/event.yaml" @@ -178,84 +172,23 @@ paths: description: The current state of the room examples: application/json: [ - { - "age": 7148266897, - "content": { - "join_rule": "public" - }, - "event_id": "$14259997323TLwtb:example.com", - "origin_server_ts": 1425999732392, - "room_id": "!636q39766251:example.com", - "state_key": "", - "type": "m.room.join_rules", - "sender": "@alice:example.com" - }, - { - "age": 6547561012, - "content": { - "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto", - "membership": "join" - }, - "event_id": "$1426600438280zExKY:example.com", - "membership": "join", - "origin_server_ts": 1426600438277, - "room_id": "!636q39766251:example.com", - "state_key": "@alice:example.com", - "type": "m.room.member", - "sender": "@alice:example.com" - }, - { - "age": 7148267200, - "content": { - "creator": "@alice:example.com" - }, - "event_id": "$14259997320KhbwJ:example.com", - "origin_server_ts": 1425999732089, - "room_id": "!636q39766251:example.com", - "state_key": "", - "type": "m.room.create", - "sender": "@alice:example.com" - }, - { - "age": 1622568720, - "content": { - "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto", - "displayname": "Bob", - "membership": "join" - }, - "event_id": "$1431525430134MxlLX:example.com", - "origin_server_ts": 1431525430569, - "replaces_state": "$142652023736BSXcM:example.com", - "room_id": "!636q39766251:example.com", - "state_key": "@bob:example.com", - "type": "m.room.member", - "sender": "@bob:example.com" - }, - { - "age": 7148267004, - "content": { - "ban": 50, - "events": { - "m.room.name": 100, - "m.room.power_levels": 100 - }, - "events_default": 0, - "kick": 50, - "redact": 50, - "state_default": 50, - "users": { - "@alice:example.com": 100 - }, - "users_default": 0 - }, - "event_id": "$14259997322mqfaq:example.com", - "origin_server_ts": 1425999732285, - "room_id": "!636q39766251:example.com", - "state_key": "", - "type": "m.room.power_levels", - "sender": "@alice:example.com" - } - ] + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.join_rules" + }, + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.member" + }, + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.create" + }, + { + "room_id": "!636q39766251:example.com", + "$ref": "definitions/event-schemas/examples/m.room.power_levels" + } + ] schema: type: array title: RoomState @@ -300,33 +233,8 @@ paths: application/json: { "chunk": [ { - "age": 6547561012, - "content": { - "avatar_url": "mxc://example.com/fzysBrHpPEeTGANCVLXWXNMI#auto", - "membership": "join" - }, - "event_id": "$1426600438280zExKY:example.com", - "membership": "join", - "origin_server_ts": 1426600438277, "room_id": "!636q39766251:example.com", - "state_key": "@alice:example.com", - "type": "m.room.member", - "sender": "@alice:example.com" - }, - { - "age": 1622568720, - "content": { - "avatar_url": "mxc://example.com/GCmhgzMPRjqgpODLsNQzVuHZ#auto", - "displayname": "Bob", - "membership": "join" - }, - "event_id": "$1431525430134MxlLX:example.com", - "origin_server_ts": 1431525430569, - "replaces_state": "$142652023736BSXcM:example.com", - "room_id": "!636q39766251:example.com", - "state_key": "@bob:example.com", - "type": "m.room.member", - "sender": "@bob:example.com" + "$ref": "definitions/event-schemas/examples/m.room.member" } ] } diff --git a/api/client-server/search.yaml b/api/client-server/search.yaml index 4a5f4515..9d5d2074 100644 --- a/api/client-server/search.yaml +++ b/api/client-server/search.yaml @@ -280,7 +280,7 @@ paths: Any groups that were requested. The outer ``string`` key is the group key requested (eg: ``room_id`` - or ``sender``). The inner ``string`` key is the grouped value (eg: + or ``sender``). The inner ``string`` key is the grouped value (eg: a room's ID or a user's ID). additionalProperties: type: object @@ -347,16 +347,9 @@ paths: { "rank": 0.00424866, "result": { - "age": 526228296, - "content": { - "body": "Test content martians and men", - "msgtype": "m.text" - }, - "event_id": "$144429830826TWwbB:localhost", - "origin_server_ts": 1444298308034, "room_id": "!qPewotXpIctQySfjSy:localhost", - "type": "m.room.message", - "sender": "@test:localhost" + "event_id": "$144429830826TWwbB:localhost", + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" } } ] diff --git a/api/client-server/sync.yaml b/api/client-server/sync.yaml index f204152a..ad5a6075 100644 --- a/api/client-server/sync.yaml +++ b/api/client-server/sync.yaml @@ -310,11 +310,7 @@ paths: "next_batch": "s72595_4483_1934", "presence": { "events": [ - { - "sender": "@alice:example.com", - "type": "m.presence", - "content": {"presence": "online"} - } + {"$ref": "definitions/event-schemas/examples/m.presence"} ] }, "account_data": { @@ -333,36 +329,20 @@ paths: "state": { "events": [ { - "sender": "@alice:example.com", - "type": "m.room.member", - "state_key": "@alice:example.com", - "content": {"membership": "join"}, - "origin_server_ts": 1417731086795, - "event_id": "$66697273743031:example.com" + "room_id": "!726s6s6q:example.com", + "$ref": "definitions/event-schemas/examples/m.room.member" } ] }, "timeline": { "events": [ { - "sender": "@bob:example.com", - "type": "m.room.member", - "state_key": "@bob:example.com", - "content": {"membership": "join"}, - "prev_content": {"membership": "invite"}, - "origin_server_ts": 1417731086795, - "event_id": "$7365636s6r6432:example.com" + "room_id": "!726s6s6q:example.com", + "$ref": "definitions/event-schemas/examples/m.room.member" }, { - "sender": "@alice:example.com", - "type": "m.room.message", - "txn_id": "1234", - "content": { - "body": "I am a fish", - "msgtype": "m.text" - }, - "origin_server_ts": 1417731086797, - "event_id": "$74686972643033:example.com" + "room_id": "!726s6s6q:example.com", + "$ref": "definitions/event-schemas/examples/m.room.message#m.text" } ], "limited": true, @@ -370,18 +350,12 @@ paths: }, "ephemeral": { "events": [ - { - "type": "m.typing", - "content": {"user_ids": ["@alice:example.com"]} - } + {"$ref": "definitions/event-schemas/examples/m.typing"} ] }, "account_data": { "events": [ - { - "type": "m.tag", - "content": {"tags": {"work": {"order": 1}}} - }, + {"$ref": "definitions/event-schemas/examples/m.tag"}, { "type": "org.example.custom.room.config", "content": { diff --git a/changelogs/client_server/newsfragments/2050.clarification b/changelogs/client_server/newsfragments/2050.clarification new file mode 100644 index 00000000..384daa11 --- /dev/null +++ b/changelogs/client_server/newsfragments/2050.clarification @@ -0,0 +1 @@ +Use consistent examples of events throughout the specification. From d6d74c4cbef04e194826cf3565b21bf5959a7d31 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 15:23:45 -0600 Subject: [PATCH 086/128] Switch to using $ instead of # for sub-types # is reserved by the swagger validator as a way to include partial content from a JSON object (eg: "#/path" would include {"test": true} from the object {"path":{"test":true}}). Instead of trying to convince the validator that it is wrong, we'll just use a different character. Note that our rendering tools do not care about #-style references to objects. It's still somewhat worth changing the character though. --- api/application-service/transactions.yaml | 2 +- api/client-server/event_context.yaml | 6 +++--- api/client-server/message_pagination.yaml | 4 ++-- api/client-server/notifications.yaml | 2 +- api/client-server/old_sync.yaml | 8 ++++---- api/client-server/peeking_events.yaml | 2 +- api/client-server/room_initial_sync.yaml | 4 ++-- api/client-server/rooms.yaml | 2 +- api/client-server/search.yaml | 2 +- api/client-server/sync.yaml | 2 +- event-schemas/check_examples.py | 4 ++-- ....encrypted#megolm => m.room.encrypted$megolm} | 0 ...m.room.encrypted#olm => m.room.encrypted$olm} | 0 ...oom_state => m.room.member$invite_room_state} | 0 ...y_invite => m.room.member$third_party_invite} | 0 ...om.message#m.audio => m.room.message$m.audio} | 2 +- ...om.message#m.emote => m.room.message$m.emote} | 0 ...room.message#m.file => m.room.message$m.file} | 0 ...om.message#m.image => m.room.message$m.image} | 0 ...sage#m.location => m.room.message$m.location} | 0 ....message#m.notice => m.room.message$m.notice} | 0 ...ver_notice => m.room.message$m.server_notice} | 0 ...room.message#m.text => m.room.message$m.text} | 0 ...om.message#m.video => m.room.message$m.video} | 0 ...request => m.room_key_request$cancel_request} | 0 ...equest#request => m.room_key_request$request} | 0 ...om.message#m.audio => m.room.message$m.audio} | 0 ...om.message#m.emote => m.room.message$m.emote} | 0 ...room.message#m.file => m.room.message$m.file} | 0 ...om.message#m.image => m.room.message$m.image} | 0 ...sage#m.location => m.room.message$m.location} | 0 ....message#m.notice => m.room.message$m.notice} | 0 ...ver_notice => m.room.message$m.server_notice} | 0 ...room.message#m.text => m.room.message$m.text} | 0 ...om.message#m.video => m.room.message$m.video} | 0 scripts/templating/matrix_templates/sections.py | 16 ++++++++-------- scripts/templating/matrix_templates/units.py | 2 +- 37 files changed, 29 insertions(+), 29 deletions(-) rename event-schemas/examples/{m.room.encrypted#megolm => m.room.encrypted$megolm} (100%) rename event-schemas/examples/{m.room.encrypted#olm => m.room.encrypted$olm} (100%) rename event-schemas/examples/{m.room.member#invite_room_state => m.room.member$invite_room_state} (100%) rename event-schemas/examples/{m.room.member#third_party_invite => m.room.member$third_party_invite} (100%) rename event-schemas/examples/{m.room.message#m.audio => m.room.message$m.audio} (99%) rename event-schemas/examples/{m.room.message#m.emote => m.room.message$m.emote} (100%) rename event-schemas/examples/{m.room.message#m.file => m.room.message$m.file} (100%) rename event-schemas/examples/{m.room.message#m.image => m.room.message$m.image} (100%) rename event-schemas/examples/{m.room.message#m.location => m.room.message$m.location} (100%) rename event-schemas/examples/{m.room.message#m.notice => m.room.message$m.notice} (100%) rename event-schemas/examples/{m.room.message#m.server_notice => m.room.message$m.server_notice} (100%) rename event-schemas/examples/{m.room.message#m.text => m.room.message$m.text} (100%) rename event-schemas/examples/{m.room.message#m.video => m.room.message$m.video} (100%) rename event-schemas/examples/{m.room_key_request#cancel_request => m.room_key_request$cancel_request} (100%) rename event-schemas/examples/{m.room_key_request#request => m.room_key_request$request} (100%) rename event-schemas/schema/{m.room.message#m.audio => m.room.message$m.audio} (100%) rename event-schemas/schema/{m.room.message#m.emote => m.room.message$m.emote} (100%) rename event-schemas/schema/{m.room.message#m.file => m.room.message$m.file} (100%) rename event-schemas/schema/{m.room.message#m.image => m.room.message$m.image} (100%) rename event-schemas/schema/{m.room.message#m.location => m.room.message$m.location} (100%) rename event-schemas/schema/{m.room.message#m.notice => m.room.message$m.notice} (100%) rename event-schemas/schema/{m.room.message#m.server_notice => m.room.message$m.server_notice} (100%) rename event-schemas/schema/{m.room.message#m.text => m.room.message$m.text} (100%) rename event-schemas/schema/{m.room.message#m.video => m.room.message$m.video} (100%) diff --git a/api/application-service/transactions.yaml b/api/application-service/transactions.yaml index 98181196..09f15276 100644 --- a/api/application-service/transactions.yaml +++ b/api/application-service/transactions.yaml @@ -56,7 +56,7 @@ paths: example: { "events": [ {"$ref": "../../event-schemas/examples/m.room.member"}, - {"$ref": "../../event-schemas/examples/m.room.message#m.text"} + {"$ref": "../../event-schemas/examples/m.room.message$m.text"} ] } description: Transaction information diff --git a/api/client-server/event_context.yaml b/api/client-server/event_context.yaml index 549902a0..e2018028 100644 --- a/api/client-server/event_context.yaml +++ b/api/client-server/event_context.yaml @@ -105,18 +105,18 @@ paths: "events_after": [ { "room_id": "!636q39766251:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" } ], "event": { "event_id": "$f3h4d129462ha:example.com", "room_id": "!636q39766251:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.image" + "$ref": "definitions/event-schemas/examples/m.room.message$m.image" }, "events_before": [ { "room_id": "!636q39766251:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.file" + "$ref": "definitions/event-schemas/examples/m.room.message$m.file" } ], "start": "t27-54_2_0_2", diff --git a/api/client-server/message_pagination.yaml b/api/client-server/message_pagination.yaml index f29f1f23..c9f9d0ae 100644 --- a/api/client-server/message_pagination.yaml +++ b/api/client-server/message_pagination.yaml @@ -115,7 +115,7 @@ paths: "chunk": [ { "room_id": "!636q39766251:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" }, { "room_id": "!636q39766251:example.com", @@ -123,7 +123,7 @@ paths: }, { "room_id": "!636q39766251:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.video" + "$ref": "definitions/event-schemas/examples/m.room.message$m.video" } ] } diff --git a/api/client-server/notifications.yaml b/api/client-server/notifications.yaml index 6d9366be..87341d41 100644 --- a/api/client-server/notifications.yaml +++ b/api/client-server/notifications.yaml @@ -75,7 +75,7 @@ paths: "room_id": "!abcdefg:example.com", "ts": 1475508881945, "event": { - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" } } ] diff --git a/api/client-server/old_sync.yaml b/api/client-server/old_sync.yaml index c9b7586b..a79c3b32 100644 --- a/api/client-server/old_sync.yaml +++ b/api/client-server/old_sync.yaml @@ -64,7 +64,7 @@ paths: "start": "s3456_9_0", "end": "s3457_9_0", "chunk": [ - {"$ref": "definitions/event-schemas/examples/m.room.message#m.text"} + {"$ref": "definitions/event-schemas/examples/m.room.message$m.text"} ] } schema: @@ -148,11 +148,11 @@ paths: "chunk": [ { "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" }, { "room_id": "!TmaZBKYIFrIPVGoUYp:localhost", - "$ref": "definitions/event-schemas/examples/m.room.message#m.video" + "$ref": "definitions/event-schemas/examples/m.room.message$m.video" } ], "end": "s3456_9_0", @@ -326,7 +326,7 @@ paths: 200: description: The full event. examples: - application/json: {"$ref": "definitions/event-schemas/examples/m.room.message#m.text"} + application/json: {"$ref": "definitions/event-schemas/examples/m.room.message$m.text"} schema: allOf: - "$ref": "definitions/event-schemas/schema/core-event-schema/event.yaml" diff --git a/api/client-server/peeking_events.yaml b/api/client-server/peeking_events.yaml index e3dc5777..feac36f4 100644 --- a/api/client-server/peeking_events.yaml +++ b/api/client-server/peeking_events.yaml @@ -76,7 +76,7 @@ paths: "chunk": [ { "room_id": "!somewhere:over.the.rainbow", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" } ] } diff --git a/api/client-server/room_initial_sync.yaml b/api/client-server/room_initial_sync.yaml index 2a354fc0..72e56ba9 100644 --- a/api/client-server/room_initial_sync.yaml +++ b/api/client-server/room_initial_sync.yaml @@ -44,11 +44,11 @@ paths: "chunk": [ { "room_id": "!636q39766251:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" }, { "room_id": "!636q39766251:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.file" + "$ref": "definitions/event-schemas/examples/m.room.message$m.file" } ], "end": "s3456_9_0", diff --git a/api/client-server/rooms.yaml b/api/client-server/rooms.yaml index 55456436..377783c6 100644 --- a/api/client-server/rooms.yaml +++ b/api/client-server/rooms.yaml @@ -55,7 +55,7 @@ paths: examples: application/json: { "room_id": "!636q39766251:matrix.org", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" } schema: allOf: diff --git a/api/client-server/search.yaml b/api/client-server/search.yaml index 9d5d2074..4fe72d5b 100644 --- a/api/client-server/search.yaml +++ b/api/client-server/search.yaml @@ -349,7 +349,7 @@ paths: "result": { "room_id": "!qPewotXpIctQySfjSy:localhost", "event_id": "$144429830826TWwbB:localhost", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" } } ] diff --git a/api/client-server/sync.yaml b/api/client-server/sync.yaml index ad5a6075..02fddb84 100644 --- a/api/client-server/sync.yaml +++ b/api/client-server/sync.yaml @@ -342,7 +342,7 @@ paths: }, { "room_id": "!726s6s6q:example.com", - "$ref": "definitions/event-schemas/examples/m.room.message#m.text" + "$ref": "definitions/event-schemas/examples/m.room.message$m.text" } ], "limited": true, diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py index 3e536ec3..2baa3ef0 100755 --- a/event-schemas/check_examples.py +++ b/event-schemas/check_examples.py @@ -112,8 +112,8 @@ def check_example_dir(exampledir, schemadir): continue examplepath = os.path.join(root, filename) schemapath = examplepath.replace(exampledir, schemadir) - if schemapath.find("#") >= 0: - schemapath = schemapath[:schemapath.find("#")] + if schemapath.find("$") >= 0: + schemapath = schemapath[:schemapath.find("$")] try: check_example_file(examplepath, schemapath) except Exception as e: diff --git a/event-schemas/examples/m.room.encrypted#megolm b/event-schemas/examples/m.room.encrypted$megolm similarity index 100% rename from event-schemas/examples/m.room.encrypted#megolm rename to event-schemas/examples/m.room.encrypted$megolm diff --git a/event-schemas/examples/m.room.encrypted#olm b/event-schemas/examples/m.room.encrypted$olm similarity index 100% rename from event-schemas/examples/m.room.encrypted#olm rename to event-schemas/examples/m.room.encrypted$olm diff --git a/event-schemas/examples/m.room.member#invite_room_state b/event-schemas/examples/m.room.member$invite_room_state similarity index 100% rename from event-schemas/examples/m.room.member#invite_room_state rename to event-schemas/examples/m.room.member$invite_room_state diff --git a/event-schemas/examples/m.room.member#third_party_invite b/event-schemas/examples/m.room.member$third_party_invite similarity index 100% rename from event-schemas/examples/m.room.member#third_party_invite rename to event-schemas/examples/m.room.member$third_party_invite diff --git a/event-schemas/examples/m.room.message#m.audio b/event-schemas/examples/m.room.message$m.audio similarity index 99% rename from event-schemas/examples/m.room.message#m.audio rename to event-schemas/examples/m.room.message$m.audio index 2f743d49..58e874e0 100644 --- a/event-schemas/examples/m.room.message#m.audio +++ b/event-schemas/examples/m.room.message$m.audio @@ -11,4 +11,4 @@ }, "msgtype": "m.audio" } -} +} diff --git a/event-schemas/examples/m.room.message#m.emote b/event-schemas/examples/m.room.message$m.emote similarity index 100% rename from event-schemas/examples/m.room.message#m.emote rename to event-schemas/examples/m.room.message$m.emote diff --git a/event-schemas/examples/m.room.message#m.file b/event-schemas/examples/m.room.message$m.file similarity index 100% rename from event-schemas/examples/m.room.message#m.file rename to event-schemas/examples/m.room.message$m.file diff --git a/event-schemas/examples/m.room.message#m.image b/event-schemas/examples/m.room.message$m.image similarity index 100% rename from event-schemas/examples/m.room.message#m.image rename to event-schemas/examples/m.room.message$m.image diff --git a/event-schemas/examples/m.room.message#m.location b/event-schemas/examples/m.room.message$m.location similarity index 100% rename from event-schemas/examples/m.room.message#m.location rename to event-schemas/examples/m.room.message$m.location diff --git a/event-schemas/examples/m.room.message#m.notice b/event-schemas/examples/m.room.message$m.notice similarity index 100% rename from event-schemas/examples/m.room.message#m.notice rename to event-schemas/examples/m.room.message$m.notice diff --git a/event-schemas/examples/m.room.message#m.server_notice b/event-schemas/examples/m.room.message$m.server_notice similarity index 100% rename from event-schemas/examples/m.room.message#m.server_notice rename to event-schemas/examples/m.room.message$m.server_notice diff --git a/event-schemas/examples/m.room.message#m.text b/event-schemas/examples/m.room.message$m.text similarity index 100% rename from event-schemas/examples/m.room.message#m.text rename to event-schemas/examples/m.room.message$m.text diff --git a/event-schemas/examples/m.room.message#m.video b/event-schemas/examples/m.room.message$m.video similarity index 100% rename from event-schemas/examples/m.room.message#m.video rename to event-schemas/examples/m.room.message$m.video diff --git a/event-schemas/examples/m.room_key_request#cancel_request b/event-schemas/examples/m.room_key_request$cancel_request similarity index 100% rename from event-schemas/examples/m.room_key_request#cancel_request rename to event-schemas/examples/m.room_key_request$cancel_request diff --git a/event-schemas/examples/m.room_key_request#request b/event-schemas/examples/m.room_key_request$request similarity index 100% rename from event-schemas/examples/m.room_key_request#request rename to event-schemas/examples/m.room_key_request$request diff --git a/event-schemas/schema/m.room.message#m.audio b/event-schemas/schema/m.room.message$m.audio similarity index 100% rename from event-schemas/schema/m.room.message#m.audio rename to event-schemas/schema/m.room.message$m.audio diff --git a/event-schemas/schema/m.room.message#m.emote b/event-schemas/schema/m.room.message$m.emote similarity index 100% rename from event-schemas/schema/m.room.message#m.emote rename to event-schemas/schema/m.room.message$m.emote diff --git a/event-schemas/schema/m.room.message#m.file b/event-schemas/schema/m.room.message$m.file similarity index 100% rename from event-schemas/schema/m.room.message#m.file rename to event-schemas/schema/m.room.message$m.file diff --git a/event-schemas/schema/m.room.message#m.image b/event-schemas/schema/m.room.message$m.image similarity index 100% rename from event-schemas/schema/m.room.message#m.image rename to event-schemas/schema/m.room.message$m.image diff --git a/event-schemas/schema/m.room.message#m.location b/event-schemas/schema/m.room.message$m.location similarity index 100% rename from event-schemas/schema/m.room.message#m.location rename to event-schemas/schema/m.room.message$m.location diff --git a/event-schemas/schema/m.room.message#m.notice b/event-schemas/schema/m.room.message$m.notice similarity index 100% rename from event-schemas/schema/m.room.message#m.notice rename to event-schemas/schema/m.room.message$m.notice diff --git a/event-schemas/schema/m.room.message#m.server_notice b/event-schemas/schema/m.room.message$m.server_notice similarity index 100% rename from event-schemas/schema/m.room.message#m.server_notice rename to event-schemas/schema/m.room.message$m.server_notice diff --git a/event-schemas/schema/m.room.message#m.text b/event-schemas/schema/m.room.message$m.text similarity index 100% rename from event-schemas/schema/m.room.message#m.text rename to event-schemas/schema/m.room.message$m.text diff --git a/event-schemas/schema/m.room.message#m.video b/event-schemas/schema/m.room.message$m.video similarity index 100% rename from event-schemas/schema/m.room.message#m.video rename to event-schemas/schema/m.room.message$m.video diff --git a/scripts/templating/matrix_templates/sections.py b/scripts/templating/matrix_templates/sections.py index 92afa5ff..c88959ed 100644 --- a/scripts/templating/matrix_templates/sections.py +++ b/scripts/templating/matrix_templates/sections.py @@ -115,7 +115,7 @@ class MatrixSections(Sections): schemas = self.units.get("event_schemas") renders = {} for event_type in schemas: - underscored_event_type = event_type.replace(".", "_").replace("#", "_") + underscored_event_type = event_type.replace(".", "_").replace("$", "_") renders[underscored_event_type + "_event"] = self._render_events( lambda x: x == event_type, sorted ) @@ -125,7 +125,7 @@ class MatrixSections(Sections): def filterFn(eventType): return ( eventType.startswith("m.room") and - not eventType.startswith("m.room.message#m.") + not eventType.startswith("m.room.message$m.") ) return self._render_events(filterFn, sorted) @@ -138,22 +138,22 @@ class MatrixSections(Sections): ]["subtitle"] sections = [] msgtype_order = [ - "m.room.message#m.text", "m.room.message#m.emote", - "m.room.message#m.notice", "m.room.message#m.image", - "m.room.message#m.file" + "m.room.message$m.text", "m.room.message$m.emote", + "m.room.message$m.notice", "m.room.message$m.image", + "m.room.message$m.file" ] excluded_types = [ # We exclude server notices from here because we handle them in a # dedicated module. We do not want to confuse developers this early # in the spec. - "m.room.message#m.server_notice", + "m.room.message$m.server_notice", ] other_msgtypes = [ - k for k in schemas.keys() if k.startswith("m.room.message#") and + k for k in schemas.keys() if k.startswith("m.room.message$") and k not in msgtype_order and k not in excluded_types ] for event_name in (msgtype_order + other_msgtypes): - if not event_name.startswith("m.room.message#m."): + if not event_name.startswith("m.room.message$m."): continue sections.append(template.render( example=examples[event_name][0], diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index d4745e4d..ddb65efe 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -795,7 +795,7 @@ class MatrixUnits(Units): if not filename.startswith("m."): continue - event_name = filename.split("#")[0] + event_name = filename.split("$")[0] filepath = os.path.join(path, filename) logger.info("Reading event example: %s" % filepath) try: From f9472bae411e8db020247eb7394d7bd42fe50d5a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 16:27:16 -0600 Subject: [PATCH 087/128] Change reference for definition of push rule condition kinds Fixes https://github.com/matrix-org/matrix-doc/issues/1970 --- api/client-server/definitions/push_condition.yaml | 12 ++++++++---- .../client_server/newsfragments/2052.clarification | 1 + 2 files changed, 9 insertions(+), 4 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2052.clarification diff --git a/api/client-server/definitions/push_condition.yaml b/api/client-server/definitions/push_condition.yaml index 796a51f4..8752274e 100644 --- a/api/client-server/definitions/push_condition.yaml +++ b/api/client-server/definitions/push_condition.yaml @@ -16,16 +16,20 @@ title: PushCondition type: object properties: kind: - enum: - - event_match - - contains_display_name - - room_member_count type: string + description: |- + The kind of condition to apply. See `conditions <#conditions>`_ for + more information on the allowed kinds and how they work. key: type: string description: |- Required for ``event_match`` conditions. The dot-separated field of the event to match. + + Required for ``sender_notification_permission`` conditions. The field in + the power level event the user needs a minimum power level for. Fields + must be specified under the ``notifications`` property in the power level + event's ``content``. x-example: content.body pattern: type: string diff --git a/changelogs/client_server/newsfragments/2052.clarification b/changelogs/client_server/newsfragments/2052.clarification new file mode 100644 index 00000000..95bdc928 --- /dev/null +++ b/changelogs/client_server/newsfragments/2052.clarification @@ -0,0 +1 @@ +Clarify which push rule condition kinds exist. From 792bb8faa49c4551067fd824e805ba4f5c195509 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 16:28:36 -0600 Subject: [PATCH 088/128] Renumber changelog to match PR --- .../newsfragments/{2050.clarification => 2051.clarification} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changelogs/client_server/newsfragments/{2050.clarification => 2051.clarification} (100%) diff --git a/changelogs/client_server/newsfragments/2050.clarification b/changelogs/client_server/newsfragments/2051.clarification similarity index 100% rename from changelogs/client_server/newsfragments/2050.clarification rename to changelogs/client_server/newsfragments/2051.clarification From 464845feb0695c99f1b81e665e980e0289ec119c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 16:36:43 -0600 Subject: [PATCH 089/128] Make url required for m.file-like messages Fixes https://github.com/matrix-org/matrix-doc/issues/2008 This also removes `filename` from `m.file` because it has never been used in practice. --- changelogs/client_server/newsfragments/2053.clarification | 1 + event-schemas/schema/m.room.message#m.audio | 1 + event-schemas/schema/m.room.message#m.file | 2 +- event-schemas/schema/m.room.message#m.image | 1 + event-schemas/schema/m.room.message#m.video | 1 + 5 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/2053.clarification diff --git a/changelogs/client_server/newsfragments/2053.clarification b/changelogs/client_server/newsfragments/2053.clarification new file mode 100644 index 00000000..2a72a88e --- /dev/null +++ b/changelogs/client_server/newsfragments/2053.clarification @@ -0,0 +1 @@ +Clarify the required fields on ``m.file`` (and similar) messages. diff --git a/event-schemas/schema/m.room.message#m.audio b/event-schemas/schema/m.room.message#m.audio index c258b85f..99e28110 100644 --- a/event-schemas/schema/m.room.message#m.audio +++ b/event-schemas/schema/m.room.message#m.audio @@ -38,6 +38,7 @@ properties: required: - msgtype - body + - url type: object type: enum: diff --git a/event-schemas/schema/m.room.message#m.file b/event-schemas/schema/m.room.message#m.file index 2fb4fe50..2389d8a9 100644 --- a/event-schemas/schema/m.room.message#m.file +++ b/event-schemas/schema/m.room.message#m.file @@ -53,7 +53,7 @@ properties: required: - msgtype - body - - filename + - url type: object type: enum: diff --git a/event-schemas/schema/m.room.message#m.image b/event-schemas/schema/m.room.message#m.image index 349f78f4..1e6ebeaa 100644 --- a/event-schemas/schema/m.room.message#m.image +++ b/event-schemas/schema/m.room.message#m.image @@ -28,6 +28,7 @@ properties: required: - msgtype - body + - url type: object type: enum: diff --git a/event-schemas/schema/m.room.message#m.video b/event-schemas/schema/m.room.message#m.video index 8a66fdeb..2da7e0bc 100644 --- a/event-schemas/schema/m.room.message#m.video +++ b/event-schemas/schema/m.room.message#m.video @@ -59,6 +59,7 @@ properties: required: - msgtype - body + - url type: object type: enum: From 0f623113f10cfc50d99305a8d61ce02371636f5c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 18:35:18 -0600 Subject: [PATCH 090/128] Clarify that UIA stages cannot be attempted twice Fixes https://github.com/matrix-org/matrix-doc/issues/1987 Note: Synapse currently does not care, however the spirit of the text in the spec implies that completed == done forever, so we're just reinforcing it here. --- .../client_server/newsfragments/2054.clarification | 1 + specification/client_server_api.rst | 11 ++++++----- 2 files changed, 7 insertions(+), 5 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2054.clarification diff --git a/changelogs/client_server/newsfragments/2054.clarification b/changelogs/client_server/newsfragments/2054.clarification new file mode 100644 index 00000000..e43aea2d --- /dev/null +++ b/changelogs/client_server/newsfragments/2054.clarification @@ -0,0 +1 @@ +Clarify that User-Interactive Authentication stages cannot be attempted more than once. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 6c2e364a..318ac08d 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -453,11 +453,10 @@ params presented, that type may be present as a key in this dictionary. For example, the public part of an OAuth client ID could be given here. session - This is a session identifier that the client must pass back to the home - server, if one is provided, in subsequent attempts to authenticate in the same - API call. + This is a session identifier that the client must pass back to the homeserver, + if one is provided, in subsequent attempts to authenticate in the same API call. -The client then chooses a flow and attempts to complete one of the stages. It +The client then chooses a flow and attempts to complete the first stage. It does this by resubmitting the same request with the addition of an ``auth`` key in the object that it submits. This dictionary contains a ``type`` key whose value is the name of the authentication type that the client is attempting to complete. @@ -558,7 +557,9 @@ message in the standard format. For example: } If the client has completed all stages of a flow, the homeserver performs the -API call and returns the result as normal. +API call and returns the result as normal. Completed stages cannot be re-tried; +The client must abandon the current session and start over. Homeservers should +treat retries as authentication errors. Some authentication types may be completed by means other than through the Matrix client, for example, an email confirmation may be completed when the user From 2ed37f5bf464d03a322184221a89f6ebe4c61630 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 21:00:37 -0600 Subject: [PATCH 091/128] Clarify guest accounts and `auth` usage on /register Fixes https://github.com/matrix-org/matrix-doc/issues/1980 Fixes https://github.com/matrix-org/matrix-doc/issues/1984 --- api/client-server/registration.yaml | 34 ++++++++++++++----- .../newsfragments/2055.clarification | 1 + 2 files changed, 26 insertions(+), 9 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2055.clarification diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index 3195ab41..10b661a3 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -29,7 +29,8 @@ paths: post: summary: Register for an account on this homeserver. description: |- - This API endpoint uses the `User-Interactive Authentication API`_. + This API endpoint uses the `User-Interactive Authentication API`_, except in + the cases where a guest account is being registered. Register for an account on this homeserver. @@ -59,6 +60,11 @@ paths: supplied by the client or generated by the server. The server may invalidate any access token previously associated with that device. See `Relationship between access tokens and devices`_. + + When registering a guest account, all parameters in the request body + with the exception of ``initial_device_display_name`` are ignored by + the server. The server will pick a ``device_id`` for the account + regardless of input. operationId: register parameters: - in: query @@ -72,7 +78,7 @@ paths: enum: - guest - user - description: The kind of account to register. Defaults to `user`. + description: The kind of account to register. Defaults to ``user``. - in: body name: body schema: @@ -80,13 +86,11 @@ paths: properties: auth: description: |- - Additional authentication information for the - user-interactive authentication API. Note that this - information is *not* used to define how the registered user - should be authenticated, but is instead used to - authenticate the ``register`` call itself. It should be - left empty, or omitted, unless an earlier call returned an - response with status code 401. + Additional authentication information for the + user-interactive authentication API. Note that this + information is *not* used to define how the registered user + should be authenticated, but is instead used to + authenticate the ``register`` call itself. "$ref": "definitions/auth_data.yaml" bind_email: type: boolean @@ -194,6 +198,18 @@ paths: The homeserver requires additional authentication information. schema: "$ref": "definitions/auth_response.yaml" + 403: + description: |- + The homeserver does not permit registering the account. This response + can be used to identify that a particular ``kind`` of account is not + allowed, or that registration is generally not supported by the homeserver. + examples: + application/json: { + "errcode": "M_FORBIDDEN", + "error": "Registration is disabled" + } + schema: + "$ref": "definitions/errors/error.yaml" 429: description: This request was rate-limited. schema: diff --git a/changelogs/client_server/newsfragments/2055.clarification b/changelogs/client_server/newsfragments/2055.clarification new file mode 100644 index 00000000..3a57ef7e --- /dev/null +++ b/changelogs/client_server/newsfragments/2055.clarification @@ -0,0 +1 @@ +Clarify which parameters apply in what scenarios on ``/register``. From f3c0c5232fb812d90ceeda1348c1fc2d17558e28 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 29 May 2019 21:44:50 -0600 Subject: [PATCH 092/128] Add a table to show how changes in membership should be interpreted Fixes https://github.com/matrix-org/matrix-doc/issues/876 --- .../newsfragments/2056.clarification | 1 + event-schemas/schema/m.room.member | 23 +++++++++++++++++++ scripts/css/tables.css | 4 ++++ 3 files changed, 28 insertions(+) create mode 100644 changelogs/client_server/newsfragments/2056.clarification create mode 100644 scripts/css/tables.css diff --git a/changelogs/client_server/newsfragments/2056.clarification b/changelogs/client_server/newsfragments/2056.clarification new file mode 100644 index 00000000..12521867 --- /dev/null +++ b/changelogs/client_server/newsfragments/2056.clarification @@ -0,0 +1 @@ +Clarify how to interpret changes of ``membership`` over time. diff --git a/event-schemas/schema/m.room.member b/event-schemas/schema/m.room.member index de14644d..20d3cbcc 100644 --- a/event-schemas/schema/m.room.member +++ b/event-schemas/schema/m.room.member @@ -21,6 +21,29 @@ description: |- This event may also include an ``invite_room_state`` key inside the event's ``unsigned`` data. If present, this contains an array of ``StrippedState`` Events. These events provide information on a subset of state events such as the room name. + + The user for which a membership applies is represented by the ``state_key``. Under some conditions, + the ``sender`` and ``state_key`` may not match - this may be interpreted as the ``sender`` affecting + the membership state of the ``state_key`` user. + + The ``membership`` for a given user can change over time. The table below represents the various changes + over time and how clients and servers must interpret those changes. Previous membership can be retrieved + from the ``prev_content`` object on an event. If not present, the user's previous membership must be assumed + as ``leave``. + + .. TODO: Improve how this table is written? We use a csv-table to get around vertical header restrictions. + + .. csv-table:: + :header-rows: 1 + :stub-columns: 1 + + "","to ``invite``","to ``join``","to ``leave``","to ``ban``","to ``knock``" + "from ``invite``","No change.","User joined the room.","If the ``state_key`` is the same as the ``sender``, the user rejected the invite. Otherwise, the ``state_key`` user had their invite revoked.","User was banned.","Not implemented." + "from ``join``","Must never happen.","``displayname`` or ``avatar_url`` changed.","If the ``state_key`` is the same as the ``sender``, the user left. Otherwise, the ``state_key`` user was kicked.","User was kicked and banned.","Not implemented." + "from ``leave``","New invitation sent.","User joined.","Must never happen.","User was banned.","Not implemented." + "from ``ban``","Must never happen.","Must never happen.","User was unbanned.","No change.","Not implemented." + "from ``knock``","Not implemented.","Not implemented.","Not implemented.","Not implemented.","Not implemented." + properties: content: properties: diff --git a/scripts/css/tables.css b/scripts/css/tables.css new file mode 100644 index 00000000..03ee1d85 --- /dev/null +++ b/scripts/css/tables.css @@ -0,0 +1,4 @@ +/* Column with header cells */ +table.docutils tbody th.stub { + background: #eeeeee; +} From 6ae7e49522e34b32fa317756b906a784317ac829 Mon Sep 17 00:00:00 2001 From: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Date: Thu, 30 May 2019 12:30:43 +0100 Subject: [PATCH 093/128] Clarify what the client should receiving on email resending It was a little unclear what the client should do when asked to send a validation email and provides a `send_attempt` value that is not greater than previous attempts. As this is intended to be for when a client mistakenly sends the request twice, it makes logical sense that the implication was to simply resend a success value so the client doesn't error even when an email may have been sent on the first attempt. This behaviour was also mimicked in Synapse/Sydent. --- api/identity/definitions/request_email_validation.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/api/identity/definitions/request_email_validation.yaml b/api/identity/definitions/request_email_validation.yaml index 1194a18e..d6606f03 100644 --- a/api/identity/definitions/request_email_validation.yaml +++ b/api/identity/definitions/request_email_validation.yaml @@ -39,7 +39,8 @@ properties: avoid repeatedly sending the same email in the case of request retries between the POSTing user and the identity server. The client should increment this value if they desire a new - email (e.g. a reminder) to be sent. + email (e.g. a reminder) to be sent. If they do not, the server + should return a success but not resend the email. example: 1 next_link: type: string From c0c462999374a51b8430b88f7abe58a5a2b4386b Mon Sep 17 00:00:00 2001 From: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Date: Thu, 30 May 2019 12:31:26 +0100 Subject: [PATCH 094/128] Slight word change --- api/identity/definitions/request_email_validation.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/identity/definitions/request_email_validation.yaml b/api/identity/definitions/request_email_validation.yaml index d6606f03..5d1aac8a 100644 --- a/api/identity/definitions/request_email_validation.yaml +++ b/api/identity/definitions/request_email_validation.yaml @@ -40,7 +40,7 @@ properties: retries between the POSTing user and the identity server. The client should increment this value if they desire a new email (e.g. a reminder) to be sent. If they do not, the server - should return a success but not resend the email. + should response with success but not resend the email. example: 1 next_link: type: string From dda8976d2544a22cb280fce243d23107e2bd4c88 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Thu, 30 May 2019 12:33:29 +0100 Subject: [PATCH 095/128] Add changelog --- changelogs/client_server/newsfragments/2057.clarification | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/client_server/newsfragments/2057.clarification diff --git a/changelogs/client_server/newsfragments/2057.clarification b/changelogs/client_server/newsfragments/2057.clarification new file mode 100644 index 00000000..de72c201 --- /dev/null +++ b/changelogs/client_server/newsfragments/2057.clarification @@ -0,0 +1 @@ +Clarify what the client should receive upon sending an identical email validation request multiple times. From 54f74cd877eb166ca53d52bd1a16f51a6df68461 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 30 May 2019 16:46:55 -0600 Subject: [PATCH 096/128] Add Olm unwedging As per [MSC1719](https://github.com/matrix-org/matrix-doc/pull/1719) No known alterations have been made to the proposal. Implementation proof: https://github.com/matrix-org/matrix-js-sdk/pull/780 --- .../client_server/newsfragments/2059.feature | 1 + event-schemas/examples/m.dummy | 4 ++ event-schemas/schema/m.dummy | 23 +++++++++++ .../modules/end_to_end_encryption.rst | 41 ++++++++++++++++--- 4 files changed, 64 insertions(+), 5 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2059.feature create mode 100644 event-schemas/examples/m.dummy create mode 100644 event-schemas/schema/m.dummy diff --git a/changelogs/client_server/newsfragments/2059.feature b/changelogs/client_server/newsfragments/2059.feature new file mode 100644 index 00000000..fde106ce --- /dev/null +++ b/changelogs/client_server/newsfragments/2059.feature @@ -0,0 +1 @@ +Add support for Olm sessions becoming un-stuck. diff --git a/event-schemas/examples/m.dummy b/event-schemas/examples/m.dummy new file mode 100644 index 00000000..0cd39166 --- /dev/null +++ b/event-schemas/examples/m.dummy @@ -0,0 +1,4 @@ +{ + "content": {}, + "type": "m.dummy" +} diff --git a/event-schemas/schema/m.dummy b/event-schemas/schema/m.dummy new file mode 100644 index 00000000..5bebe430 --- /dev/null +++ b/event-schemas/schema/m.dummy @@ -0,0 +1,23 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + This event type is used to indicate new Olm sessions for end-to-end encryption. + Typically it is encrypted as an ``m.room.encrypted`` event, then sent as a `to-device`_ + event. + + The event does not have any content associated with it. The sending client is expected + to send a key share request shortly after this message, causing the receiving client to + process this ``m.dummy`` event as the most recent event and using the keyshare request + to set up the session. The keyshare request and ``m.dummy`` combination should result + in the original sending client receiving keys over the newly establish session. +properties: + content: + properties: {} + type: object + type: + enum: + - m.dummy + type: string +type: object diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 38a0a06b..a77dbad9 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -1,4 +1,5 @@ .. Copyright 2016 OpenMarket Ltd +.. Copyright 2019 The Matrix.org Foundation C.I.C. .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -18,7 +19,7 @@ End-to-End Encryption .. _module:e2e: Matrix optionally supports end-to-end encryption, allowing rooms to be created -whose conversation contents is not decryptable or interceptable on any of the +whose conversation contents are not decryptable or interceptable on any of the participating homeservers. Key Distribution @@ -549,6 +550,31 @@ Example: ] } + +Recovering from undecryptable messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Occasionally messages may be undecryptable by clients due to a variety of reasons. +When this happens to an Olm-encrypted message, the client should assume that the Olm +session has become corrupted and create a new one to replace it. + +.. Note:: + Megolm-encrypted messages generally do not have the same problem. Usually the key + for an undecryptable Megolm-encrypted message will come later, allowing the client + to decrypt it successfully. Olm does not have a way to recover from the failure, + making this session replacement process required. + +To establish a new session, the client sends a `m.dummy <#m-dummy>`_ to-device event +to the other party to notify them of the new session details. + +Clients should rate-limit the number of sessions it creates per device that it receives +a message from. Clients should not create a new session with another device if it has +already created on for that given device in the past 1 hour. + +Clients should attempt to mitigate loss of the undecryptable messages. For example, +Megolm sessions that were sent using the old session would have been lost. The client +can attempt to retrieve the lost sessions through ``m.room_key_request`` messages. + Messaging Algorithms -------------------- @@ -658,10 +684,13 @@ part of the ed25519 key it claims to have in the Olm payload. This is crucial when the ed25519 key corresponds to a verified device. If a client has multiple sessions established with another device, it should -use the session from which it last received a message. A client may expire old -sessions by defining a maximum number of olm sessions that it will maintain for -each device, and expiring sessions on a Least Recently Used basis. The maximum -number of olm sessions maintained per device should be at least 4. +use the session from which it last received and successfully decrypted a +message. For these purposes, a session that has not received any messages +should use its creation time as the time that it last received a message. +A client may expire old sessions by defining a maximum number of olm sessions +that it will maintain for each device, and expiring sessions on a Least Recently +Used basis. The maximum number of olm sessions maintained per device should +be at least 4. ``m.megolm.v1.aes-sha2`` ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -740,6 +769,8 @@ Events {{m_forwarded_room_key_event}} +{{m_dummy_event}} + Key management API ~~~~~~~~~~~~~~~~~~ From d48265f49be7ba8b4fd4471396dd4deb0796ec25 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 30 May 2019 16:51:24 -0600 Subject: [PATCH 097/128] typo --- event-schemas/schema/m.dummy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/event-schemas/schema/m.dummy b/event-schemas/schema/m.dummy index 5bebe430..8e4b6f94 100644 --- a/event-schemas/schema/m.dummy +++ b/event-schemas/schema/m.dummy @@ -11,7 +11,7 @@ description: |- to send a key share request shortly after this message, causing the receiving client to process this ``m.dummy`` event as the most recent event and using the keyshare request to set up the session. The keyshare request and ``m.dummy`` combination should result - in the original sending client receiving keys over the newly establish session. + in the original sending client receiving keys over the newly established session. properties: content: properties: {} From 41e07ff7d64ba3bb667532bb0f69c250ba93d149 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 30 May 2019 16:54:19 -0600 Subject: [PATCH 098/128] Fix incorrect state of leave->leave As mentioned in PR review, it is possible for moderators in a room to race at kicking someone, resulting in multiple leave events. --- event-schemas/schema/m.room.member | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/event-schemas/schema/m.room.member b/event-schemas/schema/m.room.member index 20d3cbcc..8984ac5c 100644 --- a/event-schemas/schema/m.room.member +++ b/event-schemas/schema/m.room.member @@ -40,7 +40,7 @@ description: |- "","to ``invite``","to ``join``","to ``leave``","to ``ban``","to ``knock``" "from ``invite``","No change.","User joined the room.","If the ``state_key`` is the same as the ``sender``, the user rejected the invite. Otherwise, the ``state_key`` user had their invite revoked.","User was banned.","Not implemented." "from ``join``","Must never happen.","``displayname`` or ``avatar_url`` changed.","If the ``state_key`` is the same as the ``sender``, the user left. Otherwise, the ``state_key`` user was kicked.","User was kicked and banned.","Not implemented." - "from ``leave``","New invitation sent.","User joined.","Must never happen.","User was banned.","Not implemented." + "from ``leave``","New invitation sent.","User joined.","No change.","User was banned.","Not implemented." "from ``ban``","Must never happen.","Must never happen.","User was unbanned.","No change.","Not implemented." "from ``knock``","Not implemented.","Not implemented.","Not implemented.","Not implemented.","Not implemented." From 754b19bb929601fa5253470eea5dbfc45361d81b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 31 May 2019 10:21:16 -0600 Subject: [PATCH 099/128] typo Co-Authored-By: Hubert Chathi --- specification/modules/end_to_end_encryption.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index a77dbad9..628c1a60 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -569,7 +569,7 @@ to the other party to notify them of the new session details. Clients should rate-limit the number of sessions it creates per device that it receives a message from. Clients should not create a new session with another device if it has -already created on for that given device in the past 1 hour. +already created one for that given device in the past 1 hour. Clients should attempt to mitigate loss of the undecryptable messages. For example, Megolm sessions that were sent using the old session would have been lost. The client From 57e3b152b042183f1bbb424fdaec7518c820dc66 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 31 May 2019 11:37:09 -0600 Subject: [PATCH 100/128] Move section to under Olm stuff --- .../modules/end_to_end_encryption.rst | 50 +++++++++---------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 628c1a60..e6048c28 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -550,31 +550,6 @@ Example: ] } - -Recovering from undecryptable messages -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Occasionally messages may be undecryptable by clients due to a variety of reasons. -When this happens to an Olm-encrypted message, the client should assume that the Olm -session has become corrupted and create a new one to replace it. - -.. Note:: - Megolm-encrypted messages generally do not have the same problem. Usually the key - for an undecryptable Megolm-encrypted message will come later, allowing the client - to decrypt it successfully. Olm does not have a way to recover from the failure, - making this session replacement process required. - -To establish a new session, the client sends a `m.dummy <#m-dummy>`_ to-device event -to the other party to notify them of the new session details. - -Clients should rate-limit the number of sessions it creates per device that it receives -a message from. Clients should not create a new session with another device if it has -already created one for that given device in the past 1 hour. - -Clients should attempt to mitigate loss of the undecryptable messages. For example, -Megolm sessions that were sent using the old session would have been lost. The client -can attempt to retrieve the lost sessions through ``m.room_key_request`` messages. - Messaging Algorithms -------------------- @@ -692,6 +667,31 @@ that it will maintain for each device, and expiring sessions on a Least Recently Used basis. The maximum number of olm sessions maintained per device should be at least 4. +Recovering from undecryptable messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Occasionally messages may be undecryptable by clients due to a variety of reasons. +When this happens to an Olm-encrypted message, the client should assume that the Olm +session has become corrupted and create a new one to replace it. + +.. Note:: + Megolm-encrypted messages generally do not have the same problem. Usually the key + for an undecryptable Megolm-encrypted message will come later, allowing the client + to decrypt it successfully. Olm does not have a way to recover from the failure, + making this session replacement process required. + +To establish a new session, the client sends a `m.dummy <#m-dummy>`_ to-device event +to the other party to notify them of the new session details. + +Clients should rate-limit the number of sessions it creates per device that it receives +a message from. Clients should not create a new session with another device if it has +already created one for that given device in the past 1 hour. + +Clients should attempt to mitigate loss of the undecryptable messages. For example, +Megolm sessions that were sent using the old session would have been lost. The client +can attempt to retrieve the lost sessions through ``m.room_key_request`` messages. + + ``m.megolm.v1.aes-sha2`` ~~~~~~~~~~~~~~~~~~~~~~~~ From 7a07a6b358cb15cc6ef81fba07037b440ade2879 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 2 Jun 2019 17:41:39 -0600 Subject: [PATCH 101/128] Clarify exactly what StrippedState is Fixes https://github.com/matrix-org/matrix-doc/issues/2066 The expectation everywhere is that the `sender` is required. `/initialSync` references StrippedState through a `m.room.member` event reference, and does not need editing. --- api/client-server/sync.yaml | 25 +---------- api/server-server/invites-v1.yaml | 20 +-------- api/server-server/invites-v2.yaml | 20 +-------- .../newsfragments/2067.clarification | 1 + .../newsfragments/2067.clarification | 1 + .../examples/m.room.member$invite_room_state | 19 ++------ event-schemas/examples/stripped_state.json | 18 ++++++++ event-schemas/schema/m.room.member | 19 +------- event-schemas/schema/stripped_state.yaml | 44 +++++++++++++++++++ 9 files changed, 71 insertions(+), 96 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2067.clarification create mode 100644 changelogs/server_server/newsfragments/2067.clarification create mode 100644 event-schemas/examples/stripped_state.json create mode 100644 event-schemas/schema/stripped_state.yaml diff --git a/api/client-server/sync.yaml b/api/client-server/sync.yaml index 02fddb84..3db1fa54 100644 --- a/api/client-server/sync.yaml +++ b/api/client-server/sync.yaml @@ -212,30 +212,7 @@ paths: events: description: The StrippedState events that form the invite state. items: - description: |- - A stripped down state event, with only the ``type``, ``state_key``, - ``sender``, and ``content`` keys. - properties: - content: - description: The ``content`` for the event. - title: EventContent - type: object - state_key: - description: The ``state_key`` for the event. - type: string - type: - description: The ``type`` for the event. - type: string - sender: - description: The ``sender`` for the event. - type: string - required: - - type - - state_key - - content - - sender - title: StrippedState - type: object + $ref: "definitions/event-schemas/schema/stripped_state.yaml" type: array leave: title: Left rooms diff --git a/api/server-server/invites-v1.yaml b/api/server-server/invites-v1.yaml index 2ad0f220..867d7b05 100644 --- a/api/server-server/invites-v1.yaml +++ b/api/server-server/invites-v1.yaml @@ -82,25 +82,7 @@ paths: identify the room. The recommended events to include are the join rules, canonical alias, avatar, and name of the room. items: - type: object - title: Invite Room State Event - properties: - type: - type: string - description: The type of event. - example: "m.room.join_rules" - state_key: - type: string - description: The state key for the event. May be an empty string. - example: "" - content: - type: object - description: The content for the event. - sender: - type: string - description: The sender of the event. - example: "@someone:matrix.org" - required: ['type', 'state_key', 'content', 'sender'] + $ref: "../client-server/definitions/event-schemas/schema/stripped_state.yaml" example: [ { "type": "m.room.join_rules", diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index c459a848..6d5b102e 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -83,25 +83,7 @@ paths: identify the room. The recommended events to include are the join rules, canonical alias, avatar, and name of the room. items: - type: object - title: Invite Room State Event - properties: - type: - type: string - description: The type of event. - example: "m.room.join_rules" - state_key: - type: string - description: The state key for the event. May be an empty string. - example: "" - content: - type: object - description: The content for the event. - sender: - type: string - description: The sender of the event. - example: "@someone:matrix.org" - required: ['type', 'state_key', 'content', 'sender'] + $ref: "../client-server/definitions/event-schemas/schema/stripped_state.yaml" example: [ { "type": "m.room.join_rules", diff --git a/changelogs/client_server/newsfragments/2067.clarification b/changelogs/client_server/newsfragments/2067.clarification new file mode 100644 index 00000000..cc706274 --- /dev/null +++ b/changelogs/client_server/newsfragments/2067.clarification @@ -0,0 +1 @@ +Clarify exactly what invite_room_state consists of. diff --git a/changelogs/server_server/newsfragments/2067.clarification b/changelogs/server_server/newsfragments/2067.clarification new file mode 100644 index 00000000..cc706274 --- /dev/null +++ b/changelogs/server_server/newsfragments/2067.clarification @@ -0,0 +1 @@ +Clarify exactly what invite_room_state consists of. diff --git a/event-schemas/examples/m.room.member$invite_room_state b/event-schemas/examples/m.room.member$invite_room_state index f8f05484..b60abf95 100644 --- a/event-schemas/examples/m.room.member$invite_room_state +++ b/event-schemas/examples/m.room.member$invite_room_state @@ -7,21 +7,8 @@ }, "unsigned": { "age": 1234, - "invite_room_state": [ - { - "type": "m.room.name", - "state_key": "", - "content": { - "name": "Forest of Magic" - } - }, - { - "type": "m.room.join_rules", - "state_key": "", - "content": { - "join_rule": "invite" - } - } - ] + "invite_room_state": { + "$ref": "stripped_state.json" + } } } diff --git a/event-schemas/examples/stripped_state.json b/event-schemas/examples/stripped_state.json new file mode 100644 index 00000000..9d8c1b2b --- /dev/null +++ b/event-schemas/examples/stripped_state.json @@ -0,0 +1,18 @@ +[ + { + "type": "m.room.name", + "sender": "@bob:example.org", + "state_key": "", + "content": { + "name": "Example Room" + } + }, + { + "type": "m.room.join_rules", + "sender": "@bob:example.org", + "state_key": "", + "content": { + "join_rule": "invite" + } + } +] diff --git a/event-schemas/schema/m.room.member b/event-schemas/schema/m.room.member index de14644d..f846196b 100644 --- a/event-schemas/schema/m.room.member +++ b/event-schemas/schema/m.room.member @@ -81,24 +81,7 @@ properties: invite_room_state: description: 'A subset of the state of the room at the time of the invite, if ``membership`` is ``invite``. Note that this state is informational, and SHOULD NOT be trusted; once the client has joined the room, it SHOULD fetch the live state from the server and discard the invite_room_state. Also, clients must not rely on any particular state being present here; they SHOULD behave properly (with possibly a degraded but not a broken experience) in the absence of any particular events here. If they are set on the room, at least the state for ``m.room.avatar``, ``m.room.canonical_alias``, ``m.room.join_rules``, and ``m.room.name`` SHOULD be included.' items: - description: 'A stripped down state event, with only the ``type``, ``state_key`` and ``content`` keys.' - properties: - content: - description: The ``content`` for the event. - title: EventContent - type: object - state_key: - description: The ``state_key`` for the event. - type: string - type: - description: The ``type`` for the event. - type: string - required: - - type - - state_key - - content - title: StrippedState - type: object + $ref: "stripped_state.yaml" type: array required: - membership diff --git a/event-schemas/schema/stripped_state.yaml b/event-schemas/schema/stripped_state.yaml new file mode 100644 index 00000000..ec591bf1 --- /dev/null +++ b/event-schemas/schema/stripped_state.yaml @@ -0,0 +1,44 @@ +# Copyright 2019 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Note: this, and the example, are in the `event-schemas` directory because +# the CS API uses a symlink. In order for the `m.room.member` event to +# reference this, we'd need to use relative pathing. The symlink makes this +# difficult because the schema would be at two different locations, with +# different relative pathing. + +title: StrippedState +type: object +description: |- + A stripped down state event, with only the ``type``, ``state_key``, + ``sender``, and ``content`` keys. +properties: + content: + description: The ``content`` for the event. + title: EventContent + type: object + state_key: + description: The ``state_key`` for the event. + type: string + type: + description: The ``type`` for the event. + type: string + sender: + description: The ``sender`` for the event. + type: string +required: + - type + - state_key + - content + - sender From b9c4a2561ff8692b8e960e9eb977efb27b86e57b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 2 Jun 2019 17:50:27 -0600 Subject: [PATCH 102/128] Fix examples of StrippedState in s2s spec --- api/server-server/invites-v1.yaml | 53 +++---------------------------- api/server-server/invites-v2.yaml | 53 ++++--------------------------- 2 files changed, 11 insertions(+), 95 deletions(-) diff --git a/api/server-server/invites-v1.yaml b/api/server-server/invites-v1.yaml index 867d7b05..2f08dd12 100644 --- a/api/server-server/invites-v1.yaml +++ b/api/server-server/invites-v1.yaml @@ -83,16 +83,8 @@ paths: canonical alias, avatar, and name of the room. items: $ref: "../client-server/definitions/event-schemas/schema/stripped_state.yaml" - example: [ - { - "type": "m.room.join_rules", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "join_rule": "public" - } - } - ] + example: + $ref: "../client-server/definitions/event-schemas/examples/stripped_state.json" example: { "$ref": "examples/minimal_pdu.json", "type": "m.room.member", @@ -100,26 +92,6 @@ paths: "origin": "example.org", "origin_server_ts": 1549041175876, "sender": "@someone:example.org", - "unsigned": { - "invite_room_state": [ - { - "type": "m.room.join_rules", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "join_rule": "public" - } - }, - { - "type": "m.room.name", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "name": "Cool New Room" - } - } - ] - }, "content": { "membership": "invite" }, @@ -162,24 +134,9 @@ paths: "origin_server_ts": 1549041175876, "sender": "@someone:example.org", "unsigned": { - "invite_room_state": [ - { - "type": "m.room.join_rules", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "join_rule": "public" - } - }, - { - "type": "m.room.name", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "name": "Cool New Room" - } - } - ] + "invite_room_state": { + "$ref": "../../client-server/definitions/event-schemas/examples/stripped_state.json" + } }, "content": { "membership": "invite" diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index 6d5b102e..57ca99ff 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -84,16 +84,8 @@ paths: canonical alias, avatar, and name of the room. items: $ref: "../client-server/definitions/event-schemas/schema/stripped_state.yaml" - example: [ - { - "type": "m.room.join_rules", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "join_rule": "public" - } - } - ] + example: + $ref: "../client-server/definitions/event-schemas/examples/stripped_state.json" required: ['room_version', 'event'] example: { "room_version": "2", @@ -112,25 +104,7 @@ paths: "ed25519:key_version": "SomeSignatureHere" }, } - }, - "invite_room_state": [ - { - "type": "m.room.join_rules", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "join_rule": "public" - } - }, - { - "type": "m.room.name", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "name": "Cool New Room" - } - } - ] + } } responses: 200: @@ -156,24 +130,9 @@ paths: "origin_server_ts": 1549041175876, "sender": "@someone:example.org", "unsigned": { - "invite_room_state": [ - { - "type": "m.room.join_rules", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "join_rule": "public" - } - }, - { - "type": "m.room.name", - "sender": "@someone:matrix.org", - "state_key": "", - "content": { - "name": "Cool New Room" - } - } - ] + "invite_room_state": { + "$ref": "../../client-server/definitions/event-schemas/examples/stripped_state.json" + } }, "content": { "membership": "invite" From 237d585e07b376d69e0f7e887b7135bc8250aba6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 2 Jun 2019 17:59:19 -0600 Subject: [PATCH 103/128] Shorten references to StrippedState in s2s spec --- api/server-server/invites-v1.yaml | 6 +++--- api/server-server/invites-v2.yaml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/api/server-server/invites-v1.yaml b/api/server-server/invites-v1.yaml index 2f08dd12..83aafb3a 100644 --- a/api/server-server/invites-v1.yaml +++ b/api/server-server/invites-v1.yaml @@ -82,9 +82,9 @@ paths: identify the room. The recommended events to include are the join rules, canonical alias, avatar, and name of the room. items: - $ref: "../client-server/definitions/event-schemas/schema/stripped_state.yaml" + $ref: "../../event-schemas/schema/stripped_state.yaml" example: - $ref: "../client-server/definitions/event-schemas/examples/stripped_state.json" + $ref: "../../event-schemas/examples/stripped_state.json" example: { "$ref": "examples/minimal_pdu.json", "type": "m.room.member", @@ -135,7 +135,7 @@ paths: "sender": "@someone:example.org", "unsigned": { "invite_room_state": { - "$ref": "../../client-server/definitions/event-schemas/examples/stripped_state.json" + "$ref": "../../../event-schemas/examples/stripped_state.json" } }, "content": { diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index 57ca99ff..d2cc63a2 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -83,9 +83,9 @@ paths: identify the room. The recommended events to include are the join rules, canonical alias, avatar, and name of the room. items: - $ref: "../client-server/definitions/event-schemas/schema/stripped_state.yaml" + $ref: "../../event-schemas/schema/stripped_state.yaml" example: - $ref: "../client-server/definitions/event-schemas/examples/stripped_state.json" + $ref: "../../event-schemas/examples/stripped_state.json" required: ['room_version', 'event'] example: { "room_version": "2", @@ -131,7 +131,7 @@ paths: "sender": "@someone:example.org", "unsigned": { "invite_room_state": { - "$ref": "../../client-server/definitions/event-schemas/examples/stripped_state.json" + "$ref": "../../../event-schemas/examples/stripped_state.json" } }, "content": { From 0b45f3795bb6c561266f9388f850bae460c204fd Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 2 Jun 2019 18:02:22 -0600 Subject: [PATCH 104/128] Rename example to invite_room_state This is a better representation of what it actually is --- api/server-server/invites-v1.yaml | 4 ++-- api/server-server/invites-v2.yaml | 4 ++-- .../examples/{stripped_state.json => invite_room_state.json} | 0 event-schemas/examples/m.room.member$invite_room_state | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) rename event-schemas/examples/{stripped_state.json => invite_room_state.json} (100%) diff --git a/api/server-server/invites-v1.yaml b/api/server-server/invites-v1.yaml index 83aafb3a..8e1c861d 100644 --- a/api/server-server/invites-v1.yaml +++ b/api/server-server/invites-v1.yaml @@ -84,7 +84,7 @@ paths: items: $ref: "../../event-schemas/schema/stripped_state.yaml" example: - $ref: "../../event-schemas/examples/stripped_state.json" + $ref: "../../event-schemas/examples/invite_room_state.json" example: { "$ref": "examples/minimal_pdu.json", "type": "m.room.member", @@ -135,7 +135,7 @@ paths: "sender": "@someone:example.org", "unsigned": { "invite_room_state": { - "$ref": "../../../event-schemas/examples/stripped_state.json" + "$ref": "../../../event-schemas/examples/invite_room_state.json" } }, "content": { diff --git a/api/server-server/invites-v2.yaml b/api/server-server/invites-v2.yaml index d2cc63a2..cae14bb4 100644 --- a/api/server-server/invites-v2.yaml +++ b/api/server-server/invites-v2.yaml @@ -85,7 +85,7 @@ paths: items: $ref: "../../event-schemas/schema/stripped_state.yaml" example: - $ref: "../../event-schemas/examples/stripped_state.json" + $ref: "../../event-schemas/examples/invite_room_state.json" required: ['room_version', 'event'] example: { "room_version": "2", @@ -131,7 +131,7 @@ paths: "sender": "@someone:example.org", "unsigned": { "invite_room_state": { - "$ref": "../../../event-schemas/examples/stripped_state.json" + "$ref": "../../../event-schemas/examples/invite_room_state.json" } }, "content": { diff --git a/event-schemas/examples/stripped_state.json b/event-schemas/examples/invite_room_state.json similarity index 100% rename from event-schemas/examples/stripped_state.json rename to event-schemas/examples/invite_room_state.json diff --git a/event-schemas/examples/m.room.member$invite_room_state b/event-schemas/examples/m.room.member$invite_room_state index b60abf95..2c93eb9b 100644 --- a/event-schemas/examples/m.room.member$invite_room_state +++ b/event-schemas/examples/m.room.member$invite_room_state @@ -8,7 +8,7 @@ "unsigned": { "age": 1234, "invite_room_state": { - "$ref": "stripped_state.json" + "$ref": "invite_room_state.json" } } } From 86019c9adeb05a3998a03b7e12089b151293015a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Sun, 2 Jun 2019 18:02:36 -0600 Subject: [PATCH 105/128] Skip over partial event definitions in examples --- event-schemas/check_examples.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/event-schemas/check_examples.py b/event-schemas/check_examples.py index 2baa3ef0..31daa478 100755 --- a/event-schemas/check_examples.py +++ b/event-schemas/check_examples.py @@ -106,6 +106,9 @@ def check_example_dir(exampledir, schemadir): if filename.startswith("."): # Skip over any vim .swp files. continue + if filename.endswith(".json"): + # Skip over any explicit examples (partial event definitions) + continue cwd = os.path.basename(os.path.dirname(os.path.join(root, filename))) if cwd == "core": # Skip checking the underlying definitions From 53aa8fe8ecc7e4d2028894a57395fe3e9340fc2c Mon Sep 17 00:00:00 2001 From: Bruno Windels Date: Mon, 3 Jun 2019 09:47:09 +0200 Subject: [PATCH 106/128] clarify the order events in chunk for /messages --- api/client-server/message_pagination.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/client-server/message_pagination.yaml b/api/client-server/message_pagination.yaml index c9f9d0ae..7457530b 100644 --- a/api/client-server/message_pagination.yaml +++ b/api/client-server/message_pagination.yaml @@ -103,7 +103,7 @@ paths: chunk: type: array description: |- - A list of room events. + A list of room events. The order depends on the ``dir`` parameter. For ``dir=b`` events will be in reverse-chronological order, for ``dir=f`` in chronological order, so that events start at the ``from`` point. items: type: object title: RoomEvent From 0b42a17352356eb879ee21f264dddfdcc78ef647 Mon Sep 17 00:00:00 2001 From: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Date: Mon, 3 Jun 2019 09:48:38 +0100 Subject: [PATCH 107/128] Update api/identity/definitions/request_email_validation.yaml Co-Authored-By: Travis Ralston --- api/identity/definitions/request_email_validation.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/identity/definitions/request_email_validation.yaml b/api/identity/definitions/request_email_validation.yaml index 5d1aac8a..5f15bd41 100644 --- a/api/identity/definitions/request_email_validation.yaml +++ b/api/identity/definitions/request_email_validation.yaml @@ -40,7 +40,7 @@ properties: retries between the POSTing user and the identity server. The client should increment this value if they desire a new email (e.g. a reminder) to be sent. If they do not, the server - should response with success but not resend the email. + should respond with success but not resend the email. example: 1 next_link: type: string From 49b9bfc9f6ed5041807d8d893a29f9753381ad06 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Mon, 3 Jun 2019 09:48:53 +0100 Subject: [PATCH 108/128] Move changelog to the right place --- .../newsfragments/2057.clarification | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changelogs/{client_server => identity_service}/newsfragments/2057.clarification (100%) diff --git a/changelogs/client_server/newsfragments/2057.clarification b/changelogs/identity_service/newsfragments/2057.clarification similarity index 100% rename from changelogs/client_server/newsfragments/2057.clarification rename to changelogs/identity_service/newsfragments/2057.clarification From 39144942da1b235627b652909c0f21c8f57c33ea Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 3 Jun 2019 20:39:41 -0600 Subject: [PATCH 109/128] Fix title ordering --- specification/modules/end_to_end_encryption.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index e6048c28..1fd28e93 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -668,7 +668,7 @@ Used basis. The maximum number of olm sessions maintained per device should be at least 4. Recovering from undecryptable messages -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Occasionally messages may be undecryptable by clients due to a variety of reasons. When this happens to an Olm-encrypted message, the client should assume that the Olm From 8b28972a2d38935f3c5c7953ba40dbc5c36ddbcb Mon Sep 17 00:00:00 2001 From: Bruno Windels Date: Tue, 4 Jun 2019 09:28:01 +0200 Subject: [PATCH 110/128] PR feedback --- api/client-server/message_pagination.yaml | 5 ++++- changelogs/client_server/newsfragments/2069.clarification | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 changelogs/client_server/newsfragments/2069.clarification diff --git a/api/client-server/message_pagination.yaml b/api/client-server/message_pagination.yaml index 7457530b..9a24537c 100644 --- a/api/client-server/message_pagination.yaml +++ b/api/client-server/message_pagination.yaml @@ -103,7 +103,10 @@ paths: chunk: type: array description: |- - A list of room events. The order depends on the ``dir`` parameter. For ``dir=b`` events will be in reverse-chronological order, for ``dir=f`` in chronological order, so that events start at the ``from`` point. + A list of room events. The order depends on the ``dir`` parameter. + For ``dir=b`` events will be in reverse-chronological order, + for ``dir=f`` in chronological order, so that events start + at the ``from`` point. items: type: object title: RoomEvent diff --git a/changelogs/client_server/newsfragments/2069.clarification b/changelogs/client_server/newsfragments/2069.clarification new file mode 100644 index 00000000..353b545d --- /dev/null +++ b/changelogs/client_server/newsfragments/2069.clarification @@ -0,0 +1 @@ +Clarify the order events in chunk are returned in for /messages From 042455d954699b486b5c732d848697d11fd6a5e4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 4 Jun 2019 10:58:39 -0600 Subject: [PATCH 111/128] Update changelog to appease style guidelines --- changelogs/client_server/newsfragments/2069.clarification | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelogs/client_server/newsfragments/2069.clarification b/changelogs/client_server/newsfragments/2069.clarification index 353b545d..127573a6 100644 --- a/changelogs/client_server/newsfragments/2069.clarification +++ b/changelogs/client_server/newsfragments/2069.clarification @@ -1 +1 @@ -Clarify the order events in chunk are returned in for /messages +Clarify the order events in chunk are returned in for ``/messages``. From a3364ff35712be278fc4f5914a89dbb27f41d08a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 4 Jun 2019 12:41:30 -0600 Subject: [PATCH 112/128] Spec SAS verification and the common key verification framework Reference implementations: * https://gitlab.matrix.org/matrix-org/olm/commit/94f664e7256215f33639dbbad6aaf87ada082a9f * https://github.com/matrix-org/matrix-react-sdk/pull/2461 * https://github.com/matrix-org/matrix-js-sdk/pull/818 * https://github.com/matrix-org/matrix-react-sdk/pull/2596 * https://github.com/matrix-org/matrix-js-sdk/pull/837 Proposals: * [MSC1717](https://github.com/matrix-org/matrix-doc/pull/1717) * [MSC1267](https://github.com/matrix-org/matrix-doc/issues/1267) No alterations to either proposal have been made intentionally here. --- .../client_server/newsfragments/2072.feature | 1 + data-definitions/sas-emoji.json | 66 ++++ .../examples/m.key.verification.accept | 12 + .../examples/m.key.verification.cancel | 8 + event-schemas/examples/m.key.verification.key | 7 + event-schemas/examples/m.key.verification.mac | 10 + .../examples/m.key.verification.request | 11 + .../examples/m.key.verification.start | 8 + .../m.key.verification.start$m.sas.v1 | 12 + .../schema/m.key.verification.accept | 64 ++++ .../schema/m.key.verification.cancel | 70 ++++ event-schemas/schema/m.key.verification.key | 28 ++ event-schemas/schema/m.key.verification.mac | 38 ++ .../schema/m.key.verification.request | 43 +++ event-schemas/schema/m.key.verification.start | 39 +++ .../schema/m.key.verification.start$m.sas.v1 | 69 ++++ scripts/continuserv/main.go | 2 +- .../templating/matrix_templates/sections.py | 17 + scripts/templating/matrix_templates/units.py | 20 ++ .../modules/end_to_end_encryption.rst | 325 +++++++++++++++++- 20 files changed, 847 insertions(+), 3 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2072.feature create mode 100644 data-definitions/sas-emoji.json create mode 100644 event-schemas/examples/m.key.verification.accept create mode 100644 event-schemas/examples/m.key.verification.cancel create mode 100644 event-schemas/examples/m.key.verification.key create mode 100644 event-schemas/examples/m.key.verification.mac create mode 100644 event-schemas/examples/m.key.verification.request create mode 100644 event-schemas/examples/m.key.verification.start create mode 100644 event-schemas/examples/m.key.verification.start$m.sas.v1 create mode 100644 event-schemas/schema/m.key.verification.accept create mode 100644 event-schemas/schema/m.key.verification.cancel create mode 100644 event-schemas/schema/m.key.verification.key create mode 100644 event-schemas/schema/m.key.verification.mac create mode 100644 event-schemas/schema/m.key.verification.request create mode 100644 event-schemas/schema/m.key.verification.start create mode 100644 event-schemas/schema/m.key.verification.start$m.sas.v1 diff --git a/changelogs/client_server/newsfragments/2072.feature b/changelogs/client_server/newsfragments/2072.feature new file mode 100644 index 00000000..c7d8bd76 --- /dev/null +++ b/changelogs/client_server/newsfragments/2072.feature @@ -0,0 +1 @@ +Add interactive device verification, including a common framework for device verification. diff --git a/data-definitions/sas-emoji.json b/data-definitions/sas-emoji.json new file mode 100644 index 00000000..060fbd49 --- /dev/null +++ b/data-definitions/sas-emoji.json @@ -0,0 +1,66 @@ +[ + {"number": 0, "emoji": "🐶", "description": "Dog", "unicode": "U+1F436"}, + {"number": 1, "emoji": "🐱", "description": "Cat", "unicode": "U+1F431"}, + {"number": 2, "emoji": "🦁", "description": "Lion", "unicode": "U+1F981"}, + {"number": 3, "emoji": "🐎", "description": "Horse", "unicode": "U+1F40E"}, + {"number": 4, "emoji": "🦄", "description": "Unicorn", "unicode": "U+1F984"}, + {"number": 5, "emoji": "🐷", "description": "Pig", "unicode": "U+1F437"}, + {"number": 6, "emoji": "🐘", "description": "Elephant", "unicode": "U+1F418"}, + {"number": 7, "emoji": "🐰", "description": "Rabbit", "unicode": "U+1F430"}, + {"number": 8, "emoji": "🐼", "description": "Panda", "unicode": "U+1F43C"}, + {"number": 9, "emoji": "🐓", "description": "Rooster", "unicode": "U+1F413"}, + {"number": 10, "emoji": "🐧", "description": "Penguin", "unicode": "U+1F427"}, + {"number": 11, "emoji": "🐢", "description": "Turtle", "unicode": "U+1F422"}, + {"number": 12, "emoji": "🐟", "description": "Fish", "unicode": "U+1F41F"}, + {"number": 13, "emoji": "🐙", "description": "Octopus", "unicode": "U+1F419"}, + {"number": 14, "emoji": "🦋", "description": "Butterfly", "unicode": "U+1F98B"}, + {"number": 15, "emoji": "🌷", "description": "Flower", "unicode": "U+1F337"}, + {"number": 16, "emoji": "🌳", "description": "Tree", "unicode": "U+1F333"}, + {"number": 17, "emoji": "🌵", "description": "Cactus", "unicode": "U+1F335"}, + {"number": 18, "emoji": "🍄", "description": "Mushroom", "unicode": "U+1F344"}, + {"number": 19, "emoji": "🌏", "description": "Globe", "unicode": "U+1F30F"}, + {"number": 20, "emoji": "🌙", "description": "Moon", "unicode": "U+1F319"}, + {"number": 21, "emoji": "☁️", "description": "Cloud", "unicode": "U+2601U+FE0F"}, + {"number": 22, "emoji": "🔥", "description": "Fire", "unicode": "U+1F525"}, + {"number": 23, "emoji": "🍌", "description": "Banana", "unicode": "U+1F34C"}, + {"number": 24, "emoji": "🍎", "description": "Apple", "unicode": "U+1F34E"}, + {"number": 25, "emoji": "🍓", "description": "Strawberry", "unicode": "U+1F353"}, + {"number": 26, "emoji": "🌽", "description": "Corn", "unicode": "U+1F33D"}, + {"number": 27, "emoji": "🍕", "description": "Pizza", "unicode": "U+1F355"}, + {"number": 28, "emoji": "🎂", "description": "Cake", "unicode": "U+1F382"}, + {"number": 29, "emoji": "❤️", "description": "Heart", "unicode": "U+2764U+FE0F"}, + {"number": 30, "emoji": "😀", "description": "Smiley", "unicode": "U+1F600"}, + {"number": 31, "emoji": "🤖", "description": "Robot", "unicode": "U+1F916"}, + {"number": 32, "emoji": "🎩", "description": "Hat", "unicode": "U+1F3A9"}, + {"number": 33, "emoji": "👓", "description": "Glasses", "unicode": "U+1F453"}, + {"number": 34, "emoji": "🔧", "description": "Spanner", "unicode": "U+1F527"}, + {"number": 35, "emoji": "🎅", "description": "Santa", "unicode": "U+1F385"}, + {"number": 36, "emoji": "👍", "description": "Thumbs Up", "unicode": "U+1F44D"}, + {"number": 37, "emoji": "☂️", "description": "Umbrella", "unicode": "U+2602U+FE0F"}, + {"number": 38, "emoji": "⌛", "description": "Hourglass", "unicode": "U+231B"}, + {"number": 39, "emoji": "⏰", "description": "Clock", "unicode": "U+23F0"}, + {"number": 40, "emoji": "🎁", "description": "Gift", "unicode": "U+1F381"}, + {"number": 41, "emoji": "💡", "description": "Light Bulb", "unicode": "U+1F4A1"}, + {"number": 42, "emoji": "📕", "description": "Book", "unicode": "U+1F4D5"}, + {"number": 43, "emoji": "✏️", "description": "Pencil", "unicode": "U+270FU+FE0F"}, + {"number": 44, "emoji": "📎", "description": "Paperclip", "unicode": "U+1F4CE"}, + {"number": 45, "emoji": "✂️", "description": "Scissors", "unicode": "U+2702U+FE0F"}, + {"number": 46, "emoji": "🔒", "description": "Lock", "unicode": "U+1F512"}, + {"number": 47, "emoji": "🔑", "description": "Key", "unicode": "U+1F511"}, + {"number": 48, "emoji": "🔨", "description": "Hammer", "unicode": "U+1F528"}, + {"number": 49, "emoji": "☎️", "description": "Telephone", "unicode": "U+260EU+FE0F"}, + {"number": 50, "emoji": "🏁", "description": "Flag", "unicode": "U+1F3C1"}, + {"number": 51, "emoji": "🚂", "description": "Train", "unicode": "U+1F682"}, + {"number": 52, "emoji": "🚲", "description": "Bicycle", "unicode": "U+1F6B2"}, + {"number": 53, "emoji": "✈️", "description": "Aeroplane", "unicode": "U+2708U+FE0F"}, + {"number": 54, "emoji": "🚀", "description": "Rocket", "unicode": "U+1F680"}, + {"number": 55, "emoji": "🏆", "description": "Trophy", "unicode": "U+1F3C6"}, + {"number": 56, "emoji": "⚽", "description": "Ball", "unicode": "U+26BD"}, + {"number": 57, "emoji": "🎸", "description": "Guitar", "unicode": "U+1F3B8"}, + {"number": 58, "emoji": "🎺", "description": "Trumpet", "unicode": "U+1F3BA"}, + {"number": 59, "emoji": "🔔", "description": "Bell", "unicode": "U+1F514"}, + {"number": 60, "emoji": "⚓", "description": "Anchor", "unicode": "U+2693"}, + {"number": 61, "emoji": "🎧", "description": "Headphones", "unicode": "U+1F3A7"}, + {"number": 62, "emoji": "📁", "description": "Folder", "unicode": "U+1F4C1"}, + {"number": 63, "emoji": "📌", "description": "Pin", "unicode": "U+1F4CC"} +] diff --git a/event-schemas/examples/m.key.verification.accept b/event-schemas/examples/m.key.verification.accept new file mode 100644 index 00000000..98e89c06 --- /dev/null +++ b/event-schemas/examples/m.key.verification.accept @@ -0,0 +1,12 @@ +{ + "type": "m.key.verification.accept", + "content": { + "transaction_id": "S0meUniqueAndOpaqueString", + "method": "m.sas.v1", + "key_agreement_protocol": "curve25519", + "hash": "sha256", + "message_authentication_code": "hkdf-hmac-sha256", + "short_authentication_string": ["decimal", "emoji"], + "commitment": "fQpGIW1Snz+pwLZu6sTy2aHy/DYWWTspTJRPyNp0PKkymfIsNffysMl6ObMMFdIJhk6g6pwlIqZ54rxo8SLmAg" + } +} diff --git a/event-schemas/examples/m.key.verification.cancel b/event-schemas/examples/m.key.verification.cancel new file mode 100644 index 00000000..9d78f67c --- /dev/null +++ b/event-schemas/examples/m.key.verification.cancel @@ -0,0 +1,8 @@ +{ + "type": "m.key.verification.cancel", + "content": { + "transaction_id": "S0meUniqueAndOpaqueString", + "code": "m.user", + "reason": "User rejected the key verification request" + } +} diff --git a/event-schemas/examples/m.key.verification.key b/event-schemas/examples/m.key.verification.key new file mode 100644 index 00000000..608a2ebd --- /dev/null +++ b/event-schemas/examples/m.key.verification.key @@ -0,0 +1,7 @@ +{ + "type": "m.key.verification.key", + "content": { + "transaction_id": "S0meUniqueAndOpaqueString", + "key": "fQpGIW1Snz+pwLZu6sTy2aHy/DYWWTspTJRPyNp0PKkymfIsNffysMl6ObMMFdIJhk6g6pwlIqZ54rxo8SLmAg" + } +} diff --git a/event-schemas/examples/m.key.verification.mac b/event-schemas/examples/m.key.verification.mac new file mode 100644 index 00000000..c77c3a8d --- /dev/null +++ b/event-schemas/examples/m.key.verification.mac @@ -0,0 +1,10 @@ +{ + "type": "m.key.verification.mac", + "content": { + "transaction_id": "S0meUniqueAndOpaqueString", + "keys": "2Wptgo4CwmLo/Y8B8qinxApKaCkBG2fjTWB7AbP5Uy+aIbygsSdLOFzvdDjww8zUVKCmI02eP9xtyJxc/cLiBA", + "mac": { + "ed25519:ABCDEF": "fQpGIW1Snz+pwLZu6sTy2aHy/DYWWTspTJRPyNp0PKkymfIsNffysMl6ObMMFdIJhk6g6pwlIqZ54rxo8SLmAg" + } + } +} diff --git a/event-schemas/examples/m.key.verification.request b/event-schemas/examples/m.key.verification.request new file mode 100644 index 00000000..258471d2 --- /dev/null +++ b/event-schemas/examples/m.key.verification.request @@ -0,0 +1,11 @@ +{ + "type": "m.key.verification.request", + "content": { + "from_device": "AliceDevice2", + "transaction_id": "S0meUniqueAndOpaqueString", + "methods": [ + "m.sas.v1" + ], + "timestamp": 1559598944869 + } +} diff --git a/event-schemas/examples/m.key.verification.start b/event-schemas/examples/m.key.verification.start new file mode 100644 index 00000000..52f16150 --- /dev/null +++ b/event-schemas/examples/m.key.verification.start @@ -0,0 +1,8 @@ +{ + "type": "m.key.verification.start", + "content": { + "from_device": "BobDevice1", + "transaction_id": "S0meUniqueAndOpaqueString", + "method": "m.sas.v1" + } +} diff --git a/event-schemas/examples/m.key.verification.start$m.sas.v1 b/event-schemas/examples/m.key.verification.start$m.sas.v1 new file mode 100644 index 00000000..dae1d405 --- /dev/null +++ b/event-schemas/examples/m.key.verification.start$m.sas.v1 @@ -0,0 +1,12 @@ +{ + "type": "m.key.verification.start", + "content": { + "from_device": "BobDevice1", + "transaction_id": "S0meUniqueAndOpaqueString", + "method": "m.sas.v1", + "key_agreement_protocols": ["curve25519"], + "hashes": ["sha256"], + "message_authentication_codes": ["hkdf-hmac-sha256"], + "short_authentication_string": ["decimal", "emoji"] + } +} diff --git a/event-schemas/schema/m.key.verification.accept b/event-schemas/schema/m.key.verification.accept new file mode 100644 index 00000000..e52df39e --- /dev/null +++ b/event-schemas/schema/m.key.verification.accept @@ -0,0 +1,64 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + Accepts a previously sent ``m.key.verification.start`` messge. Typically sent as a + `to-device`_ event. +properties: + content: + properties: + transaction_id: + type: string + description: |- + An opaque identifier for the verification process. Must be the same as + the one used for the ``m.key.verification.start`` message. + method: + type: string + enum: ["m.sas.v1"] + description: |- + The verification method to use. Must be ``m.sas.v1``. + key_agreement_protocol: + type: string + description: |- + The key agreement protocol the device is choosing to use, out of the + options in the ``m.key.verification.start`` message. + hash: + type: string + description: |- + The hash method the device is choosing to use, out of the options in + the ``m.key.verification.start`` message. + message_authentication_code: + type: string + description: |- + The message authentication code the device is choosing to use, out of + the options in the ``m.key.verification.start`` message. + short_authentication_string: + type: array + description: |- + The SAS methods both devices involved in the verification process + understand. Must be a subset of the options in the ``m.key.verification.start`` + message. + items: + type: string + enum: ["decimal", "emoji"] + commitment: + type: string + description: |- + The hash (encoded as unpadded base64) of the concatenation of the device's + ephemeral public key (encoded as unpadded base64) and the canonical JSON + representation of the ``m.key.verification.start`` message. + required: + - transaction_id + - method + - key_agreement_protocol + - hash + - message_authentication_code + - short_authentication_string + - commitment + type: object + type: + enum: + - m.key.verification.accept + type: string +type: object diff --git a/event-schemas/schema/m.key.verification.cancel b/event-schemas/schema/m.key.verification.cancel new file mode 100644 index 00000000..36ffc9ea --- /dev/null +++ b/event-schemas/schema/m.key.verification.cancel @@ -0,0 +1,70 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + Cancels a key verification process/request. Typically sent as a `to-device`_ event. +properties: + content: + properties: + transaction_id: + type: string + description: |- + The opaque identifier for the verification process/request. + reason: + type: string + description: |- + A human readable description of the ``code``. The client should only rely on this + string if it does not understand the ``code``. + code: + type: string + # Note: this is not an enum because we go into detail about the different + # error codes. If we made this an enum, we'd be repeating information. + # Also, we can't put a real bulleted list in here because the HTML2RST parser + # cuts the text at weird points, breaking the list completely. + description: |- + The error code for why the process/request was cancelled by the user. Error + codes should use the Java package naming convention if not in the following + list: + + ``m.user``: The user cancelled the verification. + + ``m.timeout``: The verification process timed out. Verification processes + can define their own timeout parameters. + + ``m.unknown_transaction``: The device does not know about the given transaction + ID. + + ``m.unknown_method``: The device does not know how to handle the requested + method. This should be sent for ``m.key.verification.start`` messages and + messages defined by individual verification processes. + + ``m.unexpected_message``: The device received an unexpected message. Typically + raised when one of the parties is handling the verification out of order. + + ``m.key_mismatch``: The key was not verified. + + ``m.user_mismatch``: The expected user did not match the user verified. + + ``m.invalid_message``: The message received was invalid. + + ``m.accepted``: A ``m.key.verification.request`` was accepted by a different + device. The device receiving this error can ignore the verification request. + + Clients should be careful to avoid error loops. For example, if a device sends + an incorrect message and the client returns ``m.invalid_message`` to which it + gets an unexpected response with ``m.unexpected_message``, the client should not + respond again with ``m.unexpected_message`` to avoid the other device potentially + sending another error response. + + .. The above blank line is important for RST. + required: + - transaction_id + - code + - reason + type: object + type: + enum: + - m.key.verification.cancel + type: string +type: object diff --git a/event-schemas/schema/m.key.verification.key b/event-schemas/schema/m.key.verification.key new file mode 100644 index 00000000..6dc4954b --- /dev/null +++ b/event-schemas/schema/m.key.verification.key @@ -0,0 +1,28 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + Sends the ephemeral public key for a device to the partner device. Typically sent as a + `to-device`_ event. +properties: + content: + properties: + transaction_id: + type: string + description: |- + An opaque identifier for the verification process. Must be the same as + the one used for the ``m.key.verification.start`` message. + key: + type: string + description: |- + The device's ephemeral public key, encoded as unpadded base64. + required: + - transaction_id + - key + type: object + type: + enum: + - m.key.verification.key + type: string +type: object diff --git a/event-schemas/schema/m.key.verification.mac b/event-schemas/schema/m.key.verification.mac new file mode 100644 index 00000000..769ebe15 --- /dev/null +++ b/event-schemas/schema/m.key.verification.mac @@ -0,0 +1,38 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + Sends the MAC of a device's key to the partner device. Typically sent as a + `to-device`_ event. +properties: + content: + properties: + transaction_id: + type: string + description: |- + An opaque identifier for the verification process. Must be the same as + the one used for the ``m.key.verification.start`` message. + mac: + type: object + description: |- + A map of the key ID to the MAC of the key, using the algorithm in the + verification process. The MAC is encoded as unpadded base64. + additionalProperties: + type: string + description: The key's MAC, encoded as unpadded base64. + keys: + type: string + description: |- + The MAC of the comma-separated, sorted, list of key IDs given in the ``mac`` + property, encoded as unpadded base64. + required: + - transaction_id + - mac + - keys + type: object + type: + enum: + - m.key.verification.mac + type: string +type: object diff --git a/event-schemas/schema/m.key.verification.request b/event-schemas/schema/m.key.verification.request new file mode 100644 index 00000000..c9efa14e --- /dev/null +++ b/event-schemas/schema/m.key.verification.request @@ -0,0 +1,43 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + Requests a key verification with another user's devices. Typically sent as a + `to-device`_ event. +properties: + content: + properties: + from_device: + type: string + description: |- + The device ID which is initiating the request. + transaction_id: + type: string + description: |- + An opaque identifier for the verification request. Must be unique + with respect to the devices involved. + methods: + type: array + description: |- + The verification methods supported by the sender. + items: + type: string + timestamp: + type: integer + format: int64 + description: |- + The POSIX timestamp in milliseconds for when the request was made. If + the request is in the future by more than 5 minutes or more than 10 + minutes in the past, the message should be ignored by the receiver. + required: + - from_device + - transaction_id + - methods + - timestamp + type: object + type: + enum: + - m.key.verification.request + type: string +type: object diff --git a/event-schemas/schema/m.key.verification.start b/event-schemas/schema/m.key.verification.start new file mode 100644 index 00000000..ad59d6c7 --- /dev/null +++ b/event-schemas/schema/m.key.verification.start @@ -0,0 +1,39 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + Begins a key verification process. Typically sent as a `to-device`_ event. +properties: + content: + properties: + from_device: + type: string + description: |- + The device ID which is initiating the process. + transaction_id: + type: string + description: |- + An opaque identifier for the verification process. Must be unique + with respect to the devices involved. Must be the same as the + ``transaction_id`` given in the ``m.key.verification.request`` + if this process is originating from a request. + method: + type: string + description: |- + The verification method to use. + next_method: + type: string + description: |- + Optional method to use to verify the other user's key with. Applicable + when the ``method`` chosen only verifies one user's key. + required: + - from_device + - transaction_id + - method + type: object + type: + enum: + - m.key.verification.start + type: string +type: object diff --git a/event-schemas/schema/m.key.verification.start$m.sas.v1 b/event-schemas/schema/m.key.verification.start$m.sas.v1 new file mode 100644 index 00000000..867ca820 --- /dev/null +++ b/event-schemas/schema/m.key.verification.start$m.sas.v1 @@ -0,0 +1,69 @@ +--- +allOf: + - $ref: core-event-schema/event.yaml + +description: |- + Begins a SAS key verification process. Typically sent as a `to-device`_ event. +properties: + content: + properties: + from_device: + type: string + description: |- + The device ID which is initiating the process. + transaction_id: + type: string + description: |- + An opaque identifier for the verification process. Must be unique + with respect to the devices involved. Must be the same as the + ``transaction_id`` given in the ``m.key.verification.request`` + if this process is originating from a request. + method: + type: string + enum: ["m.sas.v1"] + description: |- + The verification method to use. Must be ``m.sas.v1``. + key_agreement_protocols: + type: array + description: |- + The key agreement protocols the sending device understands. Must + include at least ``curve25519``. + items: + type: string + hashes: + type: array + description: |- + The hash methods the sending device understands. Must include at least + ``sha256``. + items: + type: string + message_authentication_codes: + type: array + description: |- + The message authentication codes that the sending device understands. + Must include at least ``hkdf-hmac-sha256``. + items: + type: string + short_authentication_string: + type: array + description: |- + The SAS methods the sending device (and the sending device's user) + understands. Must include at least ``decimal``. Optionally can include + ``emoji``. + items: + type: string + enum: ["decimal", "emoji"] + required: + - from_device + - transaction_id + - method + - key_agreement_protocols + - hashes + - message_authentication_codes + - short_authentication_string + type: object + type: + enum: + - m.key.verification.start + type: string +type: object diff --git a/scripts/continuserv/main.go b/scripts/continuserv/main.go index 2ef6fed9..1bd07e6e 100644 --- a/scripts/continuserv/main.go +++ b/scripts/continuserv/main.go @@ -52,7 +52,7 @@ func main() { walker := makeWalker(dir, w) paths := []string{"api", "changelogs", "event-schemas", "scripts", - "specification"} + "specification", "schemas", "data-definitions"} for _, p := range paths { filepath.Walk(path.Join(dir, p), walker) diff --git a/scripts/templating/matrix_templates/sections.py b/scripts/templating/matrix_templates/sections.py index c88959ed..7000916b 100644 --- a/scripts/templating/matrix_templates/sections.py +++ b/scripts/templating/matrix_templates/sections.py @@ -18,6 +18,7 @@ import inspect import json import os import logging +import re logger = logging.getLogger(__name__) @@ -225,3 +226,19 @@ class MatrixSections(Sections): examples=swagger_def['examples'], title_kind=subtitle_title_char) return rendered + + def render_sas_emoji_table(self): + emoji = self.units.get("sas_emoji") + rendered = ".. csv-table::\n" + rendered += " :header: \"Number\", \"Emoji\", \"Unicode\", \"Description\"\n" + rendered += " :widths: 10, 10, 15, 20\n" + rendered += "\n" + for row in emoji: + rendered += " %d, \"%s\", \"``%s``\", \"%s\"\n" % ( + row['number'], + row['emoji'], + row['unicode'], + row['description'], + ) + rendered += "\n" + return rendered diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index ddb65efe..fe3ba5d2 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -59,6 +59,8 @@ TARGETS = os.path.join(matrix_doc_dir, "specification/targets.yaml") ROOM_EVENT = "core-event-schema/room_event.yaml" STATE_EVENT = "core-event-schema/state_event.yaml" +SAS_EMOJI_JSON = os.path.join(matrix_doc_dir, "data-definitions/sas-emoji.json") + logger = logging.getLogger(__name__) # a yaml Loader which loads mappings into OrderedDicts instead of regular @@ -1088,3 +1090,21 @@ class MatrixUnits(Units): "string": git_version, "revision": git_commit } + + def load_sas_emoji(self): + with open(SAS_EMOJI_JSON, 'r', encoding='utf-8') as sas_json: + emoji = json.load(sas_json) + + # Verify the emoji matches the unicode + for c in emoji: + e = c['emoji'] + logger.info("Checking emoji %s (%s)", e, c['description']) + u = re.sub(r'U\+([0-9a-fA-F]+)', lambda m: chr(int(m.group(1), 16)), c['unicode']) + if e != u: + raise Exception("Emoji %s should be %s not %s" % ( + c['description'], + repr(e), + c['unicode'], + )) + + return emoji diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 1fd28e93..4bd12b71 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -396,8 +396,8 @@ recommended that clients provide mechanisms by which the user can see: Ed25519 signing key for each device, again encoded using unpadded Base64. Alice can then meet Bob in person, or contact him via some other trusted -medium, and ask him to read out the Ed25519 key shown on his device. She -compares this with the value shown for his device on her client. +medium, and use `SAS Verification`_ or ask him to read out the Ed25519 key +shown on his device, comparing it to the one shown on Alice's device. Device verification may reach one of several conclusions. For example: @@ -423,6 +423,327 @@ Device verification may reach one of several conclusions. For example: decrypted by such a device. For the Olm protocol, this is documented at https://matrix.org/git/olm/about/docs/signing.rst. + +Key verification framework +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Verifying keys manually by reading out the Ed25519 key is not very user friendly, +and can lead to errors. In order to help mitigate errors, and to make the process +eaiser for users, some verification methods are supported by the specification. +The methods all use a common framework for negotiating the key verification. + +To use this framework, Alice's client would send ``m.key.verification.request`` +events to Bob's devices. All of the ``to_device`` messages sent to Bob MUST have +the same ``transaction_id`` to indicate they are part of the same request. This +allows Bob to reject the request on one device, and have it apply to all of his +devices. Similarly, it allows Bob to process the verification on one device without +having to involve all of his devices. + +When Bob's device receives a ``m.key.verification.request``, it should prompt Bob +to verify keys with Alice using one of the supported methods in the request. If +Bob's device does not understand any of the methods, it should not cancel the request +as one of his other devices may support the request. Instead, Bob's device should +tell Bob that an unsupported method was used for starting key verification. The +prompt for Bob to accept/reject Alice's request (or the unsupported method prompt) +should be automatically dismissed 10 minutes after the ``timestamp`` field or 2 +minutes after Bob's client receives the message, whichever comes first, if Bob +does not interact with the prompt. The prompt should additionally be hidden if +an appropriate ``m.key.verification.cancel`` message is received. + +If Bob rejects the request, Bob's client must send a ``m.key.verification.cancel`` +message to Alice's device. Upon receipt, Alice's device should tell her that Bob +does not want to verify her device and send ``m.key.verification.cancel`` messages +to all of Bob's devices to notify them that the request was rejected. + +If Bob accepts the request, Bob's device starts the key verification process by +sending a ``m.key.verification.start`` message to Alice's device. Upon receipt +of this message, Alice's device should send a ``m.key.verification.cancel`` message +to all of Bob's other devices to indicate the process has been started. The start +message must use the same ``transaction_id`` from the original key verification +request if it is in response to the request. The start message can be sent indepdently +of any request. + +Individual verification methods may add additional steps, events, and properties to +the verification messages. Event types for methods defined in this specification must +be under the ``m.key.verification`` namespace and any other event types must be namespaced +according to the Java package naming convention. + +Any of Alice's or Bob's devices can cancel the key verification request or process +at any time with a ``m.key.verification.cancel`` message to all applicable devices. + +This framework yields the following handshake, assuming both Alice and Bob each have +2 devices, Bob's first device accepts the key verification request, and Alice's second +device initiates the request. Note how Alice's first device is not involved in the +request or verification process. + +:: + + +---------------+ +---------------+ +-------------+ +-------------+ + | AliceDevice1 | | AliceDevice2 | | BobDevice1 | | BobDevice2 | + +---------------+ +---------------+ +-------------+ +-------------+ + | | | | + | | m.key.verification.request | | + | |---------------------------------->| | + | | | | + | | m.key.verification.request | | + | |-------------------------------------------------->| + | | | | + | | m.key.verification.start | | + | |<----------------------------------| | + | | | | + | | m.key.verification.cancel | | + | |-------------------------------------------------->| + | | | | + + +After the handshake, the verification process begins. + +{{m_key_verification_request_event}} + +{{m_key_verification_start_event}} + +{{m_key_verification_cancel_event}} + + +.. _`SAS Verification`: + +Short Authentication String (SAS) verification +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +SAS verification is a user-friendly key verification process built off the common +framework outlined above. SAS verification is intended to be a highly interactive +process for users, and as such exposes verfiication methods which are easier for +users to use. + +The verification process is heavily inspired by Phil Zimmerman's ZRTP key agreement +handshake. A key part of key agreement in ZRTP is the hash commitment: the party that +begins the Diffie-Hellman key sharing sends a hash of their part of the Diffie-Hellman +exchange, and does not send their part of the Diffie-Hellman exchange until they have +received the other party's part. Thus an attacker essentially only has one attempt to +attack the Diffie-Hellman exchange, and hence we can verify fewer bits while still +achieving a high degree of security: if we verify n bits, then an attacker has a 1 in +2\ :sup:`n` chance of success. For example, if we verify 40 bits, then an attacker has +a 1 in 1,099,511,627,776 chance (or less than 1 in 1012 chance) of success. A failed +attack would result in a mismatched Short Authentication String, alerting users to the +attack. + +The verification process takes place over `to-device`_ messages in two phases: + +1. Key agreement phase (based on `ZRTP key agreement `_). +#. Key verification phase (based on HMAC). + +The process between Alice and Bob verifying each other would be: + +.. |AlicePublicKey| replace:: :math:`K_{A}^{public}` +.. |AlicePrivateKey| replace:: :math:`K_{A}^{private}` +.. |AliceCurve25519| replace:: :math:`K_{A}^{private}K_{A}^{public}` +.. |BobPublicKey| replace:: :math:`K_{B}^{public}` +.. |BobPrivateKey| replace:: :math:`K_{B}^{private}` +.. |BobCurve25519| replace:: :math:`K_{B}^{private}K_{B}^{public}` +.. |AliceBobCurve25519| replace:: :math:`K_{A}^{private}K_{B}^{public}` +.. |BobAliceCurve25519| replace:: :math:`K_{B}^{private}K_{A}^{public}` +.. |AliceBobECDH| replace:: :math:`ECDH(K_{A}^{private},K_{B}^{public})` + +1. Alice and Bob establish a secure connection, likely meeting in-person. "Secure" + here means that either party cannot be impersonated, not explicit secrecy. +#. Alice and Bob communicate which devices they'd like to verify with each other. +#. Alice selects Bob's device from the device list and begins verification. +#. Alice's client ensures it has a copy of Bob's device key. +#. Alice's device sends Bob's device a ``m.key.verification.start`` message. +#. Bob's device receives the message and selects a key agreement protocol, hash + algorithm, message authentication code, and SAS method supported by Alice's + device. +#. Bob's device ensures it has a copy of Alice's device key. +#. Bob's device creates an ephemeral Curve25519 key pair (|BobCurve25519|), and + calculates the hash (using the chosen algorithm) of the public key |BobPublicKey|. +#. Bob's device replies to Alice's device with a ``m.key.verification.accept`` message. +#. Alice's device receives Bob's message and stores the commitment hash for later use. +#. Alice's device creates an ephemeral Curve25519 key pair (|AliceCurve25519|) and + replies to Bob's device with a ``m.key.verification.key``, sending only the public + key |AlicePublicKey|. +#. Bob's device receives Alice's message and replies with its own ``m.key.verification.key`` + message containing its public key |BobPublicKey|. +#. Alice's device receives Bob's message and verifies the commitment hash from earlier + matches the hash of the key Bob's device just sent and the content of Alice's + ``m.key.verification.start`` message. +#. Both Alice and Bob's devices perform an Elliptic-curve Diffie-Hellman (|AliceBobECDH|), + using the result as the shared secret. +#. Both Alice and Bob's devices display a SAS to their users, which is derived + from the shared key using one of the methods in this section. If multiple SAS + methods are available, clients should allow the users to select a method. +#. Alice and Bob compare the strings shown by their devices, and tell their devices if + they match or not. +#. Assuming they match, Alice and Bob's devices calculate the HMAC of their own device + and a comma-separated sorted list of of the key IDs that they wish the other user + to verify. HMAC is defined in RFC 2104, and SHA-256 as the hash function. The key for + the HMAC is different for each item and is calculated by generating 32 bytes (256 bits) + using `the key verification HKDF <#SAS-HKDF>`_. +#. Alice's device sends Bob's device a ``m.key.verification.mac`` message containing the + MAC of Alice's device keys and the MAC of her key IDs to be verified. Bob's device does + the same for Bob's device keys and key IDs concurrently with Alice. +#. When the other device receives the ``m.key.verification.mac`` message, the device + calculates the HMAC of its copies of the other device's keys given in the message, + as well as the HMAC of the comma-seperated, sorted, list of key IDs in the message. + The device compares these with the HMAC values given in the message, and if everything + matches then the device keys are verified. + +The wire protocol looks like the following between Alice and Bob's devices:: + + +-------------+ +-----------+ + | AliceDevice | | BobDevice | + +-------------+ +-----------+ + | | + | m.key.verification.start | + |-------------------------------->| + | | + | m.key.verification.accept | + |<--------------------------------| + | | + | m.key.verification.key | + |-------------------------------->| + | | + | m.key.verification.key | + |<--------------------------------| + | | + | m.key.verification.mac | + |-------------------------------->| + | | + | m.key.verification.mac | + |<--------------------------------| + | | + +Error and exception handling +<<<<<<<<<<<<<<<<<<<<<<<<<<<< + +At any point the interactive verfication can go wrong. The following describes what +to do when an error happens: + +* Alice or Bob can cancel the verification at any time. A ``m.key.verification.cancel`` + message must be sent to signify the cancelation. +* The verification can time out. Clients should time out a verification that does not + complete within 5 minutes. Additionally, clients should expire a ``transaction_id`` + which goes unused for 5 minutes after having last sent/received it. The client should + inform the user that the verification timed out, and send an appropriate ``m.key.verification.cancel`` + message to the other device. +* When the same device attempts to intiate multiple verification attempts, cancel all + attempts with that device. +* When a device receives an unknown ``transaction_id``, it should send an appropriate + ``m.key.verfication.cancel`` message to the other device indicating as such. This + does not apply for inbound ``m.key.verification.start`` or ``m.key.verification.cancel`` + messages. +* If the two devices do not share a common key share, hash, HMAC, or SAS method then + the device should notify the other device with an appropriate ``m.key.verification.cancel`` + message. +* If the user claims the Short Authentication Strings do not match, the device should + send an appropriate ``m.key.verification.cancel`` message to the other device. +* If the device receives a message out of sequence or that it was not expecting, it should + notify the other device with an appropriate ``m.key.verification.cancel`` message. + + +Verification messages specific to SAS +<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< + +Building off the common framework, the following events are involved in SAS verification. + +The ``m.key.verification.cancel`` event is unchanged, however the following error codes +are used in addition to those already specified: + +* ``m.unknown_method``: The devices are unable to agree on the key agreement, hash, MAC, + or SAS method. +* ``m.mismatched_commitment``: The hash commitment did not match. +* ``m.mismatched_sas``: The SAS did not match. + + +{{m_key_verification_start_m_sas_v1_event}} + +{{m_key_verification_accept_event}} + +{{m_key_verification_key_event}} + +{{m_key_verification_mac_event}} + + +.. _`SAS-HKDF`: + +HKDF calculation +<<<<<<<<<<<<<<<< + +In all of the SAS methods, HKDF is as defined in RFC 5869 and uses the previously +agreed upon hash function for the hash function. The shared secret is supplied +as the input keying material. No salt is used, and the input parameter is the +concatenation of: + + * The string ``MATRIX_KEY_VERIFICATION_SAS``. + * The Matrix ID of the user who sent the ``m.key.verification.start`` message. + * The Device ID of the device which sent the ``m.key.verification.start`` message. + * The Matrix ID of the user who sent the ``m.key.verification.accept`` message. + * The Device ID of the device which sent the ``m.key.verification.accept`` message. + * The ``transaction_id`` being used. + +HKDF is used over the plain shared secret as it results in a harder attack +as well as more uniform data to work with. + +For verification of each party's device keys, HKDF is as defined in RFC 5869 and +uses SHA-256 as the hash function. The shared secret is supplied as the input keying +material. No salt is used, and in the input parameter is the concatenation of: + + * The string ``MATRIX_KEY_VERIFICATION_MAC``. + * The Matrix ID of the user whose key is being MAC-ed. + * The Device ID of the device sending the MAC. + * The Matrix ID of the other user. + * The Device ID of the device receiving the MAC. + * The ``transaction_id`` being used. + * The Key ID of the key being MAC-ed, or the string ``KEY_IDS`` if the item + being MAC-ed is the list of key IDs. + +SAS method: ``decimal`` +<<<<<<<<<<<<<<<<<<<<<<< + +Generate 5 bytes using `HKDF <#SAS-HKDF>`_ then take sequences of 13 bits to +convert to decimal numbers (resulting in 3 numbers between 0 and 8191 inclusive +each). Add 1000 to each calculated number. + +The bitwise operations to get the numbers given the 5 bytes +:math:`B_{0}, B_{1}, B_{2}, B_{3}, B_{4}` would be: + +* First: :math:`(B_{0} \ll 5 | B_{1} \gg 3) + 1000` +* Second: :math:`(B_{1} \& 0x7 | B_{2} \ll 2 | B_{3} \gg 6) + 1000` +* Third: :math:`((B_{3} \& 0x3F) \ll 7 | B_{4} \gg 1) + 1000` + +The digits are displayed to the user either with an appropriate separator, +such as dashes, or with the numbers on individual lines. + +SAS method: ``emoji`` +<<<<<<<<<<<<<<<<<<<<< + +Generate 6 bytes using `HKDF <#SAS-HKDF>`_ then split the first 42 bits into +7 groups of 6 bits, similar to how one would base64 encode something. Convert +each group of 6 bits to a number and use the following table to get the corresponding +emoji: + +{{sas_emoji_table}} + +.. Note:: + This table is available as JSON at + https://github.com/matrix-org/matrix-doc/blob/master/data-definitions/sas-emoji.json + +.. admonition:: Rationale + + The emoji above were chosen to: + + * Be recognisable without colour. + * Be recognisable at a small size. + * Be recognisable by most cultures. + * Be distinguishable from each other. + * Easily described by a few words. + * Avoid symbols with negative connotations. + * Be likely similar across multiple platforms. + +Clients SHOULD show the emoji with the descriptions from the table, or appropriate +translation of those descriptions. Client authors SHOULD collaborate to create a +common set of translations for all languages. + + .. section name changed, so make sure that old links keep working .. _key-sharing: From fc4965f2748a97312cfe9930aa456ba0caea7302 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 5 Jun 2019 10:40:51 -0600 Subject: [PATCH 113/128] Stronger spec words --- api/client-server/registration.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/client-server/registration.yaml b/api/client-server/registration.yaml index 10b661a3..093bac6b 100644 --- a/api/client-server/registration.yaml +++ b/api/client-server/registration.yaml @@ -62,8 +62,8 @@ paths: `Relationship between access tokens and devices`_. When registering a guest account, all parameters in the request body - with the exception of ``initial_device_display_name`` are ignored by - the server. The server will pick a ``device_id`` for the account + with the exception of ``initial_device_display_name`` MUST BE ignored + by the server. The server MUST pick a ``device_id`` for the account regardless of input. operationId: register parameters: From 7ce1ff21351d61cf9cac88637ac9ac389ee12255 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 5 Jun 2019 20:30:21 -0600 Subject: [PATCH 114/128] Clarify which servers are supposed to sign events Fixes https://github.com/matrix-org/matrix-doc/issues/2074 --- .../server_server/newsfragments/2081.clarification | 1 + specification/server_server_api.rst | 14 +++++++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) create mode 100644 changelogs/server_server/newsfragments/2081.clarification diff --git a/changelogs/server_server/newsfragments/2081.clarification b/changelogs/server_server/newsfragments/2081.clarification new file mode 100644 index 00000000..fd291273 --- /dev/null +++ b/changelogs/server_server/newsfragments/2081.clarification @@ -0,0 +1 @@ +Clarify which servers are supposed to sign events. diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index b765e36a..812f0ffc 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -421,9 +421,8 @@ must ensure that the event: Further details of these checks, and how to handle failures, are described below. -.. TODO: - Flesh this out a bit more, and probably change the doc to group the various - checks in one place, rather than have them spread out. +The `Signing Events <#signing-events>`_ section has more information on which hashes +and signatures are expected on events, and how to calculate them. Definitions @@ -1099,6 +1098,15 @@ originating server, following the algorithm described in `Checking for a signatu Note that this step should succeed whether we have been sent the full event or a redacted copy. +The signatures expected on an event are: + +* The sender's server, unless the invite was created as a result of 3rd party invite. + The sender must already match the 3rd party invite, and the server which actually + sends the event may be a different server. +* For room versions 1 and 2, the server which created the ``event_id``. Other room + versions do not track the ``event_id`` over federation and therefore do not need + a signature from those servers. + If the signature is found to be valid, the expected content hash is calculated as described below. The content hash in the ``hashes`` property of the received event is base64-decoded, and the two are compared for equality. From 80aa5a24dc55433bd95f37cd03dd20fab55c280b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 5 Jun 2019 20:33:09 -0600 Subject: [PATCH 115/128] Make v4 the "default" room version As per [MSC2002](https://github.com/matrix-org/matrix-doc/pull/2002). This was missed in https://github.com/matrix-org/matrix-doc/pull/2019 Fixes https://github.com/matrix-org/matrix-doc/issues/2071 --- specification/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/index.rst b/specification/index.rst index 33dff5a3..a6f6f62b 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -485,7 +485,7 @@ some other reason. Versions can switch between stable and unstable periodically for a variety of reasons, including discovered security vulnerabilities and age. Clients should not ask room administrators to upgrade their rooms if the room is -running a stable version. Servers SHOULD use room version 1 as the default room +running a stable version. Servers SHOULD use room version 4 as the default room version when creating new rooms. The available room versions are: From 500f3d3bf121f30171422566a46dbbeab3d88608 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 5 Jun 2019 22:28:57 -0600 Subject: [PATCH 116/128] Clarify that the default s2s transport is JSON over HTTP Fixes https://github.com/matrix-org/matrix-doc/issues/1713 --- .../identity_service/newsfragments/2086.clarification | 1 + specification/client_server_api.rst | 3 ++- specification/server_server_api.rst | 11 +++++++++++ 3 files changed, 14 insertions(+), 1 deletion(-) create mode 100644 changelogs/identity_service/newsfragments/2086.clarification diff --git a/changelogs/identity_service/newsfragments/2086.clarification b/changelogs/identity_service/newsfragments/2086.clarification new file mode 100644 index 00000000..7016308b --- /dev/null +++ b/changelogs/identity_service/newsfragments/2086.clarification @@ -0,0 +1 @@ +Clarify that the default transport is JSON over HTTP. diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index d9342a5b..d5b6491f 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -57,6 +57,8 @@ The following other versions are also available, in reverse chronological order: API Standards ------------- +.. TODO: Move a lot of this to a common area for all specs. + .. TODO Need to specify any HMAC or access_token lifetime/ratcheting tricks We need to specify capability negotiation for extensible transports @@ -82,7 +84,6 @@ names in JSON objects passed over the API also follow this convention. ``/createRoom``. A future version of this specification will address the inconsistency. - Any errors which occur at the Matrix API level MUST return a "standard error response". This is a JSON object which looks like: diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index b765e36a..28876e44 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -83,6 +83,17 @@ The following other versions are also available, in reverse chronological order: - `r0.1.1 `_ - `r0.1.0 `_ + +API standards +------------- + +The mandatory baseline for client-server communication in Matrix is exchanging +JSON objects over HTTP APIs. More efficient optional transports will in future +be supported as optional extensions - e.g. a packed binary encoding over +stream-cipher encrypted TCP socket for low-bandwidth/low-roundtrip mobile usage. +For the default HTTP transport, all API calls use a Content-Type of +``application/json``. In addition, all strings MUST be encoded as UTF-8. + Server discovery ---------------- From 79bbb47d9f200c3d99d3c6d8e2f7456cd66ac94f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 5 Jun 2019 23:18:04 -0600 Subject: [PATCH 117/128] Clarify when authorization and rate-limiting are not applicable Fixes https://github.com/matrix-org/matrix-doc/issues/1971 --- .../newsfragments/2090.clarification | 1 + .../matrix_templates/templates/http-api.tmpl | 18 +++++++++--------- 2 files changed, 10 insertions(+), 9 deletions(-) create mode 100644 changelogs/client_server/newsfragments/2090.clarification diff --git a/changelogs/client_server/newsfragments/2090.clarification b/changelogs/client_server/newsfragments/2090.clarification new file mode 100644 index 00000000..23ab50f7 --- /dev/null +++ b/changelogs/client_server/newsfragments/2090.clarification @@ -0,0 +1 @@ +Clarify when authorization and rate-limiting are not applicable. diff --git a/scripts/templating/matrix_templates/templates/http-api.tmpl b/scripts/templating/matrix_templates/templates/http-api.tmpl index 0b9207d9..74836045 100644 --- a/scripts/templating/matrix_templates/templates/http-api.tmpl +++ b/scripts/templating/matrix_templates/templates/http-api.tmpl @@ -10,13 +10,13 @@ {{endpoint.desc}} -{{":Rate-limited: Yes." if endpoint.rate_limited else "" }} -{{":Requires auth: Yes." if endpoint.requires_auth else "" }} +{{":Rate-limited: Yes." if endpoint.rate_limited else ":Rate-limited: No." }} +{{":Requires auth: Yes." if endpoint.requires_auth else ":Requires auth: No." }} .. class:: httpheaders - + Request format: - + {% if (endpoint.req_param_by_loc | length) %} {{ tables.split_paramtable(endpoint.req_param_by_loc) }} {% if (endpoint.req_body_tables) %} @@ -33,7 +33,7 @@ {% if endpoint.res_headers is not none -%} .. class:: httpheaders - + Response headers: {{ tables.paramtable(endpoint.res_headers.rows) }} @@ -42,7 +42,7 @@ {% if endpoint.res_tables|length > 0 -%} .. class:: httpheaders - + Response format: {% for table in endpoint.res_tables -%} @@ -54,7 +54,7 @@ {% endif -%} .. class:: httpheaders - + Example request: .. code:: http @@ -64,7 +64,7 @@ {% if endpoint.responses|length > 0 -%} .. class:: httpheaders - + Response{{"s" if endpoint.responses|length > 1 else "" }}: {% endif -%} @@ -78,7 +78,7 @@ {% if res["example"] -%} .. class:: httpheaders - + Example .. code:: json From e644227f4b607b3438cb37b21616a68e7f007645 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 6 Jun 2019 14:13:20 -0600 Subject: [PATCH 118/128] Clarify that the server shouldn't process retries for UIA --- specification/client_server_api.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 318ac08d..4593311d 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -557,9 +557,10 @@ message in the standard format. For example: } If the client has completed all stages of a flow, the homeserver performs the -API call and returns the result as normal. Completed stages cannot be re-tried; -The client must abandon the current session and start over. Homeservers should -treat retries as authentication errors. +API call and returns the result as normal. Completed stages cannot be retried +by clients, therefore servers must return either a 401 response with the completed +stages, or the result of the API call if all stages were completed when a client +retries a stage. Some authentication types may be completed by means other than through the Matrix client, for example, an email confirmation may be completed when the user From 9bf0103ef359202026ffce958c243b898a828f9e Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 6 Jun 2019 14:36:30 -0600 Subject: [PATCH 119/128] Clarify how many PDUs are in a given transaction object Fixes https://github.com/matrix-org/matrix-doc/issues/2093 --- api/server-server/backfill.yaml | 2 +- .../definitions/single_pdu_transaction.yaml | 32 ++++++++++++++++++ .../unlimited_pdu_transaction.yaml | 33 +++++++++++++++++++ api/server-server/events.yaml | 2 +- .../newsfragments/2095.clarification | 1 + 5 files changed, 68 insertions(+), 2 deletions(-) create mode 100644 api/server-server/definitions/single_pdu_transaction.yaml create mode 100644 api/server-server/definitions/unlimited_pdu_transaction.yaml create mode 100644 changelogs/server_server/newsfragments/2095.clarification diff --git a/api/server-server/backfill.yaml b/api/server-server/backfill.yaml index 0da0e234..2ed6298c 100644 --- a/api/server-server/backfill.yaml +++ b/api/server-server/backfill.yaml @@ -64,7 +64,7 @@ paths: A transaction containing the PDUs that preceded the given event(s), including the given event(s), up to the given limit. schema: - $ref: "definitions/transaction.yaml" + $ref: "definitions/unlimited_pdu_transaction.yaml" "/get_missing_events/{roomId}": post: summary: Retrieves events that the sender is missing diff --git a/api/server-server/definitions/single_pdu_transaction.yaml b/api/server-server/definitions/single_pdu_transaction.yaml new file mode 100644 index 00000000..ff682a44 --- /dev/null +++ b/api/server-server/definitions/single_pdu_transaction.yaml @@ -0,0 +1,32 @@ +# Copyright 2019 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +type: object +allOf: + - $ref: "transaction.yaml" +properties: + pdus: + type: array + description: |- + A single PDU. Note that events have a different format depending on the room + version - check the `room version specification`_ for precise event formats. + items: + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the transaction. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + properties: [] + example: + $ref: "../examples/minimal_pdu.json" +required: ['origin', 'origin_server_ts', 'pdus'] diff --git a/api/server-server/definitions/unlimited_pdu_transaction.yaml b/api/server-server/definitions/unlimited_pdu_transaction.yaml new file mode 100644 index 00000000..0fc31ee4 --- /dev/null +++ b/api/server-server/definitions/unlimited_pdu_transaction.yaml @@ -0,0 +1,33 @@ +# Copyright 2019 The Matrix.org Foundation C.I.C. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +type: object +allOf: + - $ref: "transaction.yaml" +properties: + pdus: + type: array + description: |- + List of persistent updates to rooms. Note that events have a different format + depending on the room version - check the `room version specification`_ for + precise event formats. + items: + type: object + title: PDU + description: |- + The `PDUs <#pdus>`_ contained in the transaction. The event format varies depending + on the room version - check the `room version specification`_ for precise event formats. + properties: [] + example: + $ref: "../examples/minimal_pdu.json" +required: ['origin', 'origin_server_ts', 'pdus'] diff --git a/api/server-server/events.yaml b/api/server-server/events.yaml index 1f1a802d..1f8ee537 100644 --- a/api/server-server/events.yaml +++ b/api/server-server/events.yaml @@ -156,4 +156,4 @@ paths: 200: description: A transaction containing a single PDU which is the event requested. schema: - $ref: "definitions/transaction.yaml" + $ref: "definitions/single_pdu_transaction.yaml" diff --git a/changelogs/server_server/newsfragments/2095.clarification b/changelogs/server_server/newsfragments/2095.clarification new file mode 100644 index 00000000..66257e17 --- /dev/null +++ b/changelogs/server_server/newsfragments/2095.clarification @@ -0,0 +1 @@ +Clarify how many PDUs are contained in transaction objects for various endpoints. From ae9abe798ef07de9c48f739b50b01e955725c63b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 6 Jun 2019 23:41:51 -0600 Subject: [PATCH 120/128] Revert signature change for redactable event test The previous signature was calculated on the unredacted event, which means the signature produced was wrong. --- specification/appendices/test_vectors.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/appendices/test_vectors.rst b/specification/appendices/test_vectors.rst index 7759fa88..790ed878 100644 --- a/specification/appendices/test_vectors.rst +++ b/specification/appendices/test_vectors.rst @@ -162,7 +162,7 @@ The event signing algorithm should emit the following signed event: "sender": "@u:domain", "signatures": { "domain": { - "ed25519:1": "4zc79tH2cU6Y+eg4YbbF7KiDOrnwEDjlhTqIKiH4k7L9zD9XCiomD7x9odL9eEwnyy1144QyMBe8O3HK++GHBg" + "ed25519:1": "Wm+VzmOUOz08Ds+0NTWb1d4CZrVsJSikkeRxh6aCcUwu6pNC78FunoD7KNWzqFn241eYHYMGCA5McEiVPdhzBA" } }, "unsigned": { From 7f01346bbad3f0cc31bf2d2fcfde4dd7daa0b2f6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 6 Jun 2019 23:50:26 -0600 Subject: [PATCH 121/128] Provide a more complete example of a "minimally-sized event" Using all the required fields of a v1 event. --- specification/appendices/test_vectors.rst | 25 +++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/specification/appendices/test_vectors.rst b/specification/appendices/test_vectors.rst index 790ed878..1de458cc 100644 --- a/specification/appendices/test_vectors.rst +++ b/specification/appendices/test_vectors.rst @@ -91,11 +91,22 @@ Given the following minimally-sized event: .. code:: json { + "room_id": "!x:domain", + "sender": "@a:domain", "event_id": "$0:domain", "origin": "domain", "origin_server_ts": 1000000, "signatures": {}, + "hashes": {}, "type": "X", + "content": {}, + "prev_events": [ + ["$1:domain", "ExampleHash"] + ], + "auth_events": [ + ["$2", "ExampleHash2"] + ], + "depth": 3, "unsigned": { "age_ts": 1000000 } @@ -106,15 +117,25 @@ The event signing algorithm should emit the following signed event: .. code:: json { + "auth_events": [ + ["$2", "6tJjLpXtggfke8UxFhAKg82QVkJzvKOVOOSjUDK4ZSI"] + ], + "content": {}, + "depth": 3, "event_id": "$0:domain", "hashes": { - "sha256": "6tJjLpXtggfke8UxFhAKg82QVkJzvKOVOOSjUDK4ZSI" + "sha256": "6AaJICN1NJURTtaomDYfJlCPMIU+0gtkwg7qzd8FiJM" }, "origin": "domain", "origin_server_ts": 1000000, + "prev_events": [ + ["$1:domain", "onLKD1bGljeBWQhWZ1kaP9SorVmRQNdN5aM2JYU2n/g"] + ], + "room_id": "!x:domain", + "sender": "@a:domain", "signatures": { "domain": { - "ed25519:1": "JV2dlZUASAefSdywnyCxzykHlyr7xkKGK7IRir1cF8eYsnONrCSb+GRn7aXXstr1UHKvzYjRXPx0001+boD1Ag" + "ed25519:1": "51U0wpKYsaNLTQRbha2v5EGO2cVA6pCtnAKEXguu3j3efCLlmq/53vEfWhsk3tY6gnLsV0YM4Lx2NGZkzmV2Ag" } }, "type": "X", From d7858354f26aa3bb8a6cc58e0b1db1d280eee21e Mon Sep 17 00:00:00 2001 From: Jamie McClymont Date: Fri, 7 Jun 2019 20:54:47 +1200 Subject: [PATCH 122/128] Fix 404s in links from room v1 spec --- specification/rooms/v1.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 1c7a56c4..e8cbf663 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -293,5 +293,5 @@ Events in version 1 rooms have the following structure: {{definition_ss_pdu}} -.. _`auth events selection`: ../../server_server/r0.1.1.html#auth-events-selection -.. _`Signing Events`: ../../server_server/r0.1.1.html#signing-events +.. _`auth events selection`: ../server_server/r0.1.1.html#auth-events-selection +.. _`Signing Events`: ../server_server/r0.1.1.html#signing-events From 00fee7463623b38516687d1d4ea5463a5e5e3ec8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 7 Jun 2019 07:40:52 -0600 Subject: [PATCH 123/128] Update example --- specification/appendices/test_vectors.rst | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-) diff --git a/specification/appendices/test_vectors.rst b/specification/appendices/test_vectors.rst index 1de458cc..05b115db 100644 --- a/specification/appendices/test_vectors.rst +++ b/specification/appendices/test_vectors.rst @@ -93,19 +93,14 @@ Given the following minimally-sized event: { "room_id": "!x:domain", "sender": "@a:domain", - "event_id": "$0:domain", "origin": "domain", "origin_server_ts": 1000000, "signatures": {}, "hashes": {}, "type": "X", "content": {}, - "prev_events": [ - ["$1:domain", "ExampleHash"] - ], - "auth_events": [ - ["$2", "ExampleHash2"] - ], + "prev_events": [], + "auth_events": [], "depth": 3, "unsigned": { "age_ts": 1000000 @@ -117,25 +112,20 @@ The event signing algorithm should emit the following signed event: .. code:: json { - "auth_events": [ - ["$2", "6tJjLpXtggfke8UxFhAKg82QVkJzvKOVOOSjUDK4ZSI"] - ], + "auth_events": [], "content": {}, "depth": 3, - "event_id": "$0:domain", "hashes": { - "sha256": "6AaJICN1NJURTtaomDYfJlCPMIU+0gtkwg7qzd8FiJM" + "sha256": "5jM4wQpv6lnBo7CLIghJuHdW+s2CMBJPUOGOC89ncos" }, "origin": "domain", "origin_server_ts": 1000000, - "prev_events": [ - ["$1:domain", "onLKD1bGljeBWQhWZ1kaP9SorVmRQNdN5aM2JYU2n/g"] - ], + "prev_events": [], "room_id": "!x:domain", "sender": "@a:domain", "signatures": { "domain": { - "ed25519:1": "51U0wpKYsaNLTQRbha2v5EGO2cVA6pCtnAKEXguu3j3efCLlmq/53vEfWhsk3tY6gnLsV0YM4Lx2NGZkzmV2Ag" + "ed25519:1": "KxwGjPSDEtvnFgU00fwFz+l6d2pJM6XBIaMEn81SXPTRl16AqLAYqfIReFGZlHi5KLjAWbOoMszkwsQma+lYAg" } }, "type": "X", From d49c7fb3b02db22a4275bfb0f147b0b4a0477b6d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 7 Jun 2019 08:01:17 -0600 Subject: [PATCH 124/128] Apply suggestions from code review Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- event-schemas/schema/m.key.verification.accept | 2 +- specification/modules/end_to_end_encryption.rst | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/event-schemas/schema/m.key.verification.accept b/event-schemas/schema/m.key.verification.accept index e52df39e..41c59968 100644 --- a/event-schemas/schema/m.key.verification.accept +++ b/event-schemas/schema/m.key.verification.accept @@ -17,7 +17,7 @@ properties: type: string enum: ["m.sas.v1"] description: |- - The verification method to use. Must be ``m.sas.v1``. + The verification method to use. key_agreement_protocol: type: string description: |- diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 4bd12b71..fb3d2ba5 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -536,15 +536,15 @@ The process between Alice and Bob verifying each other would be: .. |AlicePublicKey| replace:: :math:`K_{A}^{public}` .. |AlicePrivateKey| replace:: :math:`K_{A}^{private}` -.. |AliceCurve25519| replace:: :math:`K_{A}^{private}K_{A}^{public}` +.. |AliceCurve25519| replace:: :math:`K_{A}^{private},K_{A}^{public}` .. |BobPublicKey| replace:: :math:`K_{B}^{public}` .. |BobPrivateKey| replace:: :math:`K_{B}^{private}` -.. |BobCurve25519| replace:: :math:`K_{B}^{private}K_{B}^{public}` +.. |BobCurve25519| replace:: :math:`K_{B}^{private},K_{B}^{public}` .. |AliceBobCurve25519| replace:: :math:`K_{A}^{private}K_{B}^{public}` .. |BobAliceCurve25519| replace:: :math:`K_{B}^{private}K_{A}^{public}` .. |AliceBobECDH| replace:: :math:`ECDH(K_{A}^{private},K_{B}^{public})` -1. Alice and Bob establish a secure connection, likely meeting in-person. "Secure" +1. Alice and Bob establish a secure out-of-band connection, such as meeting in-person or a video call. "Secure" here means that either party cannot be impersonated, not explicit secrecy. #. Alice and Bob communicate which devices they'd like to verify with each other. #. Alice selects Bob's device from the device list and begins verification. @@ -573,9 +573,9 @@ The process between Alice and Bob verifying each other would be: methods are available, clients should allow the users to select a method. #. Alice and Bob compare the strings shown by their devices, and tell their devices if they match or not. -#. Assuming they match, Alice and Bob's devices calculate the HMAC of their own device +#. Assuming they match, Alice and Bob's devices calculate the HMAC of their own device keys and a comma-separated sorted list of of the key IDs that they wish the other user - to verify. HMAC is defined in RFC 2104, and SHA-256 as the hash function. The key for + to verify, using SHA-256 as the hash function. HMAC is defined in [RFC 2104](https://tools.ietf.org/html/rfc2104). The key for the HMAC is different for each item and is calculated by generating 32 bytes (256 bits) using `the key verification HKDF <#SAS-HKDF>`_. #. Alice's device sends Bob's device a ``m.key.verification.mac`` message containing the @@ -619,7 +619,7 @@ At any point the interactive verfication can go wrong. The following describes w to do when an error happens: * Alice or Bob can cancel the verification at any time. A ``m.key.verification.cancel`` - message must be sent to signify the cancelation. + message must be sent to signify the cancellation. * The verification can time out. Clients should time out a verification that does not complete within 5 minutes. Additionally, clients should expire a ``transaction_id`` which goes unused for 5 minutes after having last sent/received it. The client should @@ -668,8 +668,8 @@ are used in addition to those already specified: HKDF calculation <<<<<<<<<<<<<<<< -In all of the SAS methods, HKDF is as defined in RFC 5869 and uses the previously -agreed upon hash function for the hash function. The shared secret is supplied +In all of the SAS methods, HKDF is as defined in [RFC 5869](https://tools.ietf.org/html/rfc5869) and uses the previously +agreed-upon hash function for the hash function. The shared secret is supplied as the input keying material. No salt is used, and the input parameter is the concatenation of: From 3877896a4cb232ce760f1556d7a1fc6de001b95f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 7 Jun 2019 08:10:52 -0600 Subject: [PATCH 125/128] Clarify how we now expect verification to be done --- .../modules/end_to_end_encryption.rst | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index fb3d2ba5..fb320dc4 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -384,20 +384,10 @@ man-in-the-middle. This verification process requires an out-of-band channel: there is no way to do it within Matrix without trusting the administrators of the homeservers. -In Matrix, the basic process for device verification is for Alice to verify -that the public Ed25519 signing key she received via ``/keys/query`` for Bob's -device corresponds to the private key in use by Bob's device. For now, it is -recommended that clients provide mechanisms by which the user can see: - -1. The public part of their device's Ed25519 signing key, encoded using - `unpadded Base64`_. - -2. The list of devices in use for each user in a room, along with the public - Ed25519 signing key for each device, again encoded using unpadded Base64. - -Alice can then meet Bob in person, or contact him via some other trusted -medium, and use `SAS Verification`_ or ask him to read out the Ed25519 key -shown on his device, comparing it to the one shown on Alice's device. +In Matrix, verification works by Alice meeting Bob in person, or contact him +via some other trusted medium, and use `SAS Verification`_ to interactively +verify Bob's devices. Alice and Bob may also read aloud their unpadded base64 +encoded Ed25519 public key, as returned by ``/keys/query``. Device verification may reach one of several conclusions. For example: From 5ec1a50b94cca9a92951b7f3139c4ff76ed6c2b2 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 7 Jun 2019 08:12:09 -0600 Subject: [PATCH 126/128] Linefeeds and other clarifications --- .../modules/end_to_end_encryption.rst | 39 ++++++++++--------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index fb320dc4..0de1328b 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -530,12 +530,12 @@ The process between Alice and Bob verifying each other would be: .. |BobPublicKey| replace:: :math:`K_{B}^{public}` .. |BobPrivateKey| replace:: :math:`K_{B}^{private}` .. |BobCurve25519| replace:: :math:`K_{B}^{private},K_{B}^{public}` -.. |AliceBobCurve25519| replace:: :math:`K_{A}^{private}K_{B}^{public}` .. |BobAliceCurve25519| replace:: :math:`K_{B}^{private}K_{A}^{public}` .. |AliceBobECDH| replace:: :math:`ECDH(K_{A}^{private},K_{B}^{public})` -1. Alice and Bob establish a secure out-of-band connection, such as meeting in-person or a video call. "Secure" - here means that either party cannot be impersonated, not explicit secrecy. +1. Alice and Bob establish a secure out-of-band connection, such as meeting + in-person or a video call. "Secure" here means that either party cannot be + impersonated, not explicit secrecy. #. Alice and Bob communicate which devices they'd like to verify with each other. #. Alice selects Bob's device from the device list and begins verification. #. Alice's client ensures it has a copy of Bob's device key. @@ -565,9 +565,9 @@ The process between Alice and Bob verifying each other would be: they match or not. #. Assuming they match, Alice and Bob's devices calculate the HMAC of their own device keys and a comma-separated sorted list of of the key IDs that they wish the other user - to verify, using SHA-256 as the hash function. HMAC is defined in [RFC 2104](https://tools.ietf.org/html/rfc2104). The key for - the HMAC is different for each item and is calculated by generating 32 bytes (256 bits) - using `the key verification HKDF <#SAS-HKDF>`_. + to verify, using SHA-256 as the hash function. HMAC is defined in [RFC 2104](https://tools.ietf.org/html/rfc2104). + The key for the HMAC is different for each item and is calculated by generating + 32 bytes (256 bits) using `the key verification HKDF <#SAS-HKDF>`_. #. Alice's device sends Bob's device a ``m.key.verification.mac`` message containing the MAC of Alice's device keys and the MAC of her key IDs to be verified. Bob's device does the same for Bob's device keys and key IDs concurrently with Alice. @@ -611,12 +611,12 @@ to do when an error happens: * Alice or Bob can cancel the verification at any time. A ``m.key.verification.cancel`` message must be sent to signify the cancellation. * The verification can time out. Clients should time out a verification that does not - complete within 5 minutes. Additionally, clients should expire a ``transaction_id`` - which goes unused for 5 minutes after having last sent/received it. The client should - inform the user that the verification timed out, and send an appropriate ``m.key.verification.cancel`` - message to the other device. -* When the same device attempts to intiate multiple verification attempts, cancel all - attempts with that device. + complete within 10 minutes. Additionally, clients should expire a ``transaction_id`` + which goes unused for 10 minutes after having last sent/received it. The client should + inform the user that the verification timed out, and send an appropriate + ``m.key.verification.cancel`` message to the other device. +* When the same device attempts to intiate multiple verification attempts, the receipient + should cancel all attempts with that device. * When a device receives an unknown ``transaction_id``, it should send an appropriate ``m.key.verfication.cancel`` message to the other device indicating as such. This does not apply for inbound ``m.key.verification.start`` or ``m.key.verification.cancel`` @@ -658,10 +658,10 @@ are used in addition to those already specified: HKDF calculation <<<<<<<<<<<<<<<< -In all of the SAS methods, HKDF is as defined in [RFC 5869](https://tools.ietf.org/html/rfc5869) and uses the previously -agreed-upon hash function for the hash function. The shared secret is supplied -as the input keying material. No salt is used, and the input parameter is the -concatenation of: +In all of the SAS methods, HKDF is as defined in [RFC 5869](https://tools.ietf.org/html/rfc5869) +and uses the previously agreed-upon hash function for the hash function. The shared +secret is supplied as the input keying material. No salt is used, and the input +parameter is the concatenation of: * The string ``MATRIX_KEY_VERIFICATION_SAS``. * The Matrix ID of the user who sent the ``m.key.verification.start`` message. @@ -670,8 +670,9 @@ concatenation of: * The Device ID of the device which sent the ``m.key.verification.accept`` message. * The ``transaction_id`` being used. -HKDF is used over the plain shared secret as it results in a harder attack -as well as more uniform data to work with. +.. admonition:: Rationale + HKDF is used over the plain shared secret as it results in a harder attack + as well as more uniform data to work with. For verification of each party's device keys, HKDF is as defined in RFC 5869 and uses SHA-256 as the hash function. The shared secret is supplied as the input keying @@ -697,7 +698,7 @@ The bitwise operations to get the numbers given the 5 bytes :math:`B_{0}, B_{1}, B_{2}, B_{3}, B_{4}` would be: * First: :math:`(B_{0} \ll 5 | B_{1} \gg 3) + 1000` -* Second: :math:`(B_{1} \& 0x7 | B_{2} \ll 2 | B_{3} \gg 6) + 1000` +* Second: :math:`((B_{1} \& 0x7) \ll 10 | B_{2} \ll 2 | B_{3} \gg 6) + 1000` * Third: :math:`((B_{3} \& 0x3F) \ll 7 | B_{4} \gg 1) + 1000` The digits are displayed to the user either with an appropriate separator, From 360ac0b90088a0b9fcb6876fb9bac847fb3f0038 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 7 Jun 2019 08:54:21 -0600 Subject: [PATCH 127/128] Update specification/server_server_api.rst Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- specification/server_server_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 812f0ffc..7acc71e4 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1100,7 +1100,7 @@ a redacted copy. The signatures expected on an event are: -* The sender's server, unless the invite was created as a result of 3rd party invite. +* The ``sender``'s server, unless the invite was created as a result of 3rd party invite. The sender must already match the 3rd party invite, and the server which actually sends the event may be a different server. * For room versions 1 and 2, the server which created the ``event_id``. Other room From 46747e897ecc0b7c4c06e6488a02b5c208c87d3d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 7 Jun 2019 10:32:06 -0600 Subject: [PATCH 128/128] ing --- specification/modules/end_to_end_encryption.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/modules/end_to_end_encryption.rst b/specification/modules/end_to_end_encryption.rst index 0de1328b..62881967 100644 --- a/specification/modules/end_to_end_encryption.rst +++ b/specification/modules/end_to_end_encryption.rst @@ -384,7 +384,7 @@ man-in-the-middle. This verification process requires an out-of-band channel: there is no way to do it within Matrix without trusting the administrators of the homeservers. -In Matrix, verification works by Alice meeting Bob in person, or contact him +In Matrix, verification works by Alice meeting Bob in person, or contacting him via some other trusted medium, and use `SAS Verification`_ to interactively verify Bob's devices. Alice and Bob may also read aloud their unpadded base64 encoded Ed25519 public key, as returned by ``/keys/query``.