From 458383585f6e58b588171d37584daccba49ff5bb Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 2 Oct 2015 15:03:55 +0100 Subject: [PATCH 01/16] Stub sections --- specification/modules/instant_messaging.rst | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 43a06aa1..619226df 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -3,6 +3,9 @@ Instant Messaging .. _module:im: +This module adds support for sending human-readable messages to a room. It also +adds human-readable information to the room itself such as a room name and topic. + Events ------ @@ -15,7 +18,7 @@ Events {{m_room_topic_event}} m.room.message msgtypes ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ .. TODO-spec How a client should handle unknown message types. @@ -27,3 +30,13 @@ outlined below. {{msgtype_events}} + +Client behaviour +---------------- + +Server behaviour +---------------- + +Security considerations +----------------------- + From 8e5c832ff9fa8d8e674346e3776c08e659c306c9 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Mon, 5 Oct 2015 13:45:23 +0100 Subject: [PATCH 02/16] Flesh out more of the IM module --- .../schema/v1/m.room.message.feedback | 2 +- specification/modules/instant_messaging.rst | 59 +++++++++++++++++-- 2 files changed, 54 insertions(+), 7 deletions(-) diff --git a/event-schemas/schema/v1/m.room.message.feedback b/event-schemas/schema/v1/m.room.message.feedback index 1bbfc1ba..2eaed999 100644 --- a/event-schemas/schema/v1/m.room.message.feedback +++ b/event-schemas/schema/v1/m.room.message.feedback @@ -1,7 +1,7 @@ { "type": "object", "title": "MessageFeedback", - "description": "Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged. N.B. not implemented in Synapse, and superceded in v2 CS API by the ``relates_to`` event field.", + "description": "**NB: Usage of this event is discouraged in favour of the** `receipts module`_. **Most clients will not recognise this event.** Feedback events are events sent to acknowledge a message in some way. There are two supported acknowledgements: ``delivered`` (sent when the event has been received) and ``read`` (sent when the event has been observed by the end-user). The ``target_event_id`` should reference the ``m.room.message`` event being acknowledged.", "allOf": [{ "$ref": "core-event-schema/room_event.json" }], diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 619226df..7934b80a 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -4,15 +4,37 @@ Instant Messaging .. _module:im: This module adds support for sending human-readable messages to a room. It also -adds human-readable information to the room itself such as a room name and topic. +adds support for associating human-readable information with the room itself +such as a room name and topic. Events ------ {{m_room_message_event}} + +.. admonition:: Rationale + + Not all clients can display all message types. The most commonly supported + message type is raw text. As a result, we chose to have a textual fallback + display method represented by the ``body`` key. This means that even if the + client cannot display a particular ``msgtype``, they can still display + *something*, even if it is just plain text. + {{m_room_message_feedback_event}} +Usage of this event is discouraged for several reasons: + - The number of feedback events will grow very quickly with the number of users + in the room. This event provides no way to "batch" feedback, unlike the + `receipts module`_. + - Pairing feedback to messages gets complicated when paginating as feedback + arrives before the message it is acknowledging. + - There are no guarantees that the client has seen the event ID being + acknowledged. + + +.. _`receipts module`: `module:receipts`_ + {{m_room_name_event}} {{m_room_topic_event}} @@ -20,13 +42,10 @@ Events m.room.message msgtypes ~~~~~~~~~~~~~~~~~~~~~~~ -.. TODO-spec - How a client should handle unknown message types. - - Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type of message being sent. Each type has their own required and optional keys, as -outlined below. +outlined below. If a client cannot display the given ``msgtype`` then it MUST +display the fallback plain text ``body`` key instead. {{msgtype_events}} @@ -34,9 +53,37 @@ outlined below. Client behaviour ---------------- +Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be +uploaded using the `content repository module`_ where available. The +resulting ``mxc://`` URI can then be used in the ``url`` key. The +attachment SHOULD be uploaded *prior* to sending the event in order to stop a +race condition where the recipient receives a link to a non-existent attachment. + +.. _`content repository module`: `module:content`_ + +Recommendations when sending messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- Advise using idempotent PUTs to send messages (and why) +- Retries (exp backoff, giveup eventually allowing manual retry) +- Queueing (bucket per room) + +Implementing local echo +~~~~~~~~~~~~~~~~~~~~~~~ +- Local echo (document bug with races) - sending state. Pairing returned event ID. + +Displaying membership information with messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Member name linking (incl. pagination aka historical display names) + Server behaviour ---------------- +- SHOULD enforce the body/msgtype keys are present (can 400 them) + Security considerations ----------------------- +- Not encrypted, link to E2E module. +- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc) + From 91ca36509b3112a83680a97a0d672a428b4d7fef Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 7 Oct 2015 11:51:49 +0100 Subject: [PATCH 03/16] Flesh out IM module --- specification/0-intro.rst | 2 + specification/modules/instant_messaging.rst | 89 ++++++++++++++++++--- 2 files changed, 80 insertions(+), 11 deletions(-) diff --git a/specification/0-intro.rst b/specification/0-intro.rst index 445b32ef..7c27e0d7 100644 --- a/specification/0-intro.rst +++ b/specification/0-intro.rst @@ -434,6 +434,8 @@ Some requests have unique error codes: :``M_LOGIN_EMAIL_URL_NOT_YET``: Encountered when polling for an email link which has not been clicked yet. +.. _sect:txn_ids: + The C-S API typically uses ``HTTP POST`` to submit requests. This means these requests are not idempotent. The C-S API also allows ``HTTP PUT`` to make requests idempotent. In order to use a ``PUT``, paths should be suffixed with diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 7934b80a..426189b5 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -53,7 +53,14 @@ display the fallback plain text ``body`` key instead. Client behaviour ---------------- -Events which have attachments (e.g. ``m.image``, ``m.file``) are advised to be +Clients SHOULD verify the structure of incoming events to ensure that the +expected keys exist and that they are of the right type. Clients can discard +malformed events or display a placeholder message to the user. Redacted +``m.room.message`` events MUST be removed from the client. This can either be +replaced with placeholder text (e.g. "[REDACTED]") or the redacted message can +be removed entirely from the messages view. + +Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be uploaded using the `content repository module`_ where available. The resulting ``mxc://`` URI can then be used in the ``url`` key. The attachment SHOULD be uploaded *prior* to sending the event in order to stop a @@ -63,27 +70,87 @@ race condition where the recipient receives a link to a non-existent attachment. Recommendations when sending messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Advise using idempotent PUTs to send messages (and why) -- Retries (exp backoff, giveup eventually allowing manual retry) -- Queueing (bucket per room) -Implementing local echo -~~~~~~~~~~~~~~~~~~~~~~~ -- Local echo (document bug with races) - sending state. Pairing returned event ID. +Clients can send messages using ``POST`` or ``PUT`` requests. Clients SHOULD use +``PUT`` requests with `transaction IDs`_ to make requests idempotent. This +ensures that messages are sent exactly once even under poor network conditions. +Clients SHOULD retry requests using an exponential-backoff algorithm for a +certain amount of time T. It is recommended that T is no longer than 5 minutes. +After this time, the client should stop retrying and mark the message as "unsent". +Users should be able to manually resend unsent messages. + +Users may type several messages at once and send them all in quick succession. +Clients SHOULD preserve the order in which they were sent by the user. This +means that clients should wait for the response to the previous request before +sending the next request. This can lead to head-of-line blocking. In order to +reduce the impact of head-of-line blocking, clients should use a queue per room +rather than a global queue, as ordering is only relevant within a single room +rather than between rooms. + +.. _`transaction IDs`: `sect:txn_ids`_ + +Local echo +~~~~~~~~~~ + +Messages SHOULD appear immediately in the message view when a user presses the +"send" button. This should occur even if the message is still sending. This is +referred to as "local echo". Clients SHOULD implement "local echo" of messages. + +Clients need to be able to pair up the "remote echo" from the server with the +"local echo" to prevent duplicate messages being displayed. Ideally this pairing +would occur transparently to the user: the UI would not flicker as it transitions +from local to remote. Flickering cannot be fully avoided in version 1 of the +client-server API. Two scenarios need to be considered: + +- The client sends a message and the remote echo arrives on the event stream + *after* the request to send the message completes. +- The client sends a message and the remote echo arrives on the event stream + *before* the request to send the message completes. + +In the first scenario, the client will receive an event ID when the request to +send the message completes. This ID can be used to identify the duplicate event +when it arrives on the event stream. However, in the second scenario, the event +arrives before the client has obtained an event ID. This makes it impossible to +identify it as a duplicate event. This results in the client displaying the +message twice for a fraction of a second before the the original request to send +the message completes. Once it completes, the client can take remedial actions +to remove the duplicate event by looking for duplicate event IDs. Version 2 of +the client-server API resolves this by attaching the transaction ID of the +sending request to the event itself. + Displaying membership information with messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Member name linking (incl. pagination aka historical display names) +Clients may wish to show the display name and avatar URL of the room member who +sent a message. This can be achieved by inspecting the ``m.room.member`` event +for that user ID. + +When a user paginates the message history, clients may wish to show the +**historical** display name and avatar URL for a room member. This is possible +because older ``m.room.member`` events are returned when paginating. This can +be implemented efficiently by keeping two sets of room state: old and current. +As new events arrive and/or the user paginates back in time, these two sets of +state diverge from each other. New events update the current state and paginated +events update the old state. When paginated events are processed sequentially, +the old state represents the state of the room *at the time the event was sent*. +This can then be used to set the historical display name and avatar URL. Server behaviour ---------------- -- SHOULD enforce the body/msgtype keys are present (can 400 them) +Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body`` +and ``msgtype`` keys by 400ing the request to send a message. Security considerations ----------------------- -- Not encrypted, link to E2E module. -- XSS: Should sanitise ALL KEYS before injecting as unsafe HTML (name/topic/body/etc) +Messages sent using this module are not encrypted. Encryption can be layered +over the top of this module: where the plaintext format is an ``m.room.message`` +conforming to this module. This can be achieved using the `E2E module`_. + +Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site +Scripting (XSS) attacks. This includes room names and topics. + +.. _`E2E module`: `module:e2e`_ From e378de8379abcb85db8048b7dbd80b3851b4ac51 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Fri, 9 Oct 2015 11:04:11 +0100 Subject: [PATCH 04/16] Shuffle history visibility module to be in the module format --- specification/0-events.rst | 2 - specification/modules/history_visibility.rst | 44 +++++++++++++++----- 2 files changed, 33 insertions(+), 13 deletions(-) diff --git a/specification/0-events.rst b/specification/0-events.rst index 16948462..e2373f96 100644 --- a/specification/0-events.rst +++ b/specification/0-events.rst @@ -26,8 +26,6 @@ prefixed with ``m.`` {{m_room_create_event}} -{{m_room_history_visibility_event}} - {{m_room_join_rules_event}} {{m_room_member_event}} diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 371282bd..4f0c8976 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -1,21 +1,36 @@ Room History Visibility ------------------------ +======================= .. _module:history-visibility: -Whether a member of a room can see the events that happened in a room from -before they joined the room is controlled by the ``history_visibility`` key -of the ``m.room.history_visibility`` state event. The valid values for -``history_visibility`` are: +This module adds support for controlling the visibility of previous events in a +room. Whether a member of a room can see the events that happened in a room from +before they joined the room is controlled by the ``m.room.history_visibility`` +event outlined below. Visibility can take the form of one of three options: -- ``shared`` -- ``invited`` -- ``joined`` +- ``shared`` - Previous events are always shown to newly joined members. +- ``invited`` - Previous events are shown to newly joined members from the point + they were invited onwards. +- ``joined`` - Previous events are shown to newly joined members from the point + they joined the room onwards. -By default if no ``history_visibility`` is set it is assumed to be ``shared``. +Events +------ -The rules governing whether a user is allowed to see an event depend solely on -the state of the room at that event: +{{m_room_history_visibility_event}} + +Client behaviour +---------------- + +Clients that implement this module MUST present to the user the possible options +for setting history visibility when creating a room. + +Server behaviour +---------------- + +By default if no ``history_visibility`` is set the visibility is assumed to be +``shared``. The rules governing whether a user is allowed to see an event depend +solely on the state of the room *at that event*: 1. If the user was joined, allow. 2. If the user was invited and the ``history_visibility`` was set to @@ -24,3 +39,10 @@ the state of the room at that event: was set to ``shared``, allow. 4. Otherwise, deny. +Security considerations +----------------------- + +The default value for ``history_visibility`` is ``shared`` for +backwards-compatibility reasons. Clients need to be aware that by not setting +this event they are exposing all of their room history to anyone in the room. + From 173d00cea04e3c607992396b7395741971578e35 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 13 Oct 2015 11:08:19 +0100 Subject: [PATCH 05/16] Review comments --- specification/modules/history_visibility.rst | 22 ++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 4f0c8976..12ea7b22 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -6,13 +6,27 @@ Room History Visibility This module adds support for controlling the visibility of previous events in a room. Whether a member of a room can see the events that happened in a room from before they joined the room is controlled by the ``m.room.history_visibility`` -event outlined below. Visibility can take the form of one of three options: +event outlined below. In all cases, the member still needs to be joined to the +room to receive events for that room. The visibility option simply determines +which subset of events in the room are presented to the client. Visibility can +take the form of one of three options: -- ``shared`` - Previous events are always shown to newly joined members. +- ``shared`` - Previous events are always shown to newly joined members. All + events in the room are shown, even those sent when the member was not a part + of the room. - ``invited`` - Previous events are shown to newly joined members from the point - they were invited onwards. + they were invited onwards. Events stop being shown when the member's state + changes to something other than ``invite`` or ``join``. - ``joined`` - Previous events are shown to newly joined members from the point - they joined the room onwards. + they joined the room onwards. Events stop being shown when the members state + changes to something other than ``join``. + +.. WARNING:: + These options are applied at the point an event is *sent*. Checks are + performed with the state of the ``m.room.history_visibility`` event when the + event in question is added to the DAG. This means clients cannot + retrospectively choose to show or hide history to new users if the setting at + that time was more restrictive. Events ------ From 4170dbd5cf065f7e1c7fde32f4c5a6e622c98055 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Tue, 13 Oct 2015 11:29:54 +0100 Subject: [PATCH 06/16] Review comments --- event-schemas/schema/v1/m.room.message | 2 +- specification/modules/instant_messaging.rst | 44 +++++++++------------ 2 files changed, 19 insertions(+), 27 deletions(-) diff --git a/event-schemas/schema/v1/m.room.message b/event-schemas/schema/v1/m.room.message index 27b7e925..91c04b7f 100644 --- a/event-schemas/schema/v1/m.room.message +++ b/event-schemas/schema/v1/m.room.message @@ -1,7 +1,7 @@ { "type": "object", "title": "Message", - "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message.", + "description": "This event is used when sending messages in a room. Messages are not limited to be text. The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. The ``body`` key is text and MUST be used with every kind of ``msgtype`` as a fallback mechanism for when a client cannot render a message. This allows clients to display *something* even if it is just plain text.", "allOf": [{ "$ref": "core-event-schema/room_event.json" }], diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 426189b5..487240de 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -12,15 +12,6 @@ Events {{m_room_message_event}} - -.. admonition:: Rationale - - Not all clients can display all message types. The most commonly supported - message type is raw text. As a result, we chose to have a textual fallback - display method represented by the ``body`` key. This means that even if the - client cannot display a particular ``msgtype``, they can still display - *something*, even if it is just plain text. - {{m_room_message_feedback_event}} Usage of this event is discouraged for several reasons: @@ -44,7 +35,7 @@ m.room.message msgtypes Each `m.room.message`_ MUST have a ``msgtype`` key which identifies the type of message being sent. Each type has their own required and optional keys, as -outlined below. If a client cannot display the given ``msgtype`` then it MUST +outlined below. If a client cannot display the given ``msgtype`` then it SHOULD display the fallback plain text ``body`` key instead. {{msgtype_events}} @@ -62,9 +53,7 @@ be removed entirely from the messages view. Events which have attachments (e.g. ``m.image``, ``m.file``) SHOULD be uploaded using the `content repository module`_ where available. The -resulting ``mxc://`` URI can then be used in the ``url`` key. The -attachment SHOULD be uploaded *prior* to sending the event in order to stop a -race condition where the recipient receives a link to a non-existent attachment. +resulting ``mxc://`` URI can then be used in the ``url`` key. .. _`content repository module`: `module:content`_ @@ -95,12 +84,15 @@ Local echo Messages SHOULD appear immediately in the message view when a user presses the "send" button. This should occur even if the message is still sending. This is referred to as "local echo". Clients SHOULD implement "local echo" of messages. - -Clients need to be able to pair up the "remote echo" from the server with the -"local echo" to prevent duplicate messages being displayed. Ideally this pairing -would occur transparently to the user: the UI would not flicker as it transitions -from local to remote. Flickering cannot be fully avoided in version 1 of the -client-server API. Two scenarios need to be considered: + +Clients need to be able to match the message they are sending with the same +message which they receive from the event stream. The echo of the same message +from the event stream is referred to as "remote echo". Both echoes need to be +identified as the same message in order to prevent duplicate messages being +displayed. Ideally this pairing would occur transparently to the user: the UI +would not flicker as it transitions from local to remote. Flickering cannot be +fully avoided in version 1 of the client-server API. Two scenarios need to be +considered: - The client sends a message and the remote echo arrives on the event stream *after* the request to send the message completes. @@ -123,8 +115,8 @@ Displaying membership information with messages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Clients may wish to show the display name and avatar URL of the room member who -sent a message. This can be achieved by inspecting the ``m.room.member`` event -for that user ID. +sent a message. This can be achieved by inspecting the ``m.room.member`` state +event for that user ID. When a user paginates the message history, clients may wish to show the **historical** display name and avatar URL for a room member. This is possible @@ -139,15 +131,15 @@ This can then be used to set the historical display name and avatar URL. Server behaviour ---------------- -Homeservers SHOULD enforce that ``m.room.message`` events have textual ``body`` -and ``msgtype`` keys by 400ing the request to send a message. +Homeservers SHOULD reject ``m.room.message`` events which don't have a +``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status +code of 400. Security considerations ----------------------- -Messages sent using this module are not encrypted. Encryption can be layered -over the top of this module: where the plaintext format is an ``m.room.message`` -conforming to this module. This can be achieved using the `E2E module`_. +Messages sent using this module are not encrypted. Messages can be encrypted +using the `E2E module`_. Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site Scripting (XSS) attacks. This includes room names and topics. From 4a9db39277a6f61bb4efc27bb6a14815a89acd05 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 14 Oct 2015 13:38:28 +0100 Subject: [PATCH 07/16] Clean up some untruths in the login api doc add "type", and "username" -> "user" --- api/client-server/v1/login.yaml | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/api/client-server/v1/login.yaml b/api/client-server/v1/login.yaml index 3d415c29..bbab46df 100644 --- a/api/client-server/v1/login.yaml +++ b/api/client-server/v1/login.yaml @@ -33,17 +33,21 @@ paths: type: object example: |- { - "username": "cheeky_monkey", + "type": "m.login.pasword", + "user": "cheeky_monkey", "password": "ilovebananas" } properties: - username: + type: + type: string + description: The login type being used. Currently only "m.login.password" is supported. + user: type: string description: The fully qualified user ID or just local part of the user ID, to log in. password: type: string description: The user's password. - required: ["username", "password"] + required: ["type", "user", "password"] responses: 200: description: The user has been authenticated. @@ -78,6 +82,15 @@ paths: home_server: type: string description: The hostname of the Home Server on which the account has been registered. + 400: + description: |- + Part of the request was invalid. For example, the login type may not be recognised. + examples: + application/json: |- + { + "errcode": "M_UNKNOWN", + "error": "Bad login type." + } 403: description: |- The login attempt failed. For example, the password may have been incorrect. From 270e1d7aa81b8692dbffe4604df81c96de99d6cb Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 14 Oct 2015 14:29:36 +0100 Subject: [PATCH 08/16] Add richvdh to list of trusted pushers for spectulator richvdh is awesome, he should have all the permissions. --- scripts/speculator/main.go | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/speculator/main.go b/scripts/speculator/main.go index d641d7da..dab624df 100644 --- a/scripts/speculator/main.go +++ b/scripts/speculator/main.go @@ -364,6 +364,7 @@ func main() { "illicitonion": true, "Kegsay": true, "NegativeMjark": true, + "richvdh": true, } rand.Seed(time.Now().Unix()) masterCloneDir, err := gitClone(matrixDocCloneURL, false) From 013b6db8958b7b02000044ee6829ccbdbd24a08f Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 14 Oct 2015 14:42:52 +0100 Subject: [PATCH 09/16] Add a note on building the speculator ... because the wiki says you need to read the README on how to build it. --- scripts/speculator/README | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/speculator/README b/scripts/speculator/README index 0a9f53fd..c54e0f2b 100644 --- a/scripts/speculator/README +++ b/scripts/speculator/README @@ -8,3 +8,7 @@ It serves the following HTTP endpoints: To run it, you must install the `go` tool, and run: `go run main.go` + +To build the binary (which is necessary for deployment to the matrix.org +servers), you must again install `go`, and then run: + `go build` From 3608f3a282d9d9eae775dafb30eb22b813d319a6 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 14:44:09 +0100 Subject: [PATCH 10/16] SPEC-144 : Use https examples not http. --- specification/modules/push.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specification/modules/push.rst b/specification/modules/push.rst index 6e9c8536..525e730d 100644 --- a/specification/modules/push.rst +++ b/specification/modules/push.rst @@ -353,14 +353,14 @@ Examples To create a rule that suppresses notifications for the room with ID ``!dj234r78wl45Gh4D:matrix.org``:: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/room/%21dj234r78wl45Gh4D%3Amatrix.org?access_token=123456" -d \ '{ "actions" : ["dont_notify"] }' To suppress notifications for the user ``@spambot:matrix.org``:: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org?access_token=123456" -d \ '{ "actions" : ["dont_notify"] }' @@ -368,7 +368,7 @@ To suppress notifications for the user ``@spambot:matrix.org``:: To always notify for messages that contain the work 'cake' and set a specific sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``):: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/content/SSByZWFsbHkgbGlrZSBjYWtl?access_token=123456" -d \ '{ "pattern": "cake", "actions" : ["notify", {"set_sound":"cakealarm.wav"}] @@ -377,7 +377,7 @@ sound (with a rule_id of ``SSByZWFsbHkgbGlrZSBjYWtl``):: To add a rule suppressing notifications for messages starting with 'cake' but ending with 'lie', superseding the previous rule:: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/content/U3BvbmdlIGNha2UgaXMgYmVzdA?access_token=123456&before=SSByZWFsbHkgbGlrZSBjYWtl" -d \ '{ "pattern": "cake*lie", "actions" : ["notify"] @@ -387,7 +387,7 @@ To add a custom sound for notifications messages containing the word 'beer' in any rooms with 10 members or fewer (with greater importance than the room, sender and content rules):: - curl -X PUT -H "Content-Type: application/json" "http://localhost:8008/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456" -d \ + curl -X PUT -H "Content-Type: application/json" "https://example.com/_matrix/client/api/v1/pushrules/global/override/U2VlIHlvdSBpbiBUaGUgRHVrZQ?access_token=123456" -d \ '{ "conditions": [ {"kind": "event_match", "key": "content.body", "pattern": "beer" }, From 44b19a8b0e16a25950899c4f5d90b2ea99b4de69 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 15:27:56 +0100 Subject: [PATCH 11/16] SPEC-165: Outline directory traversal attacks on MXC URIs --- specification/modules/content_repo.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/specification/modules/content_repo.rst b/specification/modules/content_repo.rst index 9ac5e199..52937ead 100644 --- a/specification/modules/content_repo.rst +++ b/specification/modules/content_repo.rst @@ -52,7 +52,17 @@ The HTTP GET endpoint does not require any authentication. Knowing the URL of the content is sufficient to retrieve the content, even if the entity isn't in the room. -Homeservers have additional concerns: +MXC URIs are vulnerable to directory traversal attacks such as +``mxc://127.0.0.1/../../../some_service/etc/passwd``. This would cause the target +homeserver to try to access and return this file. As such, homeservers MUST +sanitise MXC URIs by allowing only alphanumeric (``A-Za-z0-9``), ``_`` +and ``-`` characters in the ``server-name`` and ``media-id`` values. This set +of whitelisted characters allows URL-safe base64 encodings specified in RFC 4648. +Applying this character whitelist is preferable to blacklisting ``.`` and ``/`` +as there are techniques around blacklisted characters (percent-encoded characters, +UTF-8 encoded traversals, etc). + +Homeservers have additional content-specific concerns: - Clients may try to upload very large files. Homeservers should not store files that are too large and should not serve them to clients. From f0d80529517eddac9432daab9984c38784528a76 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Wed, 14 Oct 2015 15:42:44 +0100 Subject: [PATCH 12/16] SPEC-205: Warn about password strengths in m.login.password section --- specification/1-client_server_api.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/specification/1-client_server_api.rst b/specification/1-client_server_api.rst index 59e6b68e..a63c114a 100644 --- a/specification/1-client_server_api.rst +++ b/specification/1-client_server_api.rst @@ -215,6 +215,12 @@ To respond to this type, reply with an auth dict as follows:: "password": "" } +.. WARNING:: + Clients SHOULD enforce that the password provided is suitably complex. The + password SHOULD include a lower-case letter, an upper-case letter, a number + and a symbol and be at a minimum 8 characters in length. Servers MAY reject + weak passwords with an error code ``M_WEAK_PASSWORD``. + Google ReCaptcha ~~~~~~~~~~~~~~~~ :Type: From 70704240dcc16ab8e4c4a982ab5b73a0cc047d44 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 14 Oct 2015 16:29:18 +0100 Subject: [PATCH 13/16] gendoc should just add newlines rather than complaining about the lack thereof --- scripts/gendoc.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 8db604c4..74a6c1da 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -128,11 +128,8 @@ def get_rst(file_info, title_level, title_styles, spec_dir, adjust_titles): ) else: rst = f.read() - if rst[-2:] != "\n\n": - raise Exception( - ("File %s should end with TWO new-line characters to ensure " + - "file concatenation works correctly.") % (file_info,) - ) + + rst += "\n\n" return rst # dicts look like {0: filepath, 1: filepath} where the key is the title level elif isinstance(file_info, dict): From f51ee706330be5904077b85b595d209794f290d3 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 10:19:51 +0100 Subject: [PATCH 14/16] Review comments round 2 --- specification/modules/instant_messaging.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/specification/modules/instant_messaging.rst b/specification/modules/instant_messaging.rst index 487240de..0b4e3e64 100644 --- a/specification/modules/instant_messaging.rst +++ b/specification/modules/instant_messaging.rst @@ -84,6 +84,9 @@ Local echo Messages SHOULD appear immediately in the message view when a user presses the "send" button. This should occur even if the message is still sending. This is referred to as "local echo". Clients SHOULD implement "local echo" of messages. +Clients MAY display messages in a different format to indicate that the server +has not processed the message. This format should be removed when the server +responds. Clients need to be able to match the message they are sending with the same message which they receive from the event stream. The echo of the same message @@ -91,7 +94,7 @@ from the event stream is referred to as "remote echo". Both echoes need to be identified as the same message in order to prevent duplicate messages being displayed. Ideally this pairing would occur transparently to the user: the UI would not flicker as it transitions from local to remote. Flickering cannot be -fully avoided in version 1 of the client-server API. Two scenarios need to be +fully avoided in the current client-server API. Two scenarios need to be considered: - The client sends a message and the remote echo arrives on the event stream @@ -106,8 +109,8 @@ arrives before the client has obtained an event ID. This makes it impossible to identify it as a duplicate event. This results in the client displaying the message twice for a fraction of a second before the the original request to send the message completes. Once it completes, the client can take remedial actions -to remove the duplicate event by looking for duplicate event IDs. Version 2 of -the client-server API resolves this by attaching the transaction ID of the +to remove the duplicate event by looking for duplicate event IDs. A future version +of the client-server API will resolve this by attaching the transaction ID of the sending request to the event itself. @@ -132,7 +135,7 @@ Server behaviour ---------------- Homeservers SHOULD reject ``m.room.message`` events which don't have a -``msgtype`` key, or who don't have a textual ``body`` key, with an HTTP status +``msgtype`` key, or which don't have a textual ``body`` key, with an HTTP status code of 400. Security considerations @@ -141,7 +144,7 @@ Security considerations Messages sent using this module are not encrypted. Messages can be encrypted using the `E2E module`_. -Clients should sanitise **all keys** for unsafe HTML to prevent Cross-Site +Clients should sanitise **all displayed keys** for unsafe HTML to prevent Cross-Site Scripting (XSS) attacks. This includes room names and topics. .. _`E2E module`: `module:e2e`_ From 507cb29e39c0dec1f0ad51095f350d4e5f37e4e7 Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 10:37:15 +0100 Subject: [PATCH 15/16] Review comments round deux --- specification/modules/history_visibility.rst | 28 +++++++++++--------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 12ea7b22..0405b3f1 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -4,21 +4,25 @@ Room History Visibility .. _module:history-visibility: This module adds support for controlling the visibility of previous events in a -room. Whether a member of a room can see the events that happened in a room from -before they joined the room is controlled by the ``m.room.history_visibility`` -event outlined below. In all cases, the member still needs to be joined to the -room to receive events for that room. The visibility option simply determines -which subset of events in the room are presented to the client. Visibility can -take the form of one of three options: +room. -- ``shared`` - Previous events are always shown to newly joined members. All - events in the room are shown, even those sent when the member was not a part +In all cases, a user needs to join a room to view events in that room. Once they +have joined a room, they will gain access to a subset of events in the room. How +this subset is chosen is controlled by the ``m.room.history_visibility`` event +outlined below. After a user has left a room, they may seen any events which they +were allowed to see before they left the room, but no events received after they +left. + +The three options for this event are: + +- ``shared`` - Previous events are always accessible to newly joined members. All + events in the room are accessible, even those sent when the member was not a part of the room. -- ``invited`` - Previous events are shown to newly joined members from the point - they were invited onwards. Events stop being shown when the member's state +- ``invited`` - Previous events are accessible to newly joined members from the point + they were invited onwards. Events stop being accessible when the member's state changes to something other than ``invite`` or ``join``. -- ``joined`` - Previous events are shown to newly joined members from the point - they joined the room onwards. Events stop being shown when the members state +- ``joined`` - Previous events are accessible to newly joined members from the point + they joined the room onwards. Events stop being accessible when the members state changes to something other than ``join``. .. WARNING:: From 5aad67f74d7fe611981aa75c3b42d7c17726b4de Mon Sep 17 00:00:00 2001 From: Kegan Dougal Date: Thu, 15 Oct 2015 11:01:11 +0100 Subject: [PATCH 16/16] Review comments round III --- specification/modules/history_visibility.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/modules/history_visibility.rst b/specification/modules/history_visibility.rst index 0405b3f1..f26d1b59 100644 --- a/specification/modules/history_visibility.rst +++ b/specification/modules/history_visibility.rst @@ -13,7 +13,7 @@ outlined below. After a user has left a room, they may seen any events which the were allowed to see before they left the room, but no events received after they left. -The three options for this event are: +The three options for the ``m.room.history_visibility`` event are: - ``shared`` - Previous events are always accessible to newly joined members. All events in the room are accessible, even those sent when the member was not a part @@ -22,7 +22,7 @@ The three options for this event are: they were invited onwards. Events stop being accessible when the member's state changes to something other than ``invite`` or ``join``. - ``joined`` - Previous events are accessible to newly joined members from the point - they joined the room onwards. Events stop being accessible when the members state + they joined the room onwards. Events stop being accessible when the member's state changes to something other than ``join``. .. WARNING::