Add NATS JetStream support (#1866)

* Add NATS JetStream support Update shopify/sarama * Fix addresses * Don't change Addresses in Defaults * Update saramajetstream * Add missing error check Keep typing events for at least one minute * Use all configured NATS addresses * Update saramajetstream * Try setting up with NATS * Make sure NATS uses own persistent directory (TODO: make this configurable) * Update go.mod/go.sum * Jetstream package * Various other refactoring * Build fixes * Config tweaks, make random jetstream storage path for CI * Disable interest policies * Try to sane default on jetstream base path * Try to use in-memory for CI * Restore storage/retention * Update nats.go dependency * Adapt changes to config * Remove unneeded TopicFor * Dep update * Revert "Remove unneeded TopicFor" This reverts commit f5a4e4a339. * Revert changes made to streams * Fix build problems * Update nats-server * Update go.mod/go.sum * Roomserver input API queuing using NATS * Fix topic naming * Prometheus metrics * More refactoring to remove saramajetstream * Add missing topic * Don't try to populate map that doesn't exist * Roomserver output topic * Update go.mod/go.sum * Message acknowledgements * Ack tweaks * Try to resume transaction re-sends * Try to resume transaction re-sends * Update to matrix-org/gomatrixserverlib@91dadfb * Remove internal.PartitionStorer from components that don't consume keychanges * Try to reduce re-allocations a bit in resolveConflictsV2 * Tweak delivery options on RS input * Publish send-to-device messages into correct JetStream subject * Async and sync roomserver input * Update dendrite-config.yaml * Remove roomserver tests for now (they need rewriting) * Remove roomserver test again (was merged back in) * Update documentation * Docker updates * More Docker updates * Update Docker readme again * Fix lint issues * Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that * Go 1.16 instead of Go 1.13 for upgrade tests and Complement * Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283f. * Don't report any errors on `/send` to see what fun that creates * Fix panics on closed channel sends * Enforce state key matches sender * Do the same for leave * Various tweaks to make tests happier Squashed commit of the following: commit 13f9028e7a Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:47:14 2022 +0000 Do the same for leave commit e6be7f05c3 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 15:33:42 2022 +0000 Enforce state key matches sender commit 85ede6d64b Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 14:07:04 2022 +0000 Fix panics on closed channel sends commit 9755494a98 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:38:22 2022 +0000 Don't report any errors on `/send` to see what fun that creates commit 3bb4f87b5d Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 13:00:26 2022 +0000 Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that" This reverts commit 368675283f. commit fe2673ed7b Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 12:09:34 2022 +0000 Go 1.16 instead of Go 1.13 for upgrade tests and Complement commit 368675283f Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 11:51:45 2022 +0000 Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that commit b028dfc085 Author: Neil Alexander <neilalexander@users.noreply.github.com> Date: Tue Jan 4 10:29:08 2022 +0000 Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset) * Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork * Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers * Fix consumer component name in federation API * Add comment explaining where streams are defined * Tweaks to roomserver input with comments * Finish that sentence that I apparently forgot to finish in INSTALL.md * Bump version number of config to 2 * Add comments around asynchronous sends to roomserver in processEventWithMissingState * More useful error message when the config version does not match * Set version in generate-config * Fix version in config.Defaults Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2025-07-29 12:42:46 +00:00 · 2022-01-05 18:44:49 +01:00 · 2022-01-05 18:44:49 +01:00 · 161f145176
commit 161f145176
parent a47b12dc7d
75 changed files with 1317 additions and 1696 deletions
--- a/build/docker/README.md
+++ b/build/docker/README.md
@ -19,7 +19,6 @@ not contain the Go toolchain etc.

 There are three sample `docker-compose` files:

- `docker-compose.deps.yml` which runs the Postgres and Kafka prerequisites
 - `docker-compose.monolith.yml` which runs a monolith Dendrite deployment
 - `docker-compose.polylith.yml` which runs a polylith Dendrite deployment

@ -28,7 +27,7 @@ There are three sample `docker-compose` files:
 The `docker-compose` files refer to the `/etc/dendrite` volume as where the
 runtime config should come from. The mounted folder must contain:

- `dendrite.yaml` configuration file (based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml)
+- `dendrite.yaml` configuration file (from the [Docker config folder](https://github.com/matrix-org/dendrite/tree/master/build/docker/config)
   sample in the `build/docker/config` folder of this repository.)
 - `matrix_key.pem` server key, as generated using `cmd/generate-keys`
 - `server.crt` certificate file
@ -50,15 +49,9 @@ The key files will now exist in your current working directory, and can be mount

 ## Starting Dendrite as a monolith deployment

-Create your config based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml) configuration file in the `build/docker/config` folder of this repository. And rename the config file to `dendrite.yml` (and put it in your `config` directory).
+Create your config based on the [`dendrite.yaml`](https://github.com/matrix-org/dendrite/tree/master/build/docker/config) configuration file in the `build/docker/config` folder of this repository.

-Once in place, start the PostgreSQL dependency:
-
-```
-docker-compose -f docker-compose.deps.yml up postgres
-```
-
-Wait a few seconds for PostgreSQL to finish starting up, and then start a monolith:
+Then start the deployment:

 ```
 docker-compose -f docker-compose.monolith.yml up
@ -66,15 +59,9 @@ docker-compose -f docker-compose.monolith.yml up

 ## Starting Dendrite as a polylith deployment

-Create your config based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml) configuration file in the `build/docker/config` folder of this repository. And rename the config file to `dendrite.yml` (and put it in your `config` directory).
+Create your config based on the [`dendrite-config.yaml`](https://github.com/matrix-org/dendrite/tree/master/build/docker/config) configuration file in the `build/docker/config` folder of this repository.

-Once in place, start all the dependencies:
-
-```
-docker-compose -f docker-compose.deps.yml up
-```
-
-Wait a few seconds for PostgreSQL and Kafka to finish starting up, and then start a polylith:
+Then start the deployment:

 ```
 docker-compose -f docker-compose.polylith.yml up
--- a/build/docker/config/dendrite-config.yaml
+++ b/build/docker/config/dendrite-config.yaml
@ -28,7 +28,7 @@
 # connection can be idle in seconds - a negative value is unlimited.

 # The version of the configuration file. 
-version: 1
+version: 2

 # Global Matrix configuration. This configuration applies to all components.
 global:
@ -62,29 +62,28 @@ global:
  - matrix.org
  - vector.im

-  # Configuration for Kafka/Naffka.
-  kafka:
-    # List of Kafka broker addresses to connect to. This is not needed if using
-    # Naffka in monolith mode.
+  # Configuration for NATS JetStream
+  jetstream:
+    # A list of NATS Server addresses to connect to. If none are specified, an
+    # internal NATS server will be started automatically when running Dendrite
+    # in monolith mode. It is required to specify the address of at least one
+    # NATS Server node if running in polylith mode.
    addresses:
-      - kafka:9092
+      - jetstream:4222

-    # The prefix to use for Kafka topic names for this homeserver. Change this only if
-    # you are running more than one Dendrite homeserver on the same Kafka deployment.
+    # Keep all NATS streams in memory, rather than persisting it to the storage
+    # path below. This option is present primarily for integration testing and
+    # should not be used on a real world Dendrite deployment.
+    in_memory: false
+
+    # Persistent directory to store JetStream streams in. This directory 
+    # should be preserved across Dendrite restarts.
+    storage_path: ./
+
+    # The prefix to use for stream names for this homeserver - really only
+    # useful if running more than one Dendrite on the same NATS deployment.
    topic_prefix: Dendrite

-    # Whether to use Naffka instead of Kafka. This is only available in monolith
-    # mode, but means that you can run a single-process server without requiring
-    # Kafka.
-    use_naffka: false
-
-    # Naffka database options. Not required when using Kafka.
-    naffka_database:
-      connection_string: postgresql://dendrite:itsasecret@postgres/dendrite_naffka?sslmode=disable
-      max_open_conns: 10
-      max_idle_conns: 2
-      conn_max_lifetime: -1
-
  # Configuration for Prometheus metric collection.
  metrics:
    # Whether or not Prometheus metrics are enabled.
@ -266,6 +265,19 @@ media_api:
    height: 480
    method: scale

+# Configuration for experimental MSC's
+mscs:
+  # A list of enabled MSC's
+  # Currently valid values are:
+  # - msc2836    (Threading, see https://github.com/matrix-org/matrix-doc/pull/2836)
+  # - msc2946    (Spaces Summary, see https://github.com/matrix-org/matrix-doc/pull/2946)
+  mscs: []
+  database:
+    connection_string: postgresql://dendrite:itsasecret@postgres/dendrite_mscs?sslmode=disable
+    max_open_conns: 5
+    max_idle_conns: 2
+    conn_max_lifetime: -1
+
 # Configuration for the Room Server.
 room_server:
  internal_api:
--- a/build/docker/docker-compose.deps.yml
+++ b/build/docker/docker-compose.deps.yml
@ -1,42 +0,0 @@
-version: "3.4"
-services:
-  # PostgreSQL is needed for both polylith and monolith modes.
-  postgres:
-    hostname: postgres
-    image: postgres:14
-    restart: always
-    volumes:
-      - ./postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
-    # To persist your PostgreSQL databases outside of the Docker image, to 
-    # prevent data loss, you will need to add something like this:
-    #  - ./path/to/persistent/storage:/var/lib/postgresql/data
-    environment:
-      POSTGRES_PASSWORD: itsasecret
-      POSTGRES_USER: dendrite
-    networks:
-      - internal
-
-  # Zookeeper is only needed for polylith mode!
-  zookeeper:
-    hostname: zookeeper
-    image: zookeeper
-    networks:
-      - internal
-
-  # Kafka is only needed for polylith mode!
-  kafka:
-    container_name: dendrite_kafka
-    hostname: kafka
-    image: wurstmeister/kafka
-    environment:
-      KAFKA_ADVERTISED_HOST_NAME: "kafka"
-      KAFKA_DELETE_TOPIC_ENABLE: "true"
-      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
-    depends_on:
-      - zookeeper
-    networks:
-      - internal
-
-networks:
-  internal:
-    attachable: true
--- a/build/docker/docker-compose.monolith.yml
+++ b/build/docker/docker-compose.monolith.yml
@ -1,5 +1,25 @@
 version: "3.4"
 services:
+  postgres:
+    hostname: postgres
+    image: postgres:14
+    restart: always
+    volumes:
+      - ./postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
+      # To persist your PostgreSQL databases outside of the Docker image, 
+      # to prevent data loss, modify the following ./path_to path:
+      - ./path_to/postgresql:/var/lib/postgresql/data
+    environment:
+      POSTGRES_PASSWORD: itsasecret
+      POSTGRES_USER: dendrite
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U dendrite"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+    networks:
+      - internal
+
  monolith:
    hostname: monolith
    image: matrixdotorg/dendrite-monolith:latest
@ -13,8 +33,11 @@ services:
    volumes:
      - ./config:/etc/dendrite
      - ./media:/var/dendrite/media
+    depends_on:
+      - postgres
    networks:
      - internal
+    restart: unless-stopped

 networks:
  internal:
--- a/build/docker/docker-compose.polylith.yml
+++ b/build/docker/docker-compose.polylith.yml
@ -1,13 +1,51 @@
 version: "3.4"
 services:
+  postgres:
+    hostname: postgres
+    image: postgres:14
+    restart: always
+    volumes:
+      - ./postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
+      # To persist your PostgreSQL databases outside of the Docker image, 
+      # to prevent data loss, modify the following ./path_to path:
+      - ./path_to/postgresql:/var/lib/postgresql/data
+    environment:
+      POSTGRES_PASSWORD: itsasecret
+      POSTGRES_USER: dendrite
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U dendrite"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+    networks:
+      - internal
+
+  jetstream:
+    hostname: jetstream
+    image: nats:latest
+    command: |
+      --jetstream
+      --store_dir /var/lib/nats
+      --cluster_name Dendrite
+    volumes:
+      # To persist your NATS JetStream streams outside of the Docker image,
+      # prevent data loss, modify the following ./path_to path:
+      - ./path_to/nats:/var/lib/nats
+    networks:
+      - internal
+
  client_api:
    hostname: client_api
    image: matrixdotorg/dendrite-polylith:latest
    command: clientapi
    volumes:
      - ./config:/etc/dendrite
+    depends_on:
+      - jetstream
+      - postgres
    networks:
      - internal
+    restart: unless-stopped

  media_api:
    hostname: media_api
@ -18,6 +56,7 @@ services:
      - ./media:/var/dendrite/media
    networks:
      - internal
+    restart: unless-stopped

  sync_api:
    hostname: sync_api
@ -25,8 +64,12 @@ services:
    command: syncapi
    volumes:
      - ./config:/etc/dendrite
+    depends_on:
+      - jetstream
+      - postgres
    networks:
      - internal
+    restart: unless-stopped

  room_server:
    hostname: room_server
@ -34,8 +77,12 @@ services:
    command: roomserver
    volumes:
      - ./config:/etc/dendrite
+    depends_on:
+      - jetstream
+      - postgres
    networks:
      - internal
+    restart: unless-stopped

  edu_server:
    hostname: edu_server
@ -43,8 +90,11 @@ services:
    command: eduserver
    volumes:
      - ./config:/etc/dendrite
+    depends_on:
+      - jetstream
    networks:
      - internal
+    restart: unless-stopped

  federation_api:
    hostname: federation_api
@ -52,8 +102,12 @@ services:
    command: federationapi
    volumes:
      - ./config:/etc/dendrite
+    depends_on:
+      - jetstream
+      - postgres
    networks:
      - internal
+    restart: unless-stopped
      
  key_server:
    hostname: key_server
@ -61,8 +115,12 @@ services:
    command: keyserver
    volumes:
      - ./config:/etc/dendrite
+    depends_on:
+      - jetstream
+      - postgres
    networks:
      - internal
+    restart: unless-stopped
      
  user_api:
    hostname: user_api
@ -70,8 +128,12 @@ services:
    command: userapi
    volumes:
      - ./config:/etc/dendrite
+    depends_on:
+      - jetstream
+      - postgres
    networks:
      - internal
+    restart: unless-stopped

  appservice_api:
    hostname: appservice_api
@ -82,8 +144,11 @@ services:
    networks:
      - internal
    depends_on:
+      - jetstream
+      - postgres
      - room_server
      - user_api
+    restart: unless-stopped

 networks:
  internal:
--- a/build/docker/postgres/create_db.sh
+++ b/build/docker/postgres/create_db.sh
@ -1,5 +1,5 @@
 #!/bin/sh

-for db in userapi_accounts userapi_devices mediaapi syncapi roomserver keyserver federationapi appservice naffka; do
+for db in userapi_accounts userapi_devices mediaapi syncapi roomserver keyserver federationapi appservice mscs; do
    createdb -U dendrite -O dendrite dendrite_$db
 done