Docs restructure (#2953)

Needs to be merged into `gh-pages` later on.
This commit is contained in:
Till 2023-05-30 10:02:53 +02:00 committed by GitHub
parent 11b557097c
commit f956a8c1d9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
46 changed files with 447 additions and 855 deletions

View file

@ -1,106 +0,0 @@
---
title: Optimise your installation
parent: Installation
has_toc: true
nav_order: 11
permalink: /installation/start/optimisation
---
# Optimise your installation
Now that you have Dendrite running, the following tweaks will improve the reliability
and performance of your installation.
## PostgreSQL connection limit
A PostgreSQL database engine is configured to allow only a certain number of connections.
This is typically controlled by the `max_connections` and `superuser_reserved_connections`
configuration items in `postgresql.conf`. Once these limits are violated, **PostgreSQL will
immediately stop accepting new connections** until some of the existing connections are closed.
This is a common source of misconfiguration and requires particular care.
If your PostgreSQL `max_connections` is set to `100` and `superuser_reserved_connections` is
set to `3` then you have an effective connection limit of 97 database connections. It is
therefore important to ensure that Dendrite doesn't violate that limit, otherwise database
queries will unexpectedly fail and this will cause problems both within Dendrite and for users.
If you are also running other software that uses the same PostgreSQL database engine, then you
must also take into account that some connections will be already used by your other software
and therefore will not be available to Dendrite. Check the configuration of any other software
using the same database engine for their configured connection limits and adjust your calculations
accordingly.
Dendrite has a `max_open_conns` configuration item in each `database` block to control how many
connections it will open to the database.
**If you are using the `global` database pool** then you only need to configure the
`max_open_conns` setting once in the `global` section.
**If you are defining a `database` config per component** then you will need to ensure that
the **sum total** of all configured `max_open_conns` to a given database server do not exceed
the connection limit. If you configure a total that adds up to more connections than are available
then this will cause database queries to fail.
You may wish to raise the `max_connections` limit on your PostgreSQL server to accommodate
additional connections, in which case you should also update the `max_open_conns` in your
Dendrite configuration accordingly. However be aware that this is only advisable on particularly
powerful servers that can handle the concurrent load of additional queries running at one time.
## File descriptor limit
Most platforms have a limit on how many file descriptors a single process can open. All
connections made by Dendrite consume file descriptors — this includes database connections
and network requests to remote homeservers. When participating in large federated rooms
where Dendrite must talk to many remote servers, it is often very easy to exhaust default
limits which are quite low.
We currently recommend setting the file descriptor limit to 65535 to avoid such
issues. Dendrite will log immediately after startup if the file descriptor limit is too low:
```
level=warning msg="IMPORTANT: Process file descriptor limit is currently 1024, it is recommended to raise the limit for Dendrite to at least 65535 to avoid issues"
```
UNIX systems have two limits: a hard limit and a soft limit. You can view the soft limit
by running `ulimit -Sn` and the hard limit with `ulimit -Hn`:
```bash
$ ulimit -Hn
1048576
$ ulimit -Sn
1024
```
Increase the soft limit before starting Dendrite:
```bash
ulimit -Sn 65535
```
The log line at startup should no longer appear if the limit is sufficient.
If you are running under a systemd service, you can instead add `LimitNOFILE=65535` option
to the `[Service]` section of your service unit file.
## DNS caching
Dendrite has a built-in DNS cache which significantly reduces the load that Dendrite will
place on your DNS resolver. This may also speed up outbound federation.
Consider enabling the DNS cache by modifying the `global` section of your configuration file:
```yaml
dns_cache:
enabled: true
cache_size: 4096
cache_lifetime: 600s
```
## Time synchronisation
Matrix relies heavily on TLS which requires the system time to be correct. If the clock
drifts then you may find that federation no works reliably (or at all) and clients may
struggle to connect to your Dendrite server.
Ensure that the time is synchronised on your system by enabling NTP sync.

View file

@ -7,23 +7,13 @@ permalink: /installation/planning
# Planning your installation
## Modes
Dendrite consists of several components, each responsible for a different aspect of the Matrix protocol.
Users can run Dendrite in one of two modes which dictate how these components are executed and communicate.
* **Monolith mode** runs all components in a single process. Components communicate through an internal NATS
server with generally low overhead. This mode dramatically simplifies deployment complexity and offers the
best balance between performance and resource usage for low-to-mid volume deployments.
## Databases
## Database
Dendrite can run with either a PostgreSQL or a SQLite backend. There are considerable tradeoffs
to consider:
* **PostgreSQL**: Needs to run separately to Dendrite, needs to be installed and configured separately
and and will use more resources over all, but will be **considerably faster** than SQLite. PostgreSQL
and will use more resources over all, but will be **considerably faster** than SQLite. PostgreSQL
has much better write concurrency which will allow Dendrite to process more tasks in parallel. This
will be necessary for federated deployments to perform adequately.
@ -80,18 +70,17 @@ If using the PostgreSQL database engine, you should install PostgreSQL 12 or lat
### NATS Server
Dendrite comes with a built-in [NATS Server](https://github.com/nats-io/nats-server) and
therefore does not need this to be manually installed. If you are planning a monolith installation, you
do not need to do anything.
therefore does not need this to be manually installed.
### Reverse proxy
A reverse proxy such as [Caddy](https://caddyserver.com), [NGINX](https://www.nginx.com) or
[HAProxy](http://www.haproxy.org) is useful for deployments. Configuring those is not covered in this documentation, although sample configurations
[HAProxy](http://www.haproxy.org) is useful for deployments. Configuring this is not covered in this documentation, although sample configurations
for [Caddy](https://github.com/matrix-org/dendrite/blob/main/docs/caddy) and
[NGINX](https://github.com/matrix-org/dendrite/blob/main/docs/nginx) are provided.
### Windows
Finally, if you want to build Dendrite on Windows, you will need need `gcc` in the path. The best
Finally, if you want to build Dendrite on Windows, you will need `gcc` in the path. The best
way to achieve this is by installing and building Dendrite under [MinGW-w64](https://www.mingw-w64.org/).

View file

@ -20,7 +20,7 @@ Matrix servers usually discover each other when federating using the following m
well-known file to connect to the remote homeserver;
2. If a DNS SRV delegation exists on `example.com`, use the IP address and port from the DNS SRV
record to connect to the remote homeserver;
3. If neither well-known or DNS SRV delegation are configured, attempt to connect to the remote
3. If neither well-known nor DNS SRV delegation are configured, attempt to connect to the remote
homeserver by connecting to `example.com` port TCP/8448 using HTTPS.
The exact details of how server name resolution works can be found in

View file

@ -1,21 +0,0 @@
---
title: Installing as a monolith
parent: Installation
has_toc: true
nav_order: 5
permalink: /installation/install/monolith
---
# Installing as a monolith
You can install the Dendrite monolith binary into `$GOPATH/bin` by using `go install`:
```sh
go install ./cmd/dendrite
```
Alternatively, you can specify a custom path for the binary to be written to using `go build`:
```sh
go build -o /usr/local/bin/ ./cmd/dendrite
```

View file

@ -1,42 +0,0 @@
---
title: Starting the monolith
parent: Installation
has_toc: true
nav_order: 9
permalink: /installation/start/monolith
---
# Starting the monolith
Once you have completed all of the preparation and installation steps,
you can start your Dendrite monolith deployment by starting `dendrite`:
```bash
./dendrite -config /path/to/dendrite.yaml
```
By default, Dendrite will listen HTTP on port 8008. If you want to change the addresses
or ports that Dendrite listens on, you can use the `-http-bind-address` and
`-https-bind-address` command line arguments:
```bash
./dendrite -config /path/to/dendrite.yaml \
-http-bind-address 1.2.3.4:12345 \
-https-bind-address 1.2.3.4:54321
```
## Running under systemd
A common deployment pattern is to run the monolith under systemd. For this, you
will need to create a service unit file. An example service unit file is available
in the [GitHub repository](https://github.com/matrix-org/dendrite/blob/main/docs/systemd/monolith-example.service).
Once you have installed the service unit, you can notify systemd, enable and start
the service:
```bash
systemctl daemon-reload
systemctl enable dendrite
systemctl start dendrite
journalctl -fu dendrite
```

View file

@ -0,0 +1,11 @@
---
title: Docker
parent: Installation
has_children: true
nav_order: 4
permalink: /docker
---
# Installation using Docker
This section contains documentation how to install Dendrite using Docker

View file

@ -0,0 +1,57 @@
---
title: Installation
parent: Docker
grand_parent: Installation
has_toc: true
nav_order: 1
permalink: /installation/docker/install
---
# Installing Dendrite using Docker Compose
Dendrite provides an [example](https://github.com/matrix-org/dendrite/blob/main/build/docker/docker-compose.yml)
Docker compose file, which needs some preparation to start successfully.
Please note that this compose file only has Postgres as a dependency, and you need to configure
a [reverse proxy](../planning#reverse-proxy).
## Preparations
### Generate a private key
First we'll generate private key, which is used to sign events, the following will create one in `./config`:
```bash
mkdir -p ./config
docker run --rm --entrypoint="/usr/bin/generate-keys" \
-v $(pwd)/config:/mnt \
matrixdotorg/dendrite-monolith:latest \
-private-key /mnt/matrix_key.pem
```
(**NOTE**: This only needs to be executed **once**, as you otherwise overwrite the key)
### Generate a config
Similar to the command above, we can generate a config to be used, which will use the correct paths
as specified in the example docker-compose file. Change `server` to your domain and `db` according to your changes
to the docker-compose file (`services.postgres.environment` values):
```bash
mkdir -p ./config
docker run --rm --entrypoint="/bin/sh" \
-v $(pwd)/config:/mnt \
matrixdotorg/dendrite-monolith:latest \
-c "/usr/bin/generate-config \
-dir /var/dendrite/ \
-db postgres://dendrite:itsasecret@postgres/dendrite?sslmode=disable \
-server YourDomainHere > /mnt/dendrite.yaml"
```
You can then change `config/dendrite.yaml` to your liking.
## Starting Dendrite
Once you're done changing the config, you can now start up Dendrite with
```bash
docker-compose -f docker-compose.yml up
```

11
docs/installation/helm.md Normal file
View file

@ -0,0 +1,11 @@
---
title: Helm
parent: Installation
has_children: true
nav_order: 3
permalink: /helm
---
# Helm
This section contains documentation how to use [Helm](https://helm.sh/) to install Dendrite on a [Kubernetes](https://kubernetes.io/) cluster.

View file

@ -0,0 +1,58 @@
---
title: Installation
parent: Helm
grand_parent: Installation
has_toc: true
nav_order: 1
permalink: /installation/helm/install
---
# Installing Dendrite using Helm
To install Dendrite using the Helm chart, you first have to add the repository using the following commands:
```bash
helm repo add dendrite https://matrix-org.github.io/dendrite/
helm repo update
```
Next you'll need to create a `values.yaml` file and configure it to your liking. All possible values can be found
[here](https://github.com/matrix-org/dendrite/blob/main/helm/dendrite/values.yaml), but at least you need to configure
a `server_name`, otherwise the chart will complain about it:
```yaml
dendrite_config:
global:
server_name: "localhost"
```
If you are going to use an existing Postgres database, you'll also need to configure this connection:
```yaml
dendrite_config:
global:
database:
connection_string: "postgresql://PostgresUser:PostgresPassword@PostgresHostName/DendriteDatabaseName"
max_open_conns: 90
max_idle_conns: 5
conn_max_lifetime: -1
```
## Installing with PostgreSQL
The chart comes with a dependency on Postgres, which can be installed alongside Dendrite, this needs to be enabled in
the `values.yaml`:
```yaml
postgresql:
enabled: true # this installs Postgres
primary:
persistence:
size: 1Gi # defines the size for $PGDATA
dendrite_config:
global:
server_name: "localhost"
```
Using this option, the `database.connection_string` will be set for you automatically.

View file

@ -0,0 +1,11 @@
---
title: Manual
parent: Installation
has_children: true
nav_order: 5
permalink: /manual
---
# Manual Installation
This section contains documentation how to manually install Dendrite

View file

@ -1,31 +1,26 @@
---
title: Building Dendrite
parent: Installation
title: Building/Installing Dendrite
parent: Manual
grand_parent: Installation
has_toc: true
nav_order: 3
permalink: /installation/build
nav_order: 1
permalink: /installation/manual/build
---
# Build all Dendrite commands
Dendrite has numerous utility commands in addition to the actual server binaries.
Build them all from the root of the source repo with `build.sh` (Linux/Mac):
Build them all from the root of the source repo with:
```sh
./build.sh
```
or `build.cmd` (Windows):
```powershell
build.cmd
go build -o bin/ ./cmd/...
```
The resulting binaries will be placed in the `bin` subfolder.
# Installing as a monolith
# Installing Dendrite
You can install the Dendrite monolith binary into `$GOPATH/bin` by using `go install`:
You can install the Dendrite binary into `$GOPATH/bin` by using `go install`:
```sh
go install ./cmd/dendrite

View file

@ -1,8 +1,10 @@
---
title: Preparing database storage
parent: Installation
nav_order: 3
permalink: /installation/database
nav_order: 2
parent: Manual
grand_parent: Installation
permalink: /installation/manual/database
---
# Preparing database storage
@ -13,31 +15,22 @@ may need to perform some manual steps outlined below.
## PostgreSQL
Dendrite can automatically populate the database with the relevant tables and indexes, but
it is not capable of creating the databases themselves. You will need to create the databases
it is not capable of creating the database itself. You will need to create the database
manually.
The databases **must** be created with UTF-8 encoding configured or you will likely run into problems
The database **must** be created with UTF-8 encoding configured, or you will likely run into problems
with your Dendrite deployment.
At this point, you can choose to either use a single database for all Dendrite components,
or you can run each component with its own separate database:
You will need to create a single PostgreSQL database. Deployments
can use a single global connection pool, which makes updating the configuration file much easier.
Only one database connection string to manage and likely simpler to back up the database. All
components will be sharing the same database resources (CPU, RAM, storage).
* **Single database**: You will need to create a single PostgreSQL database. Monolith deployments
can use a single global connection pool, which makes updating the configuration file much easier.
Only one database connection string to manage and likely simpler to back up the database. All
components will be sharing the same database resources (CPU, RAM, storage).
* **Separate databases**: You will need to create a separate PostgreSQL database for each
component. You will need to configure each component that has storage in the Dendrite
configuration file with its own connection parameters. Allows running a different database engine
for each component on a different machine if needs be, each with their own CPU, RAM and storage —
almost certainly overkill unless you are running a very large Dendrite deployment.
For either configuration, you will want to:
You will most likely want to:
1. Configure a role (with a username and password) which Dendrite can use to connect to the
database;
2. Create the database(s) themselves, ensuring that the Dendrite role has privileges over them.
2. Create the database itself, ensuring that the Dendrite role has privileges over them.
As Dendrite will create and manage the database tables, indexes and sequences by itself, the
Dendrite role must have suitable privileges over the database.
@ -71,27 +64,6 @@ Create the database itself, using the `dendrite` role from above:
sudo -u postgres createdb -O dendrite -E UTF-8 dendrite
```
### Multiple database creation
The following eight components require a database. In this example they will be named:
| Appservice API | `dendrite_appservice` |
| Federation API | `dendrite_federationapi` |
| Media API | `dendrite_mediaapi` |
| MSCs | `dendrite_mscs` |
| Roomserver | `dendrite_roomserver` |
| Sync API | `dendrite_syncapi` |
| Key server | `dendrite_keyserver` |
| User API | `dendrite_userapi` |
... therefore you will need to create eight different databases:
```bash
for i in appservice federationapi mediaapi mscs roomserver syncapi keyserver userapi; do
sudo -u postgres createdb -O dendrite -E UTF-8 dendrite_$i
done
```
## SQLite
**WARNING:** The Dendrite SQLite backend is slower, less reliable and not recommended for

View file

@ -1,8 +1,9 @@
---
title: Configuring Dendrite
parent: Installation
nav_order: 7
permalink: /installation/configuration
parent: Manual
grand_parent: Installation
nav_order: 3
permalink: /installation/manual/configuration
---
# Configuring Dendrite
@ -20,7 +21,7 @@ sections:
First of all, you will need to configure the server name of your Matrix homeserver.
This must match the domain name that you have selected whilst [configuring the domain
name delegation](domainname).
name delegation](domainname#delegation).
In the `global` section, set the `server_name` to your delegated domain name:
@ -44,7 +45,7 @@ global:
## JetStream configuration
Monolith deployments can use the built-in NATS Server rather than running a standalone
Dendrite deployments can use the built-in NATS Server rather than running a standalone
server. If you want to use a standalone NATS Server anyway, you can also configure that too.
### Built-in NATS Server
@ -56,7 +57,6 @@ configured and set a `storage_path` to a persistent folder on the filesystem:
global:
# ...
jetstream:
in_memory: false
storage_path: /path/to/storage/folder
topic_prefix: Dendrite
```
@ -79,22 +79,17 @@ You do not need to configure the `storage_path` when using a standalone NATS Ser
In the case that you are connecting to a multi-node NATS cluster, you can configure more than
one address in the `addresses` field.
## Database connections
## Database connection using a global connection pool
Configuring database connections varies based on the [database configuration](database)
that you chose.
### Global connection pool
If you want to use a single connection pool to a single PostgreSQL database, then you must
uncomment and configure the `database` section within the `global` section:
If you want to use a single connection pool to a single PostgreSQL database,
then you must uncomment and configure the `database` section within the `global` section:
```yaml
global:
# ...
database:
connection_string: postgres://user:pass@hostname/database?sslmode=disable
max_open_conns: 100
max_open_conns: 90
max_idle_conns: 5
conn_max_lifetime: -1
```
@ -104,42 +99,13 @@ configuration file, e.g. under the `app_service_api`, `federation_api`, `key_ser
`media_api`, `mscs`, `relay_api`, `room_server`, `sync_api` and `user_api` blocks, otherwise
these will override the `global` database configuration.
### Per-component connections (all other configurations)
If you are are using SQLite databases or separate PostgreSQL
databases per component, then you must instead configure the `database` sections under each
of the component blocks ,e.g. under the `app_service_api`, `federation_api`, `key_server`,
`media_api`, `mscs`, `relay_api`, `room_server`, `sync_api` and `user_api` blocks.
For example, with PostgreSQL:
```yaml
room_server:
# ...
database:
connection_string: postgres://user:pass@hostname/dendrite_component?sslmode=disable
max_open_conns: 10
max_idle_conns: 2
conn_max_lifetime: -1
```
... or with SQLite:
```yaml
room_server:
# ...
database:
connection_string: file:roomserver.db
max_open_conns: 10
max_idle_conns: 2
conn_max_lifetime: -1
```
## Full-text search
Dendrite supports experimental full-text indexing using [Bleve](https://github.com/blevesearch/bleve). It is configured in the `sync_api` section as follows.
Dendrite supports full-text indexing using [Bleve](https://github.com/blevesearch/bleve). It is configured in the `sync_api` section as follows.
Depending on the language most likely to be used on the server, it might make sense to change the `language` used when indexing, to ensure the returned results match the expectations. A full list of possible languages can be found [here](https://github.com/blevesearch/bleve/tree/master/analysis/lang).
Depending on the language most likely to be used on the server, it might make sense to change the `language` used when indexing,
to ensure the returned results match the expectations. A full list of possible languages
can be found [here](https://github.com/matrix-org/dendrite/blob/5b73592f5a4dddf64184fcbe33f4c1835c656480/internal/fulltext/bleve.go#L25-L46).
```yaml
sync_api:

View file

@ -1,8 +1,9 @@
---
title: Generating signing keys
parent: Installation
nav_order: 8
permalink: /installation/signingkeys
parent: Manual
grand_parent: Installation
nav_order: 4
permalink: /installation/manual/signingkeys
---
# Generating signing keys
@ -11,7 +12,7 @@ All Matrix homeservers require a signing private key, which will be used to auth
federation requests and events.
The `generate-keys` utility can be used to generate a private key. Assuming that Dendrite was
built using `build.sh`, you should find the `generate-keys` utility in the `bin` folder.
built using `go build -o bin/ ./cmd/...`, you should find the `generate-keys` utility in the `bin` folder.
To generate a Matrix signing private key:

View file

@ -0,0 +1,26 @@
---
title: Starting Dendrite
parent: Manual
grand_parent: Installation
nav_order: 5
permalink: /installation/manual/start
---
# Starting Dendrite
Once you have completed all preparation and installation steps,
you can start your Dendrite deployment by executing the `dendrite` binary:
```bash
./dendrite -config /path/to/dendrite.yaml
```
By default, Dendrite will listen HTTP on port 8008. If you want to change the addresses
or ports that Dendrite listens on, you can use the `-http-bind-address` and
`-https-bind-address` command line arguments:
```bash
./dendrite -config /path/to/dendrite.yaml \
-http-bind-address 1.2.3.4:12345 \
-https-bind-address 1.2.3.4:54321
```