hoernschen/dendrite
1
0
Fork 0
mirror of https://github.com/hoernschen/dendrite.git synced 2025-07-31 13:22:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Docs restructure (#2953)

Browse source 
Needs to be merged into `gh-pages` later on.
This commit is contained in:
Till 2023-05-30 10:02:53 +02:00 • committed by GitHub
parent 11b557097c
commit f956a8c1d9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
46 changed files with 447 additions and 855 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/development/CONTRIBUTING.md
View file

@ -1,6 +1,7 @@
---
title: Contributing
parent: Development
nav_order: 1
permalink: /development/contributing
---

1
docs/development/PROFILING.md
View file

@ -1,6 +1,7 @@
---
title: Profiling
parent: Development
nav_order: 4
permalink: /development/profiling
---

164
docs/development/coverage.md
View file

@ -1,78 +1,130 @@
---
title: Coverage
parent: Development
nav_order: 3
permalink: /development/coverage
---
To generate a test coverage report for Sytest, a small patch needs to be applied to the Sytest repository to compile and use the instrumented binary:
```patch
diff --git a/lib/SyTest/Homeserver/Dendrite.pm b/lib/SyTest/Homeserver/Dendrite.pm
index 8f0e209c..ad057e52 100644
--- a/lib/SyTest/Homeserver/Dendrite.pm
+++ b/lib/SyTest/Homeserver/Dendrite.pm
@@ -337,7 +337,7 @@ sub _start_monolith
$output->diag( "Starting monolith server" );
my @command = (
- $self->{bindir} . '/dendrite',
+ $self->{bindir} . '/dendrite', '--test.coverprofile=' . $self->{hs_dir} . '/integrationcover.log', "DEVEL",
'--config', $self->{paths}{config},
'--http-bind-address', $self->{bind_host} . ':' . $self->unsecure_port,
'--https-bind-address', $self->{bind_host} . ':' . $self->secure_port,
diff --git a/scripts/dendrite_sytest.sh b/scripts/dendrite_sytest.sh
index f009332b..7ea79869 100755
--- a/scripts/dendrite_sytest.sh
+++ b/scripts/dendrite_sytest.sh
@@ -34,7 +34,8 @@ export GOBIN=/tmp/bin
echo >&2 "--- Building dendrite from source"
cd /src
mkdir -p $GOBIN
-go install -v ./cmd/dendrite
+# go install -v ./cmd/dendrite
+go test -c -cover -covermode=atomic -o $GOBIN/dendrite -coverpkg "github.com/matrix-org/..." ./cmd/dendrite
go install -v ./cmd/generate-keys
cd -
```
## Running unit tests with coverage enabled
Running unit tests with coverage enabled can be done with the following commands, this will generate a `integrationcover.log`
```bash
go test -covermode=atomic -coverpkg=./... -coverprofile=integrationcover.log $(go list ./... | grep -v '/cmd/')
go tool cover -func=integrationcover.log
```
## Running Sytest with coverage enabled
To run Sytest with coverage enabled:
```bash
docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
-v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
-v "/Users/kegan/go/:/gopath" -e "POSTGRES=1" \
-e "COVER=1" \
matrixdotorg/sytest-dendrite:latest
# to get a more accurate coverage you may also need to run Sytest using SQLite as the database:
docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
-v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
-v "/Users/kegan/go/:/gopath" \
-e "COVER=1" \
matrixdotorg/sytest-dendrite:latest
```
This will generate a folder `covdatafiles` in each server's directory, e.g `server-0/covdatafiles`. To parse them,
ensure your working directory is under the Dendrite repository then run:
Then run Sytest. This will generate a new file `integrationcover.log` in each server's directory e.g `server-0/integrationcover.log`. To parse it,
ensure your working directory is under the Dendrite repository then run:
```bash
go tool cover -func=/path/to/server-0/integrationcover.log
go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
```
which will produce an output like:
```
...
github.com/matrix-org/util/json.go:83: NewJSONRequestHandler 100.0%
github.com/matrix-org/util/json.go:90: Protect 57.1%
github.com/matrix-org/util/json.go:110: RequestWithLogging 100.0%
github.com/matrix-org/util/json.go:132: MakeJSONAPI 70.0%
github.com/matrix-org/util/json.go:151: respond 61.5%
github.com/matrix-org/util/json.go:180: WithCORSOptions 0.0%
github.com/matrix-org/util/json.go:191: SetCORSHeaders 100.0%
github.com/matrix-org/util/json.go:202: RandomString 100.0%
github.com/matrix-org/util/json.go:210: init 100.0%
github.com/matrix-org/util/unique.go:13: Unique 91.7%
github.com/matrix-org/util/unique.go:48: SortAndUnique 100.0%
github.com/matrix-org/util/unique.go:55: UniqueStrings 100.0%
total: (statements) 53.7%
github.com/matrix-org/util/json.go:132: MakeJSONAPI 70.0%
github.com/matrix-org/util/json.go:151: respond 84.6%
github.com/matrix-org/util/json.go:180: WithCORSOptions 0.0%
github.com/matrix-org/util/json.go:191: SetCORSHeaders 100.0%
github.com/matrix-org/util/json.go:202: RandomString 100.0%
github.com/matrix-org/util/json.go:210: init 100.0%
github.com/matrix-org/util/unique.go:13: Unique 91.7%
github.com/matrix-org/util/unique.go:48: SortAndUnique 100.0%
github.com/matrix-org/util/unique.go:55: UniqueStrings 100.0%
total (statements) 64.0%
```
The total coverage for this run is the last line at the bottom. However, this value is misleading because Dendrite can run in many different configurations,
which will never be tested in a single test run (e.g sqlite or postgres). To get a more accurate value, additional processing is required
to remove packages which will never be tested and extension MSCs:
(after running Sytest for Postgres _and_ SQLite)
The total coverage for this run is the last line at the bottom. However, this value is misleading because Dendrite can run in different configurations,
which will never be tested in a single test run (e.g sqlite or postgres). To get a more accurate value, you'll need run Sytest for Postgres and SQLite (see commands above).
Additional processing is required also to remove packages which will never be tested and extension MSCs:
```bash
# These commands are all similar but change which package paths are _removed_ from the output.
# If you executed both commands from above, you can get the total coverage using the following commands
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
# For Postgres
go tool cover -func=/path/to/server-0/integrationcover.log | grep 'github.com/matrix-org/dendrite' | grep -Ev 'inthttp|sqlite|setup/mscs|api_trace' > coverage.txt
# If you only executed the one for Postgres:
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|sqlite|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
# For SQLite
go tool cover -func=/path/to/server-0/integrationcover.log | grep 'github.com/matrix-org/dendrite' | grep -Ev 'inthttp|postgres|setup/mscs|api_trace' > coverage.txt
# If you only executed the one for SQLite:
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|postgres|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov
```
A total value can then be calculated using:
## Getting coverage from Complement
Getting the coverage for Complement runs is a bit more involved.
First you'll need a docker image compatible with Complement, one can be built using
```bash
cat coverage.txt | awk -F '\t+' '{x = x + $3} END {print x/NR}'
docker build -t complement-dendrite -f build/scripts/Complement.Dockerfile .
```
from within the Dendrite repository.
Clone complement to a directory of your liking:
```bash
git clone https://github.com/matrix-org/complement.git
cd complement
```
Next we'll need a script to execute after a test finishes, create a new file `posttest.sh`, make the file executable (`chmod +x posttest.sh`)
and add the following content:
```bash
#!/bin/bash
We currently do not have a way to combine Sytest/Complement/Unit Tests into a single coverage report.
mkdir -p /tmp/Complement/logs/$2/$1/
docker cp $1:/tmp/covdatafiles/. /tmp/Complement/logs/$2/$1/
```
This will copy the `covdatafiles` files from each container to something like
`/tmp/Complement/logs/TestLogin/94f9c428de95779d2b62a3ccd8eab9d5ddcf65cc259a40ece06bdc61687ffed3/`. (`$1` is the containerID, `$2` the test name)
Now that we have set up everything we need, we can finally execute Complement:
```bash
COMPLEMENT_BASE_IMAGE=complement-dendrite \
COMPLEMENT_SHARE_ENV_PREFIX=COMPLEMENT_DENDRITE_ \
COMPLEMENT_DENDRITE_COVER=1 \
COMPLEMENT_POST_TEST_SCRIPT=$(pwd)/posttest.sh \
go test -tags dendrite_blacklist ./tests/... -count=1 -v -timeout=30m -failfast=false
```
Once this is done, you can copy the resulting `covdatafiles` files to your Dendrite repository for the next step.
```bash
cp -pr /tmp/Complement/logs PathToYourDendriteRepository
```
You can also run the following to get the coverage for Complement runs alone:
```bash
go tool covdata func -i="$(find /tmp/Complement -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
```
## Combining the results of (almost) all runs
Now that we have all our `covdatafiles` files within the Dendrite repository, you can now execute the following command, to get the coverage
overall (excluding unit tests):
```bash
go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"
```

70
docs/development/sytest.md
View file

@ -1,6 +1,7 @@
---
title: SyTest
parent: Development
nav_order: 2
permalink: /development/sytest
---
@ -23,7 +24,7 @@ After running the tests, a script will print the tests you need to add to
You should proceed after you see no build problems for dendrite after running:
```sh
./build.sh
go build -o bin/ ./cmd/...
```
If you are fixing an issue marked with
@ -61,6 +62,8 @@ When debugging, the following Docker `run` options may also be useful:
* `-e "DENDRITE_TRACE_HTTP=1"`: Adds HTTP tracing to server logs.
* `-e "DENDRITE_TRACE_INTERNAL=1"`: Adds roomserver internal API tracing to
server logs.
* `-e "COVER=1"`: Run Sytest with an instrumented binary, producing a Go coverage file per server.
* `-e "RACE_DETECTION=1"`: Build the binaries with the `-race` flag (Note: This will significantly slow down test runs)
The docker command also supports a single positional argument for the test file to
run, so you can run a single `.pl` file rather than the whole test suite. For example:
@ -71,68 +74,3 @@ docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest"
-v "/Users/kegan/go/:/gopath" -e "POSTGRES=1" -e "DENDRITE_TRACE_HTTP=1"
matrixdotorg/sytest-dendrite:latest tests/50federation/40devicelists.pl
```
### Manually Setting up SyTest
**We advise AGAINST using manual SyTest setups.**
If you don't want to use the Docker image, you can also run SyTest by hand. Make
sure you have Perl 5 or above, and get SyTest with:
(Note that this guide assumes your SyTest checkout is next to your
`dendrite` checkout.)
```sh
git clone -b develop https://github.com/matrix-org/sytest
cd sytest
./install-deps.pl
```
Set up the database:
```sh
sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
sudo -u postgres psql -c "ALTER USER dendrite CREATEDB"
for i in dendrite0 dendrite1 sytest_template; do sudo -u postgres psql -c "CREATE DATABASE $i OWNER dendrite;"; done
mkdir -p "server-0"
cat > "server-0/database.yaml" << EOF
args:
user: dendrite
password: itsasecret
database: dendrite0
host: 127.0.0.1
sslmode: disable
type: pg
EOF
mkdir -p "server-1"
cat > "server-1/database.yaml" << EOF
args:
user: dendrite
password: itsasecret
database: dendrite1
host: 127.0.0.1
sslmode: disable
type: pg
EOF
```
Run the tests:
```sh
POSTGRES=1 ./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/sytest-whitelist -O tap --all | tee results.tap
```
where `tee` lets you see the results while they're being piped to the file, and
`POSTGRES=1` enables testing with PostgeSQL. If the `POSTGRES` environment
variable is not set or is set to 0, SyTest will fall back to SQLite 3. For more
flags and options, see <https://github.com/matrix-org/sytest#running>.
Once the tests are complete, run the helper script to see if you need to add
any newly passing test names to `sytest-whitelist` in the project's root
directory:
```sh
../dendrite/show-expected-fail-tests.sh results.tap ../dendrite/sytest-whitelist ../dendrite/sytest-blacklist
```
If the script prints nothing/exits with 0, then you're good to go.

114
docs/development/tracing/opentracing.md
View file

@ -1,114 +0,0 @@
---
title: OpenTracing
has_children: true
parent: Development
permalink: /development/opentracing
---
# OpenTracing
Dendrite extensively uses the [opentracing.io](http://opentracing.io) framework
to trace work across the different logical components.
At its most basic opentracing tracks "spans" of work; recording start and end
times as well as any parent span that caused the piece of work.
A typical example would be a new span being created on an incoming request that
finishes when the response is sent. When the code needs to hit out to a
different component a new span is created with the initial span as its parent.
This would end up looking roughly like:
```
Received request Sent response
|<───────────────────────────────────────>|
|<────────────────────>|
RPC call RPC call returns
```
This is useful to see where the time is being spent processing a request on a
component. However, opentracing allows tracking of spans across components. This
makes it possible to see exactly what work goes into processing a request:
```
Component 1 |<─────────────────── HTTP ────────────────────>|
|<──────────────── RPC ─────────────────>|
Component 2 |<─ SQL ─>| |<── RPC ───>|
Component 3 |<─ SQL ─>|
```
This is achieved by serializing span information during all communication
between components. For HTTP requests, this is achieved by the sender
serializing the span into a HTTP header, and the receiver deserializing the span
on receipt. (Generally a new span is then immediately created with the
deserialized span as the parent).
A collection of spans that are related is called a trace.
Spans are passed through the code via contexts, rather than manually. It is
therefore important that all spans that are created are immediately added to the
current context. Thankfully the opentracing library gives helper functions for
doing this:
```golang
span, ctx := opentracing.StartSpanFromContext(ctx, spanName)
defer span.Finish()
```
This will create a new span, adding any span already in `ctx` as a parent to the
new span.
Adding Information
------------------
Opentracing allows adding information to a trace via three mechanisms:
- "tags" ─ A span can be tagged with a key/value pair. This is typically
information that relates to the span, e.g. for spans created for incoming HTTP
requests could include the request path and response codes as tags, spans for
SQL could include the query being executed.
- "logs" ─ Key/value pairs can be looged at a particular instance in a trace.
This can be useful to log e.g. any errors that happen.
- "baggage" ─ Arbitrary key/value pairs can be added to a span to which all
child spans have access. Baggage isn't saved and so isn't available when
inspecting the traces, but can be used to add context to logs or tags in child
spans.
See
[specification.md](https://github.com/opentracing/specification/blob/master/specification.md)
for some of the common tags and log fields used.
Span Relationships
------------------
Spans can be related to each other. The most common relation is `childOf`, which
indicates the child span somehow depends on the parent span ─ typically the
parent span cannot complete until all child spans are completed.
A second relation type is `followsFrom`, where the parent has no dependence on
the child span. This usually indicates some sort of fire and forget behaviour,
e.g. adding a message to a pipeline or inserting into a kafka topic.
Jaeger
------
Opentracing is just a framework. We use
[jaeger](https://github.com/jaegertracing/jaeger) as the actual implementation.
Jaeger is responsible for recording, sending and saving traces, as well as
giving a UI for viewing and interacting with traces.
To enable jaeger a `Tracer` object must be instansiated from the config (as well
as having a jaeger server running somewhere, usually locally). A `Tracer` does
several things:
- Decides which traces to save and send to the server. There are multiple
schemes for doing this, with a simple example being to save a certain fraction
of traces.
- Communicating with the jaeger backend. If not explicitly specified uses the
default port on localhost.
- Associates a service name to all spans created by the tracer. This service
name equates to a logical component, e.g. spans created by clientapi will have
a different service name than ones created by the syncapi. Database access
will also typically use a different service name.
This means that there is a tracer per service name/component.

57
docs/development/tracing/setup.md
View file

@ -1,57 +0,0 @@
---
title: Setup
parent: OpenTracing
grand_parent: Development
permalink: /development/opentracing/setup
---
# OpenTracing Setup
Dendrite uses [Jaeger](https://www.jaegertracing.io/) for tracing between microservices.
Tracing shows the nesting of logical spans which provides visibility on how the microservices interact.
This document explains how to set up Jaeger locally on a single machine.
## Set up the Jaeger backend
The [easiest way](https://www.jaegertracing.io/docs/1.18/getting-started/) is to use the all-in-one Docker image:
```
$ docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 14250:14250 \
-p 9411:9411 \
jaegertracing/all-in-one:1.18
```
## Configuring Dendrite to talk to Jaeger
Modify your config to look like: (this will send every single span to Jaeger which will be slow on large instances, but for local testing it's fine)
```
tracing:
enabled: true
jaeger:
serviceName: "dendrite"
disabled: false
rpc_metrics: true
tags: []
sampler:
type: const
param: 1
```
then run the monolith server:
```
./dendrite --tls-cert server.crt --tls-key server.key --config dendrite.yaml
```
## Checking traces
Visit <http://localhost:16686> to see traces under `DendriteMonolith`.