Commit graph

1734 commits

Author SHA1 Message Date
Neil Alexander
bcda9729e4
Max ack wait increase, other tweaks 2022-01-24 13:21:35 +00:00
Neil Alexander
2a911aa892
Try that again 2022-01-24 12:56:14 +00:00
Neil Alexander
0308cebc6a
Detect dupes from NATS 2022-01-24 12:48:36 +00:00
Neil Alexander
9ddb8749c1
Use work queue policy, deliver all on restart 2022-01-24 11:59:28 +00:00
Neil Alexander
03a989d5c9
Set timeouts on roomserver input tasks (need to decide what timeout makes sense) 2022-01-24 11:15:20 +00:00
Neil Alexander
c68037b3e8
Apply backpressure to consumers/synchronous requests to hopefully stop things being overwhelmed 2022-01-24 10:37:19 +00:00
Neil Alexander
5ed90caaff
Allow async inputs again 2022-01-21 17:20:02 +00:00
Neil Alexander
141200d22e
Room version tweaks 2022-01-21 17:18:16 +00:00
Neil Alexander
a290dafa60
Merge branch 'master' into neilalexander/federationinput 2022-01-21 16:51:10 +00:00
Neil Alexander
42511199cb
We only have missing prev if we don't supply state 2022-01-21 16:41:38 +00:00
kegsay
de38be469a
Expand issue template (#2103) 2022-01-21 16:24:31 +00:00
Neil Alexander
cc2f8a799b
Tweaks 2022-01-21 15:32:59 +00:00
Neil Alexander
ae3c4f3c56
Only search for missing prev events or state for new events 2022-01-21 15:18:51 +00:00
Neil Alexander
7a93bb32e7
Merge branch 'master' into neilalexander/federationinput 2022-01-21 15:02:30 +00:00
Neil Alexander
96bf4aa838
Add Forward extremities remain so even after the next events are populated as outliers to sytest-whitelist 2022-01-21 14:59:47 +00:00
kegsay
cd1391fc62
Document log levels (#2101) 2022-01-21 14:46:47 +00:00
FORCHA
3c92b55bec
Update monolith-sample.conf (#2087)
* Update monolith-sample.conf

-Replaced undefined monolith value  with server_name  (my.hostname.com) value in reference tho ths issue https://github.com/matrix-org/dendrite/issues/2078

* Update monolith-sample.conf

Changed IP to location of monolith server

Co-authored-by: kegsay <kegan@matrix.org>
2022-01-21 14:37:59 +00:00
kegsay
0bf5104bbb
Fix #2027 by gracefully handling stub rooms (#2100)
The server ACL code on startup will grab all known rooms from
the rooms_table and then call `GetStateEvent` with each found
room ID to find the server ACL event. This can fail for stub
rooms, which will be present in the rooms table. Previously
this would result in an error being returned and the server
failing to start (!). Now we just return no event for stub
rooms.
2022-01-21 14:23:37 +00:00
Neil Alexander
9191b23ae0
Revert "Use given event kind when sending auth/state events"
This reverts commit 089d64d271.
2022-01-21 14:15:23 +00:00
Neil Alexander
089d64d271
Use given event kind when sending auth/state events 2022-01-21 13:44:39 +00:00
Neil Alexander
75cfec95e2
Send create room events as new 2022-01-21 13:44:27 +00:00
Neil Alexander
4253eccfb3
Handle create events better 2022-01-21 11:26:26 +00:00
Neil Alexander
61437d6e44
Try that again 2022-01-21 11:14:19 +00:00
Neil Alexander
f226066783
Don't check missing state for create events 2022-01-21 10:57:38 +00:00
Neil Alexander
d93ddb08d1
Care about state when checking missing prev events 2022-01-21 10:26:24 +00:00
kegsay
2c581377a5
Remodel how device list change IDs are created (#2098)
* Remodel how device list change IDs are created

Previously we made them using the offset Kafka supplied.
We don't run Kafka anymore, so now we make the SQL table assign
the change ID via an AUTOINCREMENTing ID. Redesign the
`keyserver_key_changes` table to have `UNIQUE(user_id)` so we
don't accumulate key changes forevermore, we now have at most 1
row per user which contains the highest change ID.

This needs a SQL migration.

* Ensure we bump the change ID on sqlite

* Actually read the DeviceChangeID not the Offset in synapi

* Add SQL migrations

* Prepare after migration; fixup dendrite-upgrade-test logging

* Use higher version numbers; fix sqlite query to increment better

* Default 0 on postgres

* fixup postgres migration on fresh dendrite instances
2022-01-21 09:56:06 +00:00
Neil Alexander
078345a29d
Tweak logic around checking missing state 2022-01-21 09:55:13 +00:00
Neil Alexander
848f5487f0
Track missing prev events separately to avoid calculating state when not possible 2022-01-20 17:42:45 +00:00
Neil Alexander
b143d7bc88
Try skipping unmarshalling errors for RespMissingEvents 2022-01-20 16:52:17 +00:00
kegsay
db7d9cba8a
BREAKING: Remove Partitioned Stream Positions (#2096)
* go mod tidy

* Break complement to check it fails CI

* Remove partitioned stream positions

This was used by the device list stream position. The device list position
now corresponds to the `Offset`, and the partition is always 0, in prep
for removing reliance on Kafka topics for device list changes.

* Linting

* Migrate old style tokens to new style because element-web doesn't soft-logoout on 4xx errors on /sync
2022-01-20 15:26:45 +00:00
Neil Alexander
8db25eaa65
Add workerForRoom for tidiness 2022-01-20 14:53:38 +00:00
Neil Alexander
95eb3545d6
Tweak metrics 2022-01-17 13:39:34 +00:00
Neil Alexander
eef6940b64
Set max payload in NATS to 16MB 2022-01-17 13:27:21 +00:00
Neil Alexander
3a62494082
Report errors on sending to RS input 2022-01-17 11:53:42 +00:00
Neil Alexander
14327faaf6
Submit async 2022-01-17 11:45:03 +00:00
Neil Alexander
8236478dc3
Simplify /send endpoint significantly 2022-01-17 11:40:42 +00:00
Neil Alexander
0f5049279c
Clean up some commented out bits 2022-01-17 11:06:22 +00:00
Neil Alexander
3c6ce82060
Merge branch 'master' into neilalexander/federationinput 2022-01-10 11:09:04 +00:00
Neil Alexander
16035b9737
NATS JetStream tweaks (#2086)
* Use named NATS durable consumers

* Build fixes

* Remove dupe call to SetFederationAPI

* Use namespaced consumer name

* Fix namespacing

* Fix unit tests hopefully
2022-01-07 17:31:57 +00:00
Neil Alexander
acf5d71c21
Merge branch 'master' into neilalexander/federationinput 2022-01-07 13:43:25 +00:00
Neil Alexander
a422321435
Fix panic at startup if roomserver was not given federation API reference by the time NATS consumes an event, tweak backpressure metrics 2022-01-07 13:41:53 +00:00
Neil Alexander
10e1d347e2
Use synchronous contexts, limit time to fetch missing events 2022-01-07 11:57:11 +00:00
Neil Alexander
af34b4abe3
Reject instead of soft-fail, don't copy roominfo so much 2022-01-07 10:50:19 +00:00
Neil Alexander
eff348bb69
Check missing state if not an outlier before storing the event 2022-01-07 09:47:53 +00:00
Neil Alexander
110ab7b8f3
Don't mark state events with zero snapshot NID as not existing 2022-01-06 17:22:46 +00:00
kegsay
173b1e8d3e
Fix #2084 - incorrect /event_auth response (#2085)
* Fix #2084

* Return early

* Linting
2022-01-06 17:13:34 +00:00
Neil Alexander
ad19c2b81a
Merge branch 'master' into neilalexander/federationinput 2022-01-06 13:14:48 +00:00
S7evinK
161f145176
Add NATS JetStream support (#1866)
* Add NATS JetStream support
Update shopify/sarama

* Fix addresses

* Don't change Addresses in Defaults

* Update saramajetstream

* Add missing error check

Keep typing events for at least one minute

* Use all configured NATS addresses

* Update saramajetstream

* Try setting up with NATS

* Make sure NATS uses own persistent directory (TODO: make this configurable)

* Update go.mod/go.sum

* Jetstream package

* Various other refactoring

* Build fixes

* Config tweaks, make random jetstream storage path for CI

* Disable interest policies

* Try to sane default on jetstream base path

* Try to use in-memory for CI

* Restore storage/retention

* Update nats.go dependency

* Adapt changes to config

* Remove unneeded TopicFor

* Dep update

* Revert "Remove unneeded TopicFor"

This reverts commit f5a4e4a339.

* Revert changes made to streams

* Fix build problems

* Update nats-server

* Update go.mod/go.sum

* Roomserver input API queuing using NATS

* Fix topic naming

* Prometheus metrics

* More refactoring to remove saramajetstream

* Add missing topic

* Don't try to populate map that doesn't exist

* Roomserver output topic

* Update go.mod/go.sum

* Message acknowledgements

* Ack tweaks

* Try to resume transaction re-sends

* Try to resume transaction re-sends

* Update to matrix-org/gomatrixserverlib@91dadfb

* Remove internal.PartitionStorer from components that don't consume keychanges

* Try to reduce re-allocations a bit in resolveConflictsV2

* Tweak delivery options on RS input

* Publish send-to-device messages into correct JetStream subject

* Async and sync roomserver input

* Update dendrite-config.yaml

* Remove roomserver tests for now (they need rewriting)

* Remove roomserver test again (was merged back in)

* Update documentation

* Docker updates

* More Docker updates

* Update Docker readme again

* Fix lint issues

* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

* Go 1.16 instead of Go 1.13 for upgrade tests and Complement

* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

This reverts commit 368675283f.

* Don't report any errors on `/send` to see what fun that creates

* Fix panics on closed channel sends

* Enforce state key matches sender

* Do the same for leave

* Various tweaks to make tests happier

Squashed commit of the following:

commit 13f9028e7a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:47:14 2022 +0000

    Do the same for leave

commit e6be7f05c3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:33:42 2022 +0000

    Enforce state key matches sender

commit 85ede6d64b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 14:07:04 2022 +0000

    Fix panics on closed channel sends

commit 9755494a98
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:38:22 2022 +0000

    Don't report any errors on `/send` to see what fun that creates

commit 3bb4f87b5d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:00:26 2022 +0000

    Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

    This reverts commit 368675283f.

commit fe2673ed7b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 12:09:34 2022 +0000

    Go 1.16 instead of Go 1.13 for upgrade tests and Complement

commit 368675283f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 11:51:45 2022 +0000

    Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

commit b028dfc085
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 10:29:08 2022 +0000

    Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork

* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers

* Fix consumer component name in  federation API

* Add comment explaining where streams are defined

* Tweaks to roomserver input with comments

* Finish that sentence that I apparently forgot to finish in INSTALL.md

* Bump version number of config to 2

* Add comments around asynchronous sends to roomserver in processEventWithMissingState

* More useful error message when the config version does not match

* Set version in generate-config

* Fix version in config.Defaults

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-01-05 17:44:49 +00:00
S7evinK
a47b12dc7d
Use non http API as fsAPI to avoid network requests (#2075)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2021-12-20 10:27:58 +00:00
S7evinK
640ba425e2
Allow partial test runs (#2076)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2021-12-20 10:26:23 +00:00