Commit graph

188 commits

Author SHA1 Message Date
Neil Alexander
91e7a82e61
Add comments around asynchronous sends to roomserver in processEventWithMissingState 2022-01-05 15:16:59 +00:00
Neil Alexander
066aeb40b4
Fix consumer component name in federation API 2022-01-05 13:46:57 +00:00
Neil Alexander
51de6612a6
Add jetstream.WithJetStreamMessage to make ack/nak-ing less messy, use process context in consumers 2022-01-05 13:45:27 +00:00
Neil Alexander
13f9028e7a
Do the same for leave 2022-01-04 15:47:14 +00:00
Neil Alexander
e6be7f05c3
Enforce state key matches sender 2022-01-04 15:33:42 +00:00
Neil Alexander
9755494a98
Don't report any errors on /send to see what fun that creates 2022-01-04 13:38:22 +00:00
Neil Alexander
b028dfc085
Send final event in processEvent synchronously (since this might stop Sytest from being so upset) 2022-01-04 10:29:08 +00:00
Neil Alexander
8d4e5b447a
Merge branch 'master' into nats 2021-12-16 10:49:09 +00:00
Neil Alexander
3113210f17
Fix keyring regressions in previous P2P demo 2021-12-13 13:24:49 +00:00
Neil Alexander
95476d36ca
Merge branch 'master' into nats 2021-11-24 12:04:32 +00:00
Neil Alexander
c9419e51af
Don't populate config defaults where it doesn't make sense (#2058)
* Don't populate config defaults where it doesn't make sense

* Fix dendritejs builds
2021-11-24 11:57:39 +00:00
Neil Alexander
843139c4f1
Merge branch 'master' into nats 2021-11-24 11:00:21 +00:00
Neil Alexander
ec716793eb
Merge federationapi, federationsender, signingkeyserver components (#2055)
* Initial federation sender -> federation API refactoring

* Move base into own package, avoids import cycle

* Fix build errors

* Fix tests

* Add signing key server tables

* Try to fold signing key server into federation API

* Fix dendritejs builds

* Update embedded interfaces

* Fix panic, fix lint error

* Update configs, docker

* Rename some things

* Reuse same keyring on the implementing side

* Fix federation tests, `NewBaseDendrite` can accept freeform options

* Fix build

* Update create_db, configs

* Name tables back

* Don't rename federationsender consumer for now
2021-11-24 10:45:23 +00:00
Neil Alexander
8cc6937fef
Async and sync roomserver input 2021-11-08 13:02:08 +00:00
Neil Alexander
4b3ae031af
Merge branch 'master' into neilalexander/natsrsinput 2021-11-08 09:53:18 +00:00
Neil Alexander
323a6fb54f
Resume federation sends (#2039)
* Resume federation sends

* Review comments

* Fix build error
2021-11-08 09:24:16 +00:00
Neil Alexander
7a4321ac85
Try to resume transaction re-sends 2021-11-03 17:28:21 +00:00
Neil Alexander
5b67f47547
Try to resume transaction re-sends 2021-11-03 17:27:18 +00:00
Neil Alexander
73d6964fb4
Merge branch 'master' into add-nats-support 2021-11-02 17:36:22 +00:00
Neil Alexander
fbd1a0ab13
Update to matrix-org/gomatrixserverlib@5e02b64 2021-11-02 10:13:38 +00:00
Ryan W
1cd4d50181
Added .well-known/matrix/server endpoint (#1988)
* Added .well-known/matrix/server endpoint

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

* Replaced tabs with spaces

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>
2021-09-10 10:05:31 +01:00
Ryan W
a624eab309
- Removed double imports (#1989)
- Lower cased error messages

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
2021-09-08 17:31:03 +01:00
kegsay
7dc8fb1fe7
Add more logs (#2005)
* Add more logs

To help debug the migration issue in #1924 along with manual data-loss-inducing fixes.
Also log the origin server on processed txns to help debug buggy server origins.

* Fix query
2021-09-07 15:07:14 +01:00
Neil Alexander
51b119107c
Don't return nonsense canonical room aliases in the public rooms responses (#1992) 2021-08-27 16:50:30 +01:00
Neil Alexander
2dd5fd1fd6
publicRooms should accept POST as well as GET (#1991) 2021-08-27 15:48:27 +01:00
Neil Alexander
ff21675c5b
Cross-signing fixes, notifications via sync, federation (#1974)
* Initial work on signing key update EDUs

* Fix build

* Produce/consume EDUs

* Producer logging

* Only produce key change notifications for local users

* Better naming

* Try to notify sync

* Enable feature

* Use key change topic

* Don't bother verifying signatures, validate key lengths if we can, notifier fixes

* Copyright notices

* Remove tests from whitelist until matrix-org/sytest#1117

* Some review comment fixes

* Update to matrix-org/gomatrixserverlib@f9416ac

* Remove unneeded parameter
2021-08-17 13:44:30 +01:00
Neil Alexander
b1377d991a
Cross-signing signature handling (#1965)
* Handle other signatures

* Decorate key ID properly

* Match by key IDs

* Tweaks

* Fixes

* Fix /user/keys/query bug, review comments, update sytest-whitelist

* Various wtweaks

* Fix wiring for keyserver in API mode

* Additional fixes
2021-08-09 14:35:24 +01:00
Neil Alexander
e95b1fd238
Cross-signing validation for self-sigs, expose signatures over /user/keys/query and /user/devices/{userId} (#1962)
* Enable unstable feature again

* Try to verify when a device signs a key

* Try to verify when a key signs a device

* It's the self-signing key, not the master key

* Fix error

* Try to verify master key uploads

* Actually we can't guarantee we can do that so nevermind

* Add signatures into /devices/list request

* Fix nil pointer

* Reprioritise map creation

* Don't skip devices that don't have signatures

* Add some debug logging

* Fix logic error in QuerySignatures

* Fix bugs

* Expose master and self-signing keys on /devices/list hopefully

* maps are tedious

* Expose signatures via /keys/query

* Upload signatures when uploading keys

* Fixes

* Disable the feature again
2021-08-06 10:13:35 +01:00
kegsay
e3df612953
Add tracing to user API (#1948)
Use the trace version in tests so we can just implement the required API functions.
2021-08-03 11:23:25 +01:00
Meenal Trivedi
fa1ec482a7
fix:Inviting to an unsupported room version return M_BAD_JSON instead of Incompatible_Version (#1930)
* fix:Inviting to an unsupported room version return M_BAD_JSON instead of M_UNSUPPORTED_ROOM_VERSION

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* fix

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* fix

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* feat: make requested changes

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* Use error typecast from matrix-org/gomatrixserverlib#272

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-07-26 10:41:58 +01:00
Neil Alexander
e2e1a966e1
Merge branch 'master' into add-nats-support 2021-07-20 10:36:13 +01:00
kegsay
af64e648d7
Fix failing Complement tests (#1931)
* Check for missing state keys to avoid panicking

* Check for not allowed errors on send_leave

* More logging

* handle send_join errors too

* Additional send_join checks

* s/join/gmsl.json/
2021-07-19 13:15:19 +01:00
kegsay
728061db03
fedsender: try to satisfy all notary key requests from the cache first (#1925)
* fedsender: try to satisfy all notary key requests from the cache first

* Linting
2021-07-16 11:35:42 +01:00
kegsay
c102adaf43
fedsender: add cache tables for notary keys (#1923)
* Add notary server tables for postgres

* Add sqlite tables

* fedsender: GetServerKeys -> QueryServerKeys

As it now checks a cache and can return multiple responses
2021-07-15 17:45:37 +01:00
Neil Alexander
48322bc2b2
Various other refactoring 2021-07-14 14:06:14 +01:00
kegsay
7df3e691f2
Fix failing complement test (#1917)
Specifically `TestBannedUserCannotSendJoin`
2021-07-13 12:22:27 +01:00
Neil Alexander
c8408a6387
Add more optimised code path for checking if we're in a room (#1909)
* Add more optimised code path for checking if we're in a room

* Fix database queries

* Fix federation API test

* Fix logging

* Review comments

* Make separate API call for room membership
2021-07-09 16:36:45 +01:00
Neil Alexander
f2974721d5
Fix concurrent map reads/writes on t.hadEvents (#1902)
* Fix concurrent map reads/writes on t.hadEvents

* Add hadEvent function
2021-07-07 18:55:44 +01:00
Neil Alexander
bcd3ef38d0
Track expiry rate on pduCountTotal 2021-07-05 13:47:37 +01:00
Neil Alexander
99d8e1c107
Federation API fixes (#1899)
* Ensure worker has work before starting goroutine

* Revert "Remove processEventWithMissingStateMutex"

This reverts commit 7f02eab47d.

* Use request context when processing transactions

* Keep goroutine count down by not starting work for things where the caller gave up

* Remove mutex, start workers at correct time
2021-07-05 12:14:31 +01:00
Neil Alexander
7f02eab47d
Remove processEventWithMissingStateMutex 2021-07-05 09:14:24 +01:00
Neil Alexander
57320897cb
Federation API workers for /send to reduce memory usage (#1897)
* Try to process rooms concurrently in FS /send

* Clean up

* Use request context so that dead things don't linger for so long

* Remove mutex

* Free up pdus slice so only references remaining are in channel

* Revert "Remove mutex"

This reverts commit 8558075e8c.

* Process EDUs in parallel

* Try refactoring /send concurrency

* Fix waitgroup

* Release on waitgroup

* Respond to transaction

* Reduce CPU usage, fix unit tests

* Tweaks

* Move into one file
2021-07-02 12:33:27 +01:00
Neil Alexander
2647f6e9c5
Fix concurrent map read/write on haveEvents (#1893) 2021-06-30 12:32:20 +01:00
Neil Alexander
b7a2d369c0
Change how servers are selected for missing auth/prev events (#1892)
* Change how servers are selected for missing auth/prev events

* Shuffle order

* Move ServersInRoomProvider into api package
2021-06-30 12:05:58 +01:00
Neil Alexander
0e69212206
Give up on loops when the context expires (#1891) 2021-06-30 10:39:47 +01:00
Neil Alexander
3afb161352
Reduce memory usage in federation /send endpoint (#1890)
* More aggressive event caching

* Deduplicate /state results

* Deduplicate more

* Ensure we use the correct list of events when excluding repeated state

* Fixes

* Ensure we track all events we already knew about properly
2021-06-30 10:01:56 +01:00
Neil Alexander
e2b6a90d90
Put gmectx back to 5 minutes 2021-06-29 10:22:26 +01:00
Neil Alexander
f645646ca9
Restore the getServers RS query (needs optimisation) 2021-06-29 09:37:28 +01:00
Neil Alexander
4417f24678
Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage
Squashed commit of the following:

commit 7fad77c10e
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 15:06:52 2021 +0100

    Fix processEventWithMissingStateMutexes

commit 138cddcac7
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:59:44 2021 +0100

    Use internal.MutexByRoom

commit 6e6f026cfa
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:50:18 2021 +0100

    Try to slow things down per room

commit b97d406dff
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:41:27 2021 +0100

    Try to slow things down

commit 8866120ebf
Merge: 9f2de8a2 4a37b19a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:40:33 2021 +0100

    Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2

commit 4a37b19a8f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:34:54 2021 +0100

    Add comments

commit f9ab3f4b81
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:31:21 2021 +0100

    Tweaks

commit 9f2de8a29c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:15:59 2021 +0100

    Ask origin only for missing things for now

commit 8fd878c75a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 11:18:11 2021 +0100

    Make sure someone wakes up

commit b63f699f1b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 11:12:58 2021 +0100

    Use a FIFO queue instead of a channel to reduce backpressure
2021-06-28 15:11:59 +01:00
Neil Alexander
080ae6a829
Move room mutex in federation API (#1830)
* Move room mutex in federation API to surround resolveStatesAndCheck

* Guard processEventWithMissingState instead

* Revert "Guard processEventWithMissingState instead"

This reverts commit 0ce88036aa79b0ce97e967afb70fe932a547d5f0.
2021-04-13 11:13:07 +01:00