Commit graph

187 commits

Author SHA1 Message Date
Neil Alexander
19ee1a4f38
Push fixes 2021-11-15 11:27:07 +00:00
Neil Alexander
d0e7bc0f27
Refactor restricted joins, modify join response to include membership in all cases, modify federation sender to use membership event from remote side 2021-11-15 10:02:48 +00:00
Neil Alexander
f4ad4db6fb
Fix return codes 2021-11-12 17:20:36 +00:00
Neil Alexander
fbc654d477
Remove some redundant logging 2021-11-12 13:59:57 +00:00
Neil Alexander
f0c169b2a2
Two-step unmarshalling in federation API too 2021-11-11 16:01:36 +00:00
Neil Alexander
9fb7b59648
Comments 2021-11-11 15:11:41 +00:00
Neil Alexander
3ba7291598
Ignore invalid room IDs in allow rules 2021-11-10 12:34:05 +00:00
Neil Alexander
e9bee83b03
Tweak join rule unmarshalling 2021-11-10 11:55:58 +00:00
Neil Alexander
5a270ba59f
Return 403s for now, might placate the tests 2021-11-09 16:16:04 +00:00
Neil Alexander
f3e83c1a07
More logging 2021-11-09 15:03:35 +00:00
Neil Alexander
20e96f5332
logging 2021-11-09 14:39:50 +00:00
Neil Alexander
4d20122d77
Improve error reporting 2021-11-09 13:53:30 +00:00
Neil Alexander
16b32316c2
Remove check that can't be satisied with the original auth events 2021-11-09 13:10:05 +00:00
Neil Alexander
27ca2da756
Revert "Signed event copies should be complete"
This reverts commit cef7db0469.
2021-11-09 11:41:19 +00:00
Neil Alexander
cef7db0469
Signed event copies should be complete 2021-11-09 11:27:48 +00:00
Neil Alexander
35bc2e98e6
Fix return codes 2021-11-09 11:05:44 +00:00
Neil Alexander
7d6f606d1c
Add debug logging 2021-11-09 09:36:32 +00:00
Neil Alexander
031cc48f1a
Merge branch 'master' into neilalexander/restrictedjoin 2021-11-09 09:18:38 +00:00
Neil Alexander
323a6fb54f
Resume federation sends (#2039)
* Resume federation sends

* Review comments

* Fix build error
2021-11-08 09:24:16 +00:00
Neil Alexander
9a38db199f
Try to handle federated joins for restricted rooms 2021-11-02 15:09:08 +00:00
Neil Alexander
fbd1a0ab13
Update to matrix-org/gomatrixserverlib@5e02b64 2021-11-02 10:13:38 +00:00
Ryan W
1cd4d50181
Added .well-known/matrix/server endpoint (#1988)
* Added .well-known/matrix/server endpoint

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

* Replaced tabs with spaces

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>
2021-09-10 10:05:31 +01:00
Ryan W
a624eab309
- Removed double imports (#1989)
- Lower cased error messages

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
2021-09-08 17:31:03 +01:00
kegsay
7dc8fb1fe7
Add more logs (#2005)
* Add more logs

To help debug the migration issue in #1924 along with manual data-loss-inducing fixes.
Also log the origin server on processed txns to help debug buggy server origins.

* Fix query
2021-09-07 15:07:14 +01:00
Neil Alexander
51b119107c
Don't return nonsense canonical room aliases in the public rooms responses (#1992) 2021-08-27 16:50:30 +01:00
Neil Alexander
2dd5fd1fd6
publicRooms should accept POST as well as GET (#1991) 2021-08-27 15:48:27 +01:00
Neil Alexander
ff21675c5b
Cross-signing fixes, notifications via sync, federation (#1974)
* Initial work on signing key update EDUs

* Fix build

* Produce/consume EDUs

* Producer logging

* Only produce key change notifications for local users

* Better naming

* Try to notify sync

* Enable feature

* Use key change topic

* Don't bother verifying signatures, validate key lengths if we can, notifier fixes

* Copyright notices

* Remove tests from whitelist until matrix-org/sytest#1117

* Some review comment fixes

* Update to matrix-org/gomatrixserverlib@f9416ac

* Remove unneeded parameter
2021-08-17 13:44:30 +01:00
Neil Alexander
b1377d991a
Cross-signing signature handling (#1965)
* Handle other signatures

* Decorate key ID properly

* Match by key IDs

* Tweaks

* Fixes

* Fix /user/keys/query bug, review comments, update sytest-whitelist

* Various wtweaks

* Fix wiring for keyserver in API mode

* Additional fixes
2021-08-09 14:35:24 +01:00
Neil Alexander
e95b1fd238
Cross-signing validation for self-sigs, expose signatures over /user/keys/query and /user/devices/{userId} (#1962)
* Enable unstable feature again

* Try to verify when a device signs a key

* Try to verify when a key signs a device

* It's the self-signing key, not the master key

* Fix error

* Try to verify master key uploads

* Actually we can't guarantee we can do that so nevermind

* Add signatures into /devices/list request

* Fix nil pointer

* Reprioritise map creation

* Don't skip devices that don't have signatures

* Add some debug logging

* Fix logic error in QuerySignatures

* Fix bugs

* Expose master and self-signing keys on /devices/list hopefully

* maps are tedious

* Expose signatures via /keys/query

* Upload signatures when uploading keys

* Fixes

* Disable the feature again
2021-08-06 10:13:35 +01:00
kegsay
e3df612953
Add tracing to user API (#1948)
Use the trace version in tests so we can just implement the required API functions.
2021-08-03 11:23:25 +01:00
Meenal Trivedi
fa1ec482a7
fix:Inviting to an unsupported room version return M_BAD_JSON instead of Incompatible_Version (#1930)
* fix:Inviting to an unsupported room version return M_BAD_JSON instead of M_UNSUPPORTED_ROOM_VERSION

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* fix

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* fix

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* feat: make requested changes

Signed-off-by: Meenal Trivedi <meenaltrivedi6102@gmail.com>

* Use error typecast from matrix-org/gomatrixserverlib#272

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-07-26 10:41:58 +01:00
kegsay
af64e648d7
Fix failing Complement tests (#1931)
* Check for missing state keys to avoid panicking

* Check for not allowed errors on send_leave

* More logging

* handle send_join errors too

* Additional send_join checks

* s/join/gmsl.json/
2021-07-19 13:15:19 +01:00
kegsay
728061db03
fedsender: try to satisfy all notary key requests from the cache first (#1925)
* fedsender: try to satisfy all notary key requests from the cache first

* Linting
2021-07-16 11:35:42 +01:00
kegsay
c102adaf43
fedsender: add cache tables for notary keys (#1923)
* Add notary server tables for postgres

* Add sqlite tables

* fedsender: GetServerKeys -> QueryServerKeys

As it now checks a cache and can return multiple responses
2021-07-15 17:45:37 +01:00
kegsay
7df3e691f2
Fix failing complement test (#1917)
Specifically `TestBannedUserCannotSendJoin`
2021-07-13 12:22:27 +01:00
Neil Alexander
c8408a6387
Add more optimised code path for checking if we're in a room (#1909)
* Add more optimised code path for checking if we're in a room

* Fix database queries

* Fix federation API test

* Fix logging

* Review comments

* Make separate API call for room membership
2021-07-09 16:36:45 +01:00
Neil Alexander
f2974721d5
Fix concurrent map reads/writes on t.hadEvents (#1902)
* Fix concurrent map reads/writes on t.hadEvents

* Add hadEvent function
2021-07-07 18:55:44 +01:00
Neil Alexander
bcd3ef38d0
Track expiry rate on pduCountTotal 2021-07-05 13:47:37 +01:00
Neil Alexander
99d8e1c107
Federation API fixes (#1899)
* Ensure worker has work before starting goroutine

* Revert "Remove processEventWithMissingStateMutex"

This reverts commit 7f02eab47d.

* Use request context when processing transactions

* Keep goroutine count down by not starting work for things where the caller gave up

* Remove mutex, start workers at correct time
2021-07-05 12:14:31 +01:00
Neil Alexander
7f02eab47d
Remove processEventWithMissingStateMutex 2021-07-05 09:14:24 +01:00
Neil Alexander
57320897cb
Federation API workers for /send to reduce memory usage (#1897)
* Try to process rooms concurrently in FS /send

* Clean up

* Use request context so that dead things don't linger for so long

* Remove mutex

* Free up pdus slice so only references remaining are in channel

* Revert "Remove mutex"

This reverts commit 8558075e8c.

* Process EDUs in parallel

* Try refactoring /send concurrency

* Fix waitgroup

* Release on waitgroup

* Respond to transaction

* Reduce CPU usage, fix unit tests

* Tweaks

* Move into one file
2021-07-02 12:33:27 +01:00
Neil Alexander
2647f6e9c5
Fix concurrent map read/write on haveEvents (#1893) 2021-06-30 12:32:20 +01:00
Neil Alexander
b7a2d369c0
Change how servers are selected for missing auth/prev events (#1892)
* Change how servers are selected for missing auth/prev events

* Shuffle order

* Move ServersInRoomProvider into api package
2021-06-30 12:05:58 +01:00
Neil Alexander
0e69212206
Give up on loops when the context expires (#1891) 2021-06-30 10:39:47 +01:00
Neil Alexander
3afb161352
Reduce memory usage in federation /send endpoint (#1890)
* More aggressive event caching

* Deduplicate /state results

* Deduplicate more

* Ensure we use the correct list of events when excluding repeated state

* Fixes

* Ensure we track all events we already knew about properly
2021-06-30 10:01:56 +01:00
Neil Alexander
e2b6a90d90
Put gmectx back to 5 minutes 2021-06-29 10:22:26 +01:00
Neil Alexander
f645646ca9
Restore the getServers RS query (needs optimisation) 2021-06-29 09:37:28 +01:00
Neil Alexander
4417f24678
Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage
Squashed commit of the following:

commit 7fad77c10e
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 15:06:52 2021 +0100

    Fix processEventWithMissingStateMutexes

commit 138cddcac7
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:59:44 2021 +0100

    Use internal.MutexByRoom

commit 6e6f026cfa
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:50:18 2021 +0100

    Try to slow things down per room

commit b97d406dff
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:41:27 2021 +0100

    Try to slow things down

commit 8866120ebf
Merge: 9f2de8a2 4a37b19a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:40:33 2021 +0100

    Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2

commit 4a37b19a8f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:34:54 2021 +0100

    Add comments

commit f9ab3f4b81
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:31:21 2021 +0100

    Tweaks

commit 9f2de8a29c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:15:59 2021 +0100

    Ask origin only for missing things for now

commit 8fd878c75a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 11:18:11 2021 +0100

    Make sure someone wakes up

commit b63f699f1b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 11:12:58 2021 +0100

    Use a FIFO queue instead of a channel to reduce backpressure
2021-06-28 15:11:59 +01:00
Neil Alexander
080ae6a829
Move room mutex in federation API (#1830)
* Move room mutex in federation API to surround resolveStatesAndCheck

* Guard processEventWithMissingState instead

* Revert "Guard processEventWithMissingState instead"

This reverts commit 0ce88036aa79b0ce97e967afb70fe932a547d5f0.
2021-04-13 11:13:07 +01:00
Kegsay
b769d5a25e
Optimise memory usage when calling /g_m_e (#1819)
* Optimise memory usage when calling /g_m_e

* cache more events

* refactor handling of device list update pokes

* Sigh
2021-04-08 13:50:39 +01:00