Commit graph

2540 commits

Author SHA1 Message Date
Till
33129c02f7
Add timeout parameter & trim URL (#2666)
A timeout of 10 seconds could cause issues with servers having a high `bcrypt_cost` configured in the config.
This adds a parameter to manually configure the timeout, defaults to 30 seconds.
2022-08-23 11:10:41 +02:00
Neil Alexander
2668050e53
Tweak soft-failure handling in roomserver
commit 1929b688e31987c46e0c8a546f0f9cb0a46bf9a3
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:09:44 2022 +0100

    Still process state-before for soft-failed events

commit e83c0b701d40d78b92072c4643f6bc6f71b72800
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 10:06:50 2022 +0100

    Improve logging

commit 29e26124bc27cb83d449de2a4214b253c594aa93
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Aug 22 09:58:13 2022 +0100

    Don't store soft-failed events as rejected
2022-08-22 10:34:07 +01:00
Till
9dc57122d9
Fetch more data for newly joined rooms in an incremental sync (#2657)
If we've joined a new room in an incremental sync, try fetching more data.
This deflakes the complement server notices test (at least in my tests).
2022-08-19 15:32:24 +02:00
Neil Alexander
56b55a28f5
Version 0.9.4 (#2658)
### Fixes

* A bug in the roomserver around handling rejected outliers has been fixed
* Backfilled events will now use the correct history visibility where possible
* The device list updater backoff has been fixed, which should reduce the number of outbound HTTP requests and `Failed to query device keys for some users` log entries for dead servers
* The `/sync` endpoint will no longer incorrectly return room entries for retired invites which could cause some rooms to show up in the client "Historical" section
* The `/createRoom` endpoint will now correctly populate `is_direct` in invite membership events, which may help clients to classify direct messages correctly
* The `create-account` tool will now log an error if the shared secret is not set in the Dendrite config
* A couple of minor bugs have been fixed in the membership lazy-loading
* Queued EDUs in the federation API are now cached properly
2022-08-19 12:46:14 +01:00
Neil Alexander
5513f182cc
Enforce device list backoffs (#2653)
This ensures that if the device list updater is already backing off a node, we don't try to call processServer again anyway for server just because the server name arrived in the channel. Otherwise we can keep trying to hit a remote server that is offline or not behaving every second and that spams the logs too.
2022-08-19 10:23:09 +01:00
Till
365da70a23
Set historyVisibility for backfilled events over federation (#2656)
This should hopefully deflake Backfill works correctly with history visibility set to joined as we were using the default shared visibility, even if the events are set to joined (or something else)
2022-08-19 11:04:26 +02:00
Till
5cacca92d2
Make SyncAPI unit tests more reliable (#2655)
This should hopefully make some SyncAPI tests more reliable
2022-08-19 11:03:55 +02:00
Winter
a379d3e814
De-race TestExpireEDUs (#2654)
In some conditions (fast CPUs), this test would race the clock for EDU expiration when all we want to make sure of is that the expired EDUs are properly deleted. Given this, we set the expiry time to 0 so the specified EDUs are always deleted when DeleteExpiredEDUs is called.
Fixes #2650.

Signed-off-by: Winter <winter@winter.cafe>
2022-08-19 07:28:33 +02:00
Neil Alexander
6b48ce0d75
State handling tweaks (#2652)
This tweaks how rejected events are handled in room state and also to not apply checks we can't complete to outliers.
2022-08-18 17:06:13 +01:00
Neil Alexander
606cb67506
Enable workflow_dispatch in GHA 2022-08-18 13:50:58 +01:00
Till
7484689ad1
Actually store EDUs once we retrieved from the database (#2651)
We now actually cache the EDUs once we got them from the database and ensures we only evict them if we successfully deleted them.
2022-08-18 11:14:42 +01:00
Neil Alexander
59bc0a6f4e
Reprocess rejected input events (#2647)
* Reprocess outliers that were previously rejected

* Might as well do all events this way

* More useful errors

* Fix queries

* Tweak condition

* Don't wrap errors

* Report more useful error

* Flatten error on `r.Queryer.QueryStateAfterEvents`

* Some more debug logging

* Flatten error in `QueryRestrictedJoinAllowed`

* Revert "Flatten error in `QueryRestrictedJoinAllowed`"

This reverts commit 1238b4184c30e0c31ffb0f364806fa1275aba483.

* Tweak `QueryStateAfterEvents`

* Handle MissingStateError too

* Scope to room

* Clean up

* Fix the error

* Only apply rejection check to outliers
2022-08-18 10:37:47 +01:00
Till Faelligen
8d9c8f11c5
Add a delay after sending events to the roomserver 2022-08-18 08:56:57 +02:00
Neil Alexander
ad4ac2c016
Stop spamming the logs with StateBetween: ignoring deleted state event IDs 2022-08-16 14:42:35 +01:00
Neil Alexander
ec16c944eb
Lazy-loading fixes (#2646)
* Use existing current room state if we have it

* Don't dedupe before applying the history vis filter

* Revert "Don't dedupe before applying the history vis filter"

This reverts commit d27c4a0874dabb77c2eda6b23eb7c00478bc9e90.

* Revert "Use existing current room state if we have it"

This reverts commit 5819b4a7ce511204c4fb48d3c4741612b136e2ea.

* Tweaks
2022-08-16 14:42:00 +01:00
Till
804653e551
Verify a shared secret is set in create-account (#2645) 2022-08-16 13:21:22 +02:00
Neil Alexander
5424b88f30
Use is_direct flag from /createRoom, update stripped state (#2644)
* Use `is_direct` flag from `/createRoom`, update stripped state

* Add comment
2022-08-16 11:55:06 +01:00
Till
0642ffc0f6
Only return non-retired invites (#2643)
* Only return non-retired invites

* Revert "Only return non-retired invites"

This reverts commit 1150aa7f385b7d7cf5378297f3e17566d5aabcc6.

* Check if we're doing an initial sync in the stream
2022-08-16 10:29:36 +02:00
Neil Alexander
bcdbd5c00a
Version 0.9.3 2022-08-15 10:56:54 +01:00
Till Faelligen
65b7db633c
Fix FAQ question 2022-08-13 20:31:43 +02:00
timaeos
da7d209b10
Admin endpoint documentation (#2640)
* Adding documentation for administration end points that aren't currently documented

* Additional information on using the administration API

* Fix inaccurate information on server notice documentation

* Fix typo in the curl command

* Add details about using the whois endpoint

* fix formatting in documentation for admin whois endpoint

* Fix more documentation formatting

* Additional formatting updates to match other formats

* Add a link to the server notice information in the Matrix Spec

* Fix broken link on server notices

* Adding access token information and admin info

* adding a warning regarding access_token

Co-authored-by: Timothy Arnold <Yourmove@gmail.com>
2022-08-13 20:25:39 +02:00
timaeos
30dbc5ac44
Adding FAQ Questions with common questions (#2641)
* Adding FAQ Questions with common questions

* changing order of the FAQ to reflect suggestions
2022-08-13 20:24:52 +02:00
Neil Alexander
ef7262a7a2
Version 0.9.2 (#2638) 2022-08-12 15:18:31 +01:00
texuf
e55cd6ea78
/hierarchy - return public and knockable rooms for authed users (#2578)
When requesting the room hierarchy with an authenticated user, return public and knockable rooms.

According to the spec, https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2946-spaces-summary.md

```
Any child room that the user is joined or is potentially joinable is included in the response.
```

This is currently not the case. See discussion here: https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$t2Csj-6y1PVsn8GOnFZfXzeQW13NfqvrFCxB-XI_uhA?via=matrix.org&via=libera.chat&via=element.io and here: https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$EHp1x1DY7tnYZtx_PVEb-sKB9lmJajqHx2uGlhrRh6k?via=matrix.org&via=libera.chat&via=element.io

Test Plan:
create and register clients bob and alice
have bob create a public space
have bob create a public room parented to the space
have alice join the space(room)
have alice sync the space
expect alice to see two rooms in the space hierarchy, the space and the child room

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 14:07:45 +01:00
Neil Alexander
6d0d7a0bc3
Update example in readme 2022-08-12 13:34:05 +01:00
Neil Alexander
9fa30f5d3c
Update NATS Server and nats.go libraries 2022-08-12 13:29:09 +01:00
Till Faelligen
20bf00b743
Update README 2022-08-12 13:55:45 +02:00
Neil Alexander
f1e9108db8
Update create-account help text 2022-08-12 12:53:29 +01:00
Till
b4647fbb7e
Show/hide users in user directory (#2637)
* CS API changes

* Query remote profiles

* Add passing tests

* Don't create a new FullyQualifiedProfile

* Handle sql.ErrNoRows

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 13:33:31 +02:00
Till
48600d5540
Use /admin/v1/register in create-account (#2484)
* Get all account data on CompleteSync

* Revert "Get all account data on CompleteSync"

This reverts commit 44a3e566d8fb940b0b757aea9b8408fa19ea9f54.

* Use /_synapse/admin/v1/register to create account

* Linting

* Linter again :)

* Update docs

* Use HTTP API to reset password, add option to User API `PerformPasswordUpdate` to invalidate sessions

* Fix routing name

* Tell me more about what went wrong

* Deprecate the `-reset-password` flag, document the new API

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 12:00:07 +01:00
Tak Wai Wong
fad3ac8e78
Protect user_interactive reads and writes with locks (#2635)
* Protect user_interactive reads and writes with locks

* Ignore golangci-lint false positive

* fix lint

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-08-12 09:12:05 +01:00
Neil Alexander
2b352915a1
Update golangci-lint component in GHA workflow 2022-08-11 17:40:57 +01:00
Neil Alexander
a01af55ec6
Restore the room version cache in the roomserver internal API HTTP client 2022-08-11 17:34:09 +01:00
Till
05cafbd197
Implement history visibility on /messages, /context, /sync (#2511)
* Add possibility to set history_visibility and user AccountType

* Add new DB queries

* Add actual history_visibility changes for /messages

* Add passing tests

* Extract check function

* Cleanup

* Cleanup

* Fix build on 386

* Move ApplyHistoryVisibilityFilter to internal

* Move queries to topology table

* Add filtering to /sync and /context
Some cleanup

* Add passing tests; Remove failing tests :(

* Re-add passing tests

* Move filtering to own function to avoid duplication

* Re-add passing test

* Use newly added GMSL HistoryVisibility

* Update gomatrixserverlib

* Set the visibility when creating events

* Default to shared history visibility

* Remove unused query

* Update history visibility checks to use gmsl
Update tests

* Remove unused statement

* Update migrations to set "correct" history visibility

* Add method to fetch the membership at a given event

* Tweaks and logging

* Use actual internal rsAPI, default to shared visibility in tests

* Revert "Move queries to topology table"

This reverts commit 4f0d41be9c194a46379796435ce73e79203edbd6.

* Remove noise/unneeded code

* More cleanup

* Try to optimize database requests

* Fix imports

* PR peview fixes/changes

* Move setting history visibility to own migration, be more restrictive

* Fix unit tests

* Lint

* Fix missing entries

* Tweaks for incremental syncs

* Adapt generic changes

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-11 18:23:35 +02:00
Neil Alexander
371336c6b5
Set default room version to 9 2022-08-11 16:31:44 +01:00
Neil Alexander
25ad40f5e5
Remove test from sytest-blacklist 2022-08-11 16:06:13 +01:00
Neil Alexander
c45d0936b5
Generic-based internal HTTP API (#2626)
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)

* Add `PerformInvite`

* More tweaks

* Fix metric name

* Fix LookupStateIDs

* Lots of changes to clients

* Some serverside stuff

* Some error handling

* Use paths as metric names

* Revert "Use paths as metric names"

This reverts commit a9323a6a343f5ce6461a2e5bd570fe06465f1b15.

* Namespace metric names

* Remove duplicate entry

* Remove another duplicate entry

* Tweak error handling

* Some more tweaks

* Update error behaviour

* Some more error tweaking

* Fix API path for `PerformDeleteKeys`

* Fix another path

* Tweak federation client proxying

* Fix another path

* Don't return typed nils

* Some more tweaks, not that it makes any difference

* Tweak federation client proxying

* Maybe fix the key backup test
2022-08-11 15:29:33 +01:00
Till
240ae257de
Add housekeeping function to delete old/expired EDUs (#2399)
* Add housekeeping function to delete old/expired EDUs

* Add migrations

* Evict EDUs from cache

* Fix queries

* Fix upgrade

* Use map[string]time.Duration to specify different expiry times

* Fix copy & paste mistake

* Set expires_at to tomorrow

* Don't allow NULL

* Add comment

* Add tests

* Use new testrig package

* Fix migrations

* Never expire m.direct_to_device

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-09 11:15:58 +02:00
Till
e930959e49
Send-to-device/sync tweaks (#2630)
* Always delete send to device messages

* Omit empty to_device

* Tweak /sync response to omit empty values
2022-08-09 10:40:46 +02:00
Till
03ddd98f5e
Fix issues with migrations not getting executed (#2628)
* Fix issues with migrations not getting executed

* Check actual postgres error

* Return error if it's not "column does not exist"
2022-08-08 10:18:57 +02:00
Till Faelligen
10a151cb55
Don't panic if we fail to upsert account data 2022-08-05 15:37:13 +02:00
Till
3a156a434a
Invalidate lazyLoadCache if we're doing an initial sync (#2623)
* Bypass lazyLoadCache if we're doing an initial sync

* Make the linter happy again?

* Revert "Make the linter happy again?"

This reverts commit 52a5691ba3c17c05698bcc6a13092090f27ace63.

* Try that again

* Invalidate LazyLoadCache on initial syncs

* Remove unneeded check

* Add TODO

* Rename Invalite -> InvalidateLazyLoadedUser

* Thanks IDE
2022-08-05 14:27:27 +02:00
Neil Alexander
3efc646f25
Update TURN config in sample config files 2022-08-05 13:16:54 +01:00
Till
cecd11be9a
Partly fix notification counts (#2621)
* Fix notification query

* Also for SQLite

* Move tests to whitelist

* Revert "Move tests to whitelist"

This reverts commit a7d0120019a111ce45a447ba40233d9c101e6e9b.
2022-08-05 13:44:20 +02:00
Neil Alexander
2a1df0129e
Update readme 2022-08-05 10:44:08 +01:00
Neil Alexander
c8935fb53f
Do not use ioutil as it is deprecated (#2625) 2022-08-05 10:26:59 +01:00
Till
1b7f84250a
Fix linter issues (#2624)
* Try that again

* All hail the mighty linter?

* And once again

* goimport all the things
2022-08-05 11:12:41 +02:00
Brian Meek
de78eab63a
Add race testing to tests, and fix a few small race conditions in the tests (#2587)
* Add race testing to tests, and fix a few small race conditions in the tests

* Enable run-sytest on MacOS

* Remove deadlock detecting mutex, per code review feedback

* Remove autoformatting related changes and a closure that is not needed

* Adjust to importing nats client as 'natsclient'

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Clarify the use of gooseMutex to proect goose internal state

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Remove no longer needed mutex for guarding goose

Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-05 09:19:33 +01:00
Till
9a655cb5e7
Only create a new destinationQueue if we don't have one (#2620) 2022-08-05 07:20:34 +02:00
Neil Alexander
a2bed259dd
Version 0.9.1 (#2616)
* Version 0.9.1

* Update CHANGES.md
2022-08-03 17:42:13 +01:00