Commit graph

2308 commits

Author SHA1 Message Date
Neil Alexander
a01af55ec6
Restore the room version cache in the roomserver internal API HTTP client 2022-08-11 17:34:09 +01:00
Till
05cafbd197
Implement history visibility on /messages, /context, /sync (#2511)
* Add possibility to set history_visibility and user AccountType

* Add new DB queries

* Add actual history_visibility changes for /messages

* Add passing tests

* Extract check function

* Cleanup

* Cleanup

* Fix build on 386

* Move ApplyHistoryVisibilityFilter to internal

* Move queries to topology table

* Add filtering to /sync and /context
Some cleanup

* Add passing tests; Remove failing tests :(

* Re-add passing tests

* Move filtering to own function to avoid duplication

* Re-add passing test

* Use newly added GMSL HistoryVisibility

* Update gomatrixserverlib

* Set the visibility when creating events

* Default to shared history visibility

* Remove unused query

* Update history visibility checks to use gmsl
Update tests

* Remove unused statement

* Update migrations to set "correct" history visibility

* Add method to fetch the membership at a given event

* Tweaks and logging

* Use actual internal rsAPI, default to shared visibility in tests

* Revert "Move queries to topology table"

This reverts commit 4f0d41be9c194a46379796435ce73e79203edbd6.

* Remove noise/unneeded code

* More cleanup

* Try to optimize database requests

* Fix imports

* PR peview fixes/changes

* Move setting history visibility to own migration, be more restrictive

* Fix unit tests

* Lint

* Fix missing entries

* Tweaks for incremental syncs

* Adapt generic changes

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-11 18:23:35 +02:00
Neil Alexander
371336c6b5
Set default room version to 9 2022-08-11 16:31:44 +01:00
Neil Alexander
25ad40f5e5
Remove test from sytest-blacklist 2022-08-11 16:06:13 +01:00
Neil Alexander
c45d0936b5
Generic-based internal HTTP API (#2626)
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)

* Add `PerformInvite`

* More tweaks

* Fix metric name

* Fix LookupStateIDs

* Lots of changes to clients

* Some serverside stuff

* Some error handling

* Use paths as metric names

* Revert "Use paths as metric names"

This reverts commit a9323a6a343f5ce6461a2e5bd570fe06465f1b15.

* Namespace metric names

* Remove duplicate entry

* Remove another duplicate entry

* Tweak error handling

* Some more tweaks

* Update error behaviour

* Some more error tweaking

* Fix API path for `PerformDeleteKeys`

* Fix another path

* Tweak federation client proxying

* Fix another path

* Don't return typed nils

* Some more tweaks, not that it makes any difference

* Tweak federation client proxying

* Maybe fix the key backup test
2022-08-11 15:29:33 +01:00
Till
240ae257de
Add housekeeping function to delete old/expired EDUs (#2399)
* Add housekeeping function to delete old/expired EDUs

* Add migrations

* Evict EDUs from cache

* Fix queries

* Fix upgrade

* Use map[string]time.Duration to specify different expiry times

* Fix copy & paste mistake

* Set expires_at to tomorrow

* Don't allow NULL

* Add comment

* Add tests

* Use new testrig package

* Fix migrations

* Never expire m.direct_to_device

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-09 11:15:58 +02:00
Till
e930959e49
Send-to-device/sync tweaks (#2630)
* Always delete send to device messages

* Omit empty to_device

* Tweak /sync response to omit empty values
2022-08-09 10:40:46 +02:00
Till
03ddd98f5e
Fix issues with migrations not getting executed (#2628)
* Fix issues with migrations not getting executed

* Check actual postgres error

* Return error if it's not "column does not exist"
2022-08-08 10:18:57 +02:00
Till Faelligen
10a151cb55
Don't panic if we fail to upsert account data 2022-08-05 15:37:13 +02:00
Till
3a156a434a
Invalidate lazyLoadCache if we're doing an initial sync (#2623)
* Bypass lazyLoadCache if we're doing an initial sync

* Make the linter happy again?

* Revert "Make the linter happy again?"

This reverts commit 52a5691ba3c17c05698bcc6a13092090f27ace63.

* Try that again

* Invalidate LazyLoadCache on initial syncs

* Remove unneeded check

* Add TODO

* Rename Invalite -> InvalidateLazyLoadedUser

* Thanks IDE
2022-08-05 14:27:27 +02:00
Neil Alexander
3efc646f25
Update TURN config in sample config files 2022-08-05 13:16:54 +01:00
Till
cecd11be9a
Partly fix notification counts (#2621)
* Fix notification query

* Also for SQLite

* Move tests to whitelist

* Revert "Move tests to whitelist"

This reverts commit a7d0120019a111ce45a447ba40233d9c101e6e9b.
2022-08-05 13:44:20 +02:00
Neil Alexander
2a1df0129e
Update readme 2022-08-05 10:44:08 +01:00
Neil Alexander
c8935fb53f
Do not use ioutil as it is deprecated (#2625) 2022-08-05 10:26:59 +01:00
Till
1b7f84250a
Fix linter issues (#2624)
* Try that again

* All hail the mighty linter?

* And once again

* goimport all the things
2022-08-05 11:12:41 +02:00
Brian Meek
de78eab63a
Add race testing to tests, and fix a few small race conditions in the tests (#2587)
* Add race testing to tests, and fix a few small race conditions in the tests

* Enable run-sytest on MacOS

* Remove deadlock detecting mutex, per code review feedback

* Remove autoformatting related changes and a closure that is not needed

* Adjust to importing nats client as 'natsclient'

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Clarify the use of gooseMutex to proect goose internal state

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Remove no longer needed mutex for guarding goose

Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-05 09:19:33 +01:00
Till
9a655cb5e7
Only create a new destinationQueue if we don't have one (#2620) 2022-08-05 07:20:34 +02:00
Neil Alexander
a2bed259dd
Version 0.9.1 (#2616)
* Version 0.9.1

* Update CHANGES.md
2022-08-03 17:42:13 +01:00
Neil Alexander
3bf5ae5ffe
Try more servers when calling /state_ids (#2610)
* Try more servers when calling `/state_ids`

* More logging

* Maybe fix concurrent map write

* Revert "Maybe fix concurrent map write"

This reverts commit da0dbb836207a911afe77e6f6d63c4809669693c.

* Enforce a limit of 20s per server, 5 mins total
2022-08-03 17:37:27 +01:00
Till
9fe509b18d
Fix syncapi shared users query & device lists (#2614)
* Fix query issue, only add "changed" users if we actually share a room

* Avoid log spam if context is done

* Undo changes to filterSharedUsers

* Add logging again..

* Fix SQLite shared users query

* Change query to include invited users
2022-08-03 18:35:17 +02:00
Neil Alexander
2250768be1
Remove roominfo cache (#2615)
* Remove roominfo cache

It's the source of a number of race conditions which are seemingly causing bugs and CI failures.

* Make the linter less sad
2022-08-03 17:14:21 +01:00
Neil Alexander
bbff41b44b
Disable stack protector on Linux CI build pipelines for now (to avoid relocation target __stack_chk_fail_local not defined errors) 2022-08-03 10:50:45 +01:00
Neil Alexander
376391d1c7
Update Pinecone 2022-08-03 10:38:36 +01:00
Neil Alexander
f7f2453a85
Test Go 1.19 in CI 2022-08-03 10:35:57 +01:00
Till
ac2dbb3513
Add Cache-Control header to media endpoints (#2612)
* Add Cache-Control header

* Raise rate_limiting threshold to 20
2022-08-03 10:55:21 +02:00
Till
df5d4dc7a3
Delete correct Send-to-Device messages (#2608)
* Add send-to-device tests

* Update tests, fix message deletion

* PR comments
2022-08-02 17:00:16 +02:00
Neil Alexander
e384eb683f
Disable flakey test 2022-08-02 15:33:28 +01:00
Neil Alexander
f4345dafde
Fix data race in lookupMissingStateViaStateIDs 2022-08-02 13:01:03 +01:00
Till
7ec70272d2
Disable NATS Server logging, allow self-signed certificates (#2605)
* Disable NATS Server logs in CI

* Add option to disable TLS validation for NATS
2022-08-02 13:58:08 +02:00
Neil Alexander
ca3fa58388
Various roominfo tweaks (#2607) 2022-08-02 12:27:15 +01:00
Neil Alexander
eab87ef07d
Stronger checks for /send_join (#2604) 2022-08-02 10:22:17 +01:00
sergekh2
6b6b420b9f
Fix issue with sync API not advancing. (#2603)
Issue: During conversation, under some conditions, sync cookie is not advanced, and, as a result, client loops on the same sync API call creating high traffic and CPU load.
Fix: pdu component of cookie was updated incorrectly.
2022-08-02 09:43:48 +01:00
Neil Alexander
3d51624fef
Version 0.9.0 (#2602) 2022-08-01 17:34:29 +01:00
Neil Alexander
e94ef84aab
De-race CompleteSync (#2601)
The `err` was coming from outside of the goroutine and being written to by concurrent goroutines.
2022-08-01 15:55:56 +01:00
Neil Alexander
119cde3766
De-race types.RoomInfo (#2600) 2022-08-01 15:29:19 +01:00
Neil Alexander
05c83923e3
Optimise checking other servers allowed to see events (#2596)
* Try optimising checking if server is allowed to see event

* Fix error

* Handle case where snapshot NID is 0

* Fix query

* Update SQL

* Clean up `CheckServerAllowedToSeeEvent`

* Not supported on SQLite

* Maybe placate the unit tests

* Review comments
2022-08-01 14:11:00 +01:00
Neil Alexander
c7f7aec4d0
Set CORS headers for HTTP 404 and 405 errors (#2599)
* Set CORS headers for the 404s

* Use custom handlers, plus one for HTTP 405 too

* Tweak setup

* Add to muxes too

* Tidy up some more

* Use built-in HTTP 404 handler

* Don't bother setting it for federation-facing
2022-08-01 11:34:27 +01:00
Neil Alexander
645f31ae24
Update NATS Server and nats.go to use upstream 2022-08-01 09:51:13 +01:00
Neil Alexander
962b76da44
Make the User API responsible for sending account data output events (#2592)
* Make the User API responsible for sending account data output events

* Clean up producer

* Review comments
2022-07-25 17:30:07 +01:00
Neil Alexander
497ab4e1b7
Remove goose from Dockerfiles 2022-07-25 14:46:35 +01:00
Neil Alexander
ccea23cd40
Remove room_id field from MSC2946 stripped events (closes #2588) 2022-07-25 11:42:22 +01:00
Jonathan Bartlett
b836243a24
Add .well-known/matrix/client to clientapi (#2551)
Signed-off-by: Jonathan Bartlett <jonathan@jonnobrow.co.uk>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:57 +01:00
Till
081f5e7226
Update database migrations, remove goose (#2264)
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Neil Alexander
c7d978274d
Try to fix HTTP 500s on /members (#2581) 2022-07-22 19:43:48 +01:00
Neil Alexander
f0c8a03649
Membership updater refactoring (#2541)
* Membership updater refactoring

* Pass in membership state

* Use membership check rather than referring to state directly

* Delete irrelevant membership states

* We don't need the leave event after all

* Tweaks

* Put a log entry in that I might stand a chance of finding

* Be less panicky

* Tweak invite handling

* Don't freak if we can't find the event NID

* Use event NID from `types.Event`

* Clean up

* Better invite handling

* Placate the almighty linter

* Blacklist a Sytest which is otherwise fine under Complement for reasons I don't understand

* Fix the sytest after all (thanks @S7evinK for the spot)
2022-07-22 14:44:04 +01:00
dependabot[bot]
a201b4400d
Bump tzinfo from 1.2.9 to 1.2.10 in /docs (#2584)
Bumps [tzinfo](https://github.com/tzinfo/tzinfo) from 1.2.9 to 1.2.10.
- [Release notes](https://github.com/tzinfo/tzinfo/releases)
- [Changelog](https://github.com/tzinfo/tzinfo/blob/master/CHANGES.md)
- [Commits](https://github.com/tzinfo/tzinfo/compare/v1.2.9...v1.2.10)

---
updated-dependencies:
- dependency-name: tzinfo
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-22 10:36:10 +01:00
Matt Holt
35ce551c8f
docs: Add build page; correct proxy info; fix Caddy example (#2579)
* Add build page; correct proxy info; fix Caddy example

* Improve Caddyfile example

* Apply review comments; add polylith Caddyfile
2022-07-21 07:08:17 +02:00
Till
9507966ebd
Fix issue with membership event_nid being 0 (#2580) 2022-07-20 12:39:06 +02:00
Tulir Asokan
84a7797883
Explain how SRV works in Matrix and discourage using it (#2577)
* Explain how SRV works in Matrix and discourage using it

* Minor tweaks to formatting

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-19 15:19:03 +01:00
Neil Alexander
5c01306bb5
Add event state key cache (#2576) 2022-07-19 12:15:48 +01:00