Commit graph

1487 commits

Author SHA1 Message Date
Neil Alexander
1957497e69
Use pq.Int64Array for newblocks 2021-04-19 14:14:47 +01:00
Neil Alexander
c3fb1c1ba5
Tweaks 2021-04-19 14:12:04 +01:00
Neil Alexander
f975caa1b6
Try to close rows more 2021-04-19 14:09:27 +01:00
Neil Alexander
4c2d2dbe03
Use eventsarray on insert 2021-04-19 13:48:02 +01:00
Neil Alexander
42623e1f42
Convert back to pq.Int64Array 2021-04-19 13:47:03 +01:00
Neil Alexander
2d71273fdc
Don't delete after all, we don't need to as we move the table aside 2021-04-19 13:42:12 +01:00
Neil Alexander
a4e3068653
Fix scan, delete old blocks 2021-04-19 13:41:30 +01:00
Neil Alexander
2c1b6da746
Fix bug in counting state snapshot 2021-04-19 13:38:42 +01:00
Neil Alexander
7dd9228245
Try to refactor PostgreSQL migration 2021-04-19 13:02:17 +01:00
Neil Alexander
8e527b55e5
Tweaks 2021-04-16 16:44:26 +01:00
Neil Alexander
9f4ffe4c81
Use pq arrays 2021-04-16 16:41:43 +01:00
Neil Alexander
2f41a4d2bf
Bit of output 2021-04-16 16:36:35 +01:00
Neil Alexander
9048315ffe
Bit of output 2021-04-16 16:34:23 +01:00
Neil Alexander
97f2e3f94f
Wire in delta for Postgres 2021-04-16 16:27:14 +01:00
Neil Alexander
8d4d28fedc
Add experimental Postgres migration 2021-04-16 16:21:06 +01:00
Neil Alexander
30a93f5330
Add SQLite migration (up only) 2021-04-16 16:15:40 +01:00
Neil Alexander
027a957f45
Deeper checking of duplicates 2021-04-16 14:14:53 +01:00
Neil Alexander
b4d1ebf54a
Clean up a bit 2021-04-16 14:01:37 +01:00
Neil Alexander
8c2bb18697
Try that again 2021-04-16 13:32:49 +01:00
Neil Alexander
d369acf512
Fix Postgres, hopefully 2021-04-16 13:20:53 +01:00
Neil Alexander
45bcca2cb6
Try to filter event types/state keys at the database level instead of in Go 2021-04-16 12:52:46 +01:00
Neil Alexander
6900e0f495
Deduplicate state block contents 2021-04-15 15:51:15 +01:00
Neil Alexander
dbd53fa9ff
Enforce uniqueness for state snapshot to state block mappings 2021-04-15 13:28:39 +01:00
Neil Alexander
4a90bc86dd
Update go-sqlite3 2021-04-15 13:06:42 +01:00
Neil Alexander
a9faa1bc44
Fix registration error when disabled 2021-04-15 09:58:26 +01:00
Kegsay
656d11ec90
fedsender: tolerate dupe membership events (#1824)
* fedsender: tolerate dupe membership events

Previously if the fedsender got a duplicate membership event it would cause
the entire process to crash. Now it doesn't. This masks an issue with the
roomserver where it can emit duplicate membership events.

* Update joined_hosts_table.go
2021-04-14 11:11:54 +01:00
Fero
653e30619c
Remove the Content-Length requirement on upload request (#1831)
* Remove the Content-Length requirement

* Make sure that the file size does not exceed max content size

* Address review comment - universally check if temp file size exceeds max file size
2021-04-14 10:53:24 +01:00
Neil Alexander
080ae6a829
Move room mutex in federation API (#1830)
* Move room mutex in federation API to surround resolveStatesAndCheck

* Guard processEventWithMissingState instead

* Revert "Guard processEventWithMissingState instead"

This reverts commit 0ce88036aa79b0ce97e967afb70fe932a547d5f0.
2021-04-13 11:13:07 +01:00
Kegsay
e08942fb00
Remove legacy register endpoint (#1822)
* Remove legacy register endpoint

We only support `/r0` CS API paths, not `/v1`.

* Finish removing
2021-04-09 10:21:35 +01:00
Kegsay
b769d5a25e
Optimise memory usage when calling /g_m_e (#1819)
* Optimise memory usage when calling /g_m_e

* cache more events

* refactor handling of device list update pokes

* Sigh
2021-04-08 13:50:39 +01:00
Tim McCormack
5ade348d14
Document need for max-body-size change in reverse proxy as well (#1816)
Just changing the Media API's `max_file_size_bytes` isn't enough if
Dendrite is running behind a proxy; document the need for a proxy config
change in the place the admin is most likely to notice it.

Signed-off-by: Tim McCormack <cortex@brainonfire.net>
2021-04-08 12:08:38 +01:00
Bruce MacDonald
d27607af78
Implement OpenID module (#599) (#1812)
* Implement OpenID module (#599)

- Unrelated: change Riot references to Element in client API routing

Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>

* OpenID module tweaks (#599)

- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex

* Add default OpenID token lifetime (#599)

* Update dendrite-config.yaml

Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2021-04-07 13:26:20 +01:00
Kegsay
f8d3a762c4
Add a per-room mutex to federationapi when processing transactions (#1810)
* Add a per-room mutex to federationapi when processing transactions

This has numerous benefits:
 - Prevents us doing lots of state resolutions in busy rooms. Previously, room forks would always result
   in a state resolution being performed immediately, without checking if we were already doing this in
   a different transaction. Now they will queue up, resulting in fewer calls to `/state_ids`, `/g_m_e`, etc.
 - Prevents memory usage from growing too large as a result and potentially OOMing.

And costs:
 - High traffic rooms will be slightly slower due to head-of-line blocking from other servers,
   though this has always been an issue as roomserver has a per-room mutex already.

* Fix unit tests

* Correct mutex lock ordering
2021-03-30 10:01:32 +01:00
Eric Eastwood
0ee1c56ffd
Use log directory that we will be able to write to (#1799)
Fix https://github.com/matrix-org/dendrite/issues/1644
2021-03-30 09:53:02 +01:00
Kegsay
af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Kegsay
802f1c96f8
Add more metrics (#1802)
* Add more metrics

* Linting
2021-03-23 15:22:00 +00:00
Kegsay
a1b7e4ef3f
log less for failed key querys, add counters for incoming pdus/edus (#1801)
* log less for failed key querys, add counters for incoming pdus/edus

* use labels

* Blacklist flakey test

* Fix metrics
2021-03-23 11:33:36 +00:00
Neil Alexander
01267a34b9
Fix nil pointer crash in QueryMembershipsForRoom 2021-03-17 13:58:04 +00:00
Kegsay
3c419be6af
roomserver: don't make_join with ourselves if clients ask us to (#1797)
* roomserver: don't make_join with ourselves if clients ask us to

* delete properly
2021-03-08 18:16:28 +00:00
Kegsay
77fb981da5
device lists: backoff for longer if the wrong error type is returned (#1796) 2021-03-08 17:45:20 +00:00
Kegan Dougal
e865a1507a Make sure the component max open conns doesn't exceed 100 2021-03-08 14:50:37 +00:00
Neil Alexander
5912429d53
Return a more useful error on /register spec compliance violation (#1792) 2021-03-08 13:57:15 +00:00
Neil Alexander
5acf30cd3c
Update sytest-whitelist 2021-03-08 13:32:21 +00:00
Kegsay
850abb1dde
Make bcrypt cost configurable (#1793) 2021-03-08 13:19:02 +00:00
Neil Alexander
c3ad2cca49
Fix database default connection limits for CI (#1794) 2021-03-08 13:18:29 +00:00
Neil Alexander
6aa262ead8
Use default transport for AS traffic (#1789)
* Use default transport for AS traffic

* Update gmsl and use default client

* Remove replace

* Fix go.sum

* Update gomatrixserverlib

* Go back to appservices managing their own HTTP clients because argh

* Add missing context
2021-03-05 16:40:32 +00:00
Will Hunt
fe021d3742
Treat the sender_localpart as an exclusive namespace of one user (#1790) 2021-03-05 14:57:42 +00:00
Neil Alexander
1ad96e2e2d
Tweak AS registration check and AS component HTTP clients (#1785)
* Tweak AS registration check

* Check appservice usernames using correct function

* Update sytest-whitelist

* Use gomatrixserverlib.Client since that allows us to disable TLS validation using the config

* Add appservice-specific client and ability to control TLS validation for appservices only

* Set timeout on appservice client

* Review comments

* Remove dead code

* Enforce LoginTypeApplicationService after all

* Check correct auth type field
2021-03-05 10:40:27 +00:00
Will Hunt
9557ccada4
Fix appsevice alias queries part 2 (#1684)
* Check membership of room

* Use QueryStateAfterEventsResponse

* Fix complexity

* Add field ShouldHitAppservice to GetRoomIDForAlias

* Hit appservice when trying to join a non-existent alias

* remove unused

* Changes that I made a long time ago

* Rename to appserviceJoinedAtEvent

* Check membership in GetMemberships

* Update QueryMembershipsForRoom

* Tweaks in client API

* Update appserviceJoinedAtEvent

* Comments

* Try QueryMembershipForUser instead

* Undo some changes to client API that shouldn't be needed

* More /event tweaks

* Refactor /event bit

* Go back to QueryMembershipsForRoom because appservices are hard

* Fix bugs in onMessage

* Add comments

* More logical naming, clean up a bit

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-03-03 17:00:31 +00:00
Will Hunt
a2773922d2
Send events to appservice based on room membership (#1680)
* Check membership of room

* Use QueryStateAfterEventsResponse

* Fix complexity

* Changes that I made a long time ago

* Rename to appserviceJoinedAtEvent

* Check membership in GetMemberships

* Update QueryMembershipsForRoom

* Tweaks in client API

* Update appserviceJoinedAtEvent

* Comments

* Try QueryMembershipForUser instead

* Undo some changes to client API that shouldn't be needed

* More /event tweaks

* Refactor /event bit

* Go back to QueryMembershipsForRoom because appservices are hard

* Fix bugs in onMessage

* Add comments

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-03-03 16:27:44 +00:00