Commit graph

230 commits

Author SHA1 Message Date
Neil Alexander
49d3c73250
Try awful hack to generate indexes using SHA512 2021-04-19 14:56:00 +01:00
Neil Alexander
4198adfab8
Print number of events 2021-04-19 14:49:18 +01:00
Neil Alexander
3bae18d9cc
Try to fix offsets 2021-04-19 14:27:38 +01:00
Neil Alexander
e665490e58
Fix output 2021-04-19 14:18:17 +01:00
Neil Alexander
2168c13d67
Fix updating rooms/events 2021-04-19 14:17:06 +01:00
Neil Alexander
1957497e69
Use pq.Int64Array for newblocks 2021-04-19 14:14:47 +01:00
Neil Alexander
c3fb1c1ba5
Tweaks 2021-04-19 14:12:04 +01:00
Neil Alexander
f975caa1b6
Try to close rows more 2021-04-19 14:09:27 +01:00
Neil Alexander
4c2d2dbe03
Use eventsarray on insert 2021-04-19 13:48:02 +01:00
Neil Alexander
42623e1f42
Convert back to pq.Int64Array 2021-04-19 13:47:03 +01:00
Neil Alexander
2d71273fdc
Don't delete after all, we don't need to as we move the table aside 2021-04-19 13:42:12 +01:00
Neil Alexander
a4e3068653
Fix scan, delete old blocks 2021-04-19 13:41:30 +01:00
Neil Alexander
2c1b6da746
Fix bug in counting state snapshot 2021-04-19 13:38:42 +01:00
Neil Alexander
7dd9228245
Try to refactor PostgreSQL migration 2021-04-19 13:02:17 +01:00
Neil Alexander
8e527b55e5
Tweaks 2021-04-16 16:44:26 +01:00
Neil Alexander
9f4ffe4c81
Use pq arrays 2021-04-16 16:41:43 +01:00
Neil Alexander
2f41a4d2bf
Bit of output 2021-04-16 16:36:35 +01:00
Neil Alexander
9048315ffe
Bit of output 2021-04-16 16:34:23 +01:00
Neil Alexander
97f2e3f94f
Wire in delta for Postgres 2021-04-16 16:27:14 +01:00
Neil Alexander
8d4d28fedc
Add experimental Postgres migration 2021-04-16 16:21:06 +01:00
Neil Alexander
30a93f5330
Add SQLite migration (up only) 2021-04-16 16:15:40 +01:00
Neil Alexander
027a957f45
Deeper checking of duplicates 2021-04-16 14:14:53 +01:00
Neil Alexander
b4d1ebf54a
Clean up a bit 2021-04-16 14:01:37 +01:00
Neil Alexander
8c2bb18697
Try that again 2021-04-16 13:32:49 +01:00
Neil Alexander
d369acf512
Fix Postgres, hopefully 2021-04-16 13:20:53 +01:00
Neil Alexander
45bcca2cb6
Try to filter event types/state keys at the database level instead of in Go 2021-04-16 12:52:46 +01:00
Neil Alexander
6900e0f495
Deduplicate state block contents 2021-04-15 15:51:15 +01:00
Neil Alexander
dbd53fa9ff
Enforce uniqueness for state snapshot to state block mappings 2021-04-15 13:28:39 +01:00
Kegsay
af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Neil Alexander
01267a34b9
Fix nil pointer crash in QueryMembershipsForRoom 2021-03-17 13:58:04 +00:00
Kegsay
3c419be6af
roomserver: don't make_join with ourselves if clients ask us to (#1797)
* roomserver: don't make_join with ourselves if clients ask us to

* delete properly
2021-03-08 18:16:28 +00:00
Will Hunt
9557ccada4
Fix appsevice alias queries part 2 (#1684)
* Check membership of room

* Use QueryStateAfterEventsResponse

* Fix complexity

* Add field ShouldHitAppservice to GetRoomIDForAlias

* Hit appservice when trying to join a non-existent alias

* remove unused

* Changes that I made a long time ago

* Rename to appserviceJoinedAtEvent

* Check membership in GetMemberships

* Update QueryMembershipsForRoom

* Tweaks in client API

* Update appserviceJoinedAtEvent

* Comments

* Try QueryMembershipForUser instead

* Undo some changes to client API that shouldn't be needed

* More /event tweaks

* Refactor /event bit

* Go back to QueryMembershipsForRoom because appservices are hard

* Fix bugs in onMessage

* Add comments

* More logical naming, clean up a bit

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-03-03 17:00:31 +00:00
Will Hunt
a2773922d2
Send events to appservice based on room membership (#1680)
* Check membership of room

* Use QueryStateAfterEventsResponse

* Fix complexity

* Changes that I made a long time ago

* Rename to appserviceJoinedAtEvent

* Check membership in GetMemberships

* Update QueryMembershipsForRoom

* Tweaks in client API

* Update appserviceJoinedAtEvent

* Comments

* Try QueryMembershipForUser instead

* Undo some changes to client API that shouldn't be needed

* More /event tweaks

* Refactor /event bit

* Go back to QueryMembershipsForRoom because appservices are hard

* Fix bugs in onMessage

* Add comments

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-03-03 16:27:44 +00:00
Neil Alexander
d15836e260
Increase gocyclo complexity to 25 (and remove all but 2 golint directives related to it) (#1783) 2021-03-03 14:35:57 +00:00
Neil Alexander
5d74a1757f
Don't query for servers so often in /send (#1766)
* Look up servers less often, don't hit API for missing auth events unless there are actually missing auth events

* Remove ResolveConflictsAdhoc (since it is already in GMSL), other tweaks

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#254

* Fix resolve-state

* Initialise t.servers on first use
2021-02-16 17:12:17 +00:00
Neil Alexander
02e6d89cc2
Fix crash in membership updater (#1753)
* Fix nil pointer exception in membership updater

* goimports
2021-02-06 11:49:18 +00:00
Neil Alexander
de5f22a469
Remove redundant check (#1748) 2021-02-04 11:12:52 +00:00
Kegsay
6d1c6f29e0
Add m.room.create to invite stripped state (#1740)
MSC1772 needs this because the create event contains info on if
the room is a space or not. The create event itself isn't sensitive
so other people may find this useful too.
2021-01-29 11:36:26 +00:00
Neil Alexander
64fb6de6d4
Don't retrieve same state events over and over again (#1737) 2021-01-26 09:12:17 +00:00
Matthew Hodgson
0571d395b5
Peeking over federation via MSC2444 (#1391)
* a very very WIP first cut of peeking via MSC2753.

doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.

* make PeekingDeviceSet private

* add server_name param

* blind stab at adding a `peek` section to /sync

* make it build

* make it launch

* add peeking to getResponseWithPDUsForCompleteSync

* cancel any peeks when we join a room

* spell out how to runoutside of docker if you want speed

* fix SQL

* remove unnecessary txn for SelectPeeks

* fix s/join/peek/ cargocult fail

* HACK: Track goroutine IDs to determine when we write by the wrong thread

To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`

* Track partition offsets and only log unsafe for non-selects

* Put redactions in the writer goroutine

* Update filters on writer goroutine

* wrap peek storage in goid hack

* use exclusive writer, and MarkPeeksAsOld more efficiently

* don't log ascii in binary at sql trace...

* strip out empty roomd deltas

* re-add txn to SelectPeeks

* re-add accidentally deleted field

* reject peeks for non-worldreadable rooms

* move perform_peek

* fix package

* correctly refactor perform_peek

* WIP of implementing MSC2444

* typo

* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"

This reverts commit 3cebd8dbfb, reversing
changes made to ed4b3a58a7.

* (almost) make it build

* clean up bad merge

* support SendEventWithState with optional event

* fix build & lint

* fix build & lint

* reinstate federated peeks in the roomserver (doh)

* fix sql thinko

* todo for authenticating state returned by /peek

* support returning current state from QueryStateAndAuthChain

* handle SS /peek

* reimplement SS /peek to prod the RS to tell the FS about the peek

* rename RemotePeeks as OutboundPeeks

* rename remote_peeks_table as outbound_peeks_table

* add perform_handle_remote_peek.go

* flesh out federation doc

* add inbound peeks table and hook it up

* rename ambiguous RemotePeek as InboundPeek

* rename FSAPI's PerformPeek as PerformOutboundPeek

* setup inbound peeks db correctly

* fix api.SendEventWithState with no event

* track latestevent on /peek

* go fmt

* document the peek send stream race better

* fix SendEventWithRewrite not to bail if handed a non-state event

* add fixme

* switch SS /peek to use SendEventWithRewrite

* fix comment

* use reverse topo ordering to find latest extrem

* support postgres for federated peeking

* go fmt

* back out bogus go.mod change

* Fix performOutboundPeekUsingServer

* Fix getAuthChain -> GetAuthChain

* Fix build issues

* Fix build again

* Fix getAuthChain -> GetAuthChain

* Don't repeat outbound peeks for the same room ID to the same servers

* Fix lint

* Don't omitempty to appease sytest

Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-22 14:55:08 +00:00
Neil Alexander
244ff0dccb
Don't create so many state snapshots when updating forward extremities (#1718)
* Light-weight checking of state changes when updating forward extremities

* Only do this for non-state events, since state events will always result in state change at extremities
2021-01-18 13:21:33 +00:00
Neil Alexander
3ac693c7a5
Add dendrite_roomserver_processroomevent_duration_millis to prometheus
Squashed commit of the following:

commit e5e2d793119733ecbcf9b85f966e018ab0318741
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jan 13 17:28:12 2021 +0000

    Add dendrite_roomserver_processroomevent_duration_millis to prometheus
2021-01-13 17:31:46 +00:00
Neil Alexander
e1e34b8994
Deep-checking forward extremities (#1698) 2021-01-11 12:47:25 +00:00
Neil Alexander
fac71edc62
Fix #1655 by re-adding the appservice alias query (#1660) 2020-12-18 13:33:28 +00:00
Neil Alexander
b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander
9057143033
Hit the database far less in Events to find room NIDs and room versions (#1643)
* Hit the database far less to find room NIDs for event NIDs

* Close the rows

* Fix SQLite selectRoomNIDsForEventNIDsSQL

* Give same treatment to room version lookups
2020-12-16 10:33:28 +00:00
Neil Alexander
2885eb0422
Don't use request context for input room event queued tasks (#1640) 2020-12-14 14:40:57 +00:00
Neil Alexander
f5869daaab
Don't start more goroutines than needed on RS input, increase input worker buffer size (#1638) 2020-12-14 10:42:21 +00:00
Neil Alexander
d9b3035342
Adjust latest events updater (#1623)
* Adjust forward elatest events updater

* Populate newLatest in all cases

* Re-add existingPrevs loop
2020-12-09 13:34:37 +00:00
Kegsay
b507312d4c
MSC2836 threading: part 2 (#1596)
* Update GMSL

* Add MSC2836EventRelationships to fedsender

* Call MSC2836EventRelationships in reqCtx

* auth remote servers

* Extract room ID and servers from previous events; refactor a bit

* initial cut of federated threading

* Use the right client/fed struct in the response

* Add QueryAuthChain for use with MSC2836

* Add auth chain to federated response

* Fix pointers

* under CI: more logging and enable mscs, nil fix

* Handle direction: up

* Actually send message events to the roomserver..

* Add children and children_hash to unsigned, with tests

* Add logic for exploring threads and tracking children; missing storage functions

* Implement storage functions for children

* Add fetchUnknownEvent

* Do federated hits for include_children if we have unexplored children

* Use /ev_rel rather than /event as the former includes child metadata

* Remove cross-room threading impl

* Enable MSC2836 in the p2p demo

* Namespace mscs db

* Enable msc2836 for ygg

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-04 14:11:01 +00:00