Commit graph

115 commits

Author SHA1 Message Date
Neil Alexander
49d3c73250
Try awful hack to generate indexes using SHA512 2021-04-19 14:56:00 +01:00
Neil Alexander
4198adfab8
Print number of events 2021-04-19 14:49:18 +01:00
Neil Alexander
3bae18d9cc
Try to fix offsets 2021-04-19 14:27:38 +01:00
Neil Alexander
e665490e58
Fix output 2021-04-19 14:18:17 +01:00
Neil Alexander
2168c13d67
Fix updating rooms/events 2021-04-19 14:17:06 +01:00
Neil Alexander
1957497e69
Use pq.Int64Array for newblocks 2021-04-19 14:14:47 +01:00
Neil Alexander
c3fb1c1ba5
Tweaks 2021-04-19 14:12:04 +01:00
Neil Alexander
f975caa1b6
Try to close rows more 2021-04-19 14:09:27 +01:00
Neil Alexander
4c2d2dbe03
Use eventsarray on insert 2021-04-19 13:48:02 +01:00
Neil Alexander
42623e1f42
Convert back to pq.Int64Array 2021-04-19 13:47:03 +01:00
Neil Alexander
2d71273fdc
Don't delete after all, we don't need to as we move the table aside 2021-04-19 13:42:12 +01:00
Neil Alexander
a4e3068653
Fix scan, delete old blocks 2021-04-19 13:41:30 +01:00
Neil Alexander
2c1b6da746
Fix bug in counting state snapshot 2021-04-19 13:38:42 +01:00
Neil Alexander
7dd9228245
Try to refactor PostgreSQL migration 2021-04-19 13:02:17 +01:00
Neil Alexander
8e527b55e5
Tweaks 2021-04-16 16:44:26 +01:00
Neil Alexander
9f4ffe4c81
Use pq arrays 2021-04-16 16:41:43 +01:00
Neil Alexander
2f41a4d2bf
Bit of output 2021-04-16 16:36:35 +01:00
Neil Alexander
9048315ffe
Bit of output 2021-04-16 16:34:23 +01:00
Neil Alexander
97f2e3f94f
Wire in delta for Postgres 2021-04-16 16:27:14 +01:00
Neil Alexander
8d4d28fedc
Add experimental Postgres migration 2021-04-16 16:21:06 +01:00
Neil Alexander
30a93f5330
Add SQLite migration (up only) 2021-04-16 16:15:40 +01:00
Neil Alexander
027a957f45
Deeper checking of duplicates 2021-04-16 14:14:53 +01:00
Neil Alexander
b4d1ebf54a
Clean up a bit 2021-04-16 14:01:37 +01:00
Neil Alexander
8c2bb18697
Try that again 2021-04-16 13:32:49 +01:00
Neil Alexander
d369acf512
Fix Postgres, hopefully 2021-04-16 13:20:53 +01:00
Neil Alexander
45bcca2cb6
Try to filter event types/state keys at the database level instead of in Go 2021-04-16 12:52:46 +01:00
Neil Alexander
6900e0f495
Deduplicate state block contents 2021-04-15 15:51:15 +01:00
Neil Alexander
dbd53fa9ff
Enforce uniqueness for state snapshot to state block mappings 2021-04-15 13:28:39 +01:00
Neil Alexander
d15836e260
Increase gocyclo complexity to 25 (and remove all but 2 golint directives related to it) (#1783) 2021-03-03 14:35:57 +00:00
Neil Alexander
b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander
9057143033
Hit the database far less in Events to find room NIDs and room versions (#1643)
* Hit the database far less to find room NIDs for event NIDs

* Close the rows

* Fix SQLite selectRoomNIDsForEventNIDsSQL

* Give same treatment to room version lookups
2020-12-16 10:33:28 +00:00
Neil Alexander
1ce9c52442
Don't recalculate event IDs so often (#1610)
* Don't recalculate event IDs so often

* Revert invite change

* Make sure we're using the right NIDs

* Update gomatrixserverlib

* Update to NewEventFromTrustedJSONWithEventID

* Fix go.mod

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#243

* Use BulkSelectEventID
2020-12-04 10:41:07 +00:00
Neil Alexander
b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
S7evinK
eccd0d2c1b
Implement forgetting about rooms (#1572)
* Add basic storage methods

* Add internal api handler

* Add check for forgotten room

* Add /rooms/{roomID}/forget endpoint

* Add missing rsAPI method

* Remove unused parameters

* Add passing tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing file

* Add postgres migration

* Add sqlite migration

* Use Forgetter to forget room

* Remove empty line

* Update HTTP status codes

It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-05 10:19:23 +00:00
Kegsay
6b8791b868
Always call overridden form of GetLatestEventsForUpdate (#1554)
This ensures we don't make txns on sqlite still, which can cause
'database is locked' errors.
2020-10-20 19:32:33 +01:00
Kegsay
eb86e2b336
Fix sqlite locking bugs present on sytest (#1543)
* Fix sqite locking bugs present on sytest

Comments do the explaining.

* Fix deadlock in sqlite mode

Caused by starting a writer whilst within a writer

* Only complain about invalid state deltas for non-overwrite events

* Do not re-process outlier unnecessarily
2020-10-20 11:42:54 +01:00
Neil Alexander
e154c45b51
Better logging around db.StoreEvent 2020-10-15 14:14:17 +01:00
Kegsay
e3c2b081c7
txn nil guard when rolling back LatestEventsUpdater (#1524)
* txn nil guard when rolling back LatestEventsUpdater

* Spell lint correctly
2020-10-14 17:05:09 +01:00
Neil Alexander
8d9ecb3996
Ignore duplicate redaction entries (#1522) 2020-10-14 15:24:43 +01:00
Kegsay
d7ea814fa8
Wrap NewMembershipUpdater in a db writer (#1515) 2020-10-13 10:20:27 +01:00
Neil Alexander
d821f9d3c9
Deep checking of forward extremities (#1491)
* Deep forward extremity calculation

* Use updater txn

* Update error

* Update error

* Create previous event references in StoreEvent

* Use latest events updater to row-lock prev events

* Fix unexpected fallthrough

* Fix deadlock

* Don't roll back

* Update comments in calculateLatest

* Don't include events that we can't find references for in the forward extremities

* Add another passing test
2020-10-07 14:05:33 +01:00
Neil Alexander
91fc1f1c92
Fix bug in error handling in SQLite InsertPreviousEvent (#1456) 2020-10-01 10:55:00 +01:00
Sam
a6700331ce
Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)
* Replace all usages of txn.Stmt with sqlutil.TxStmt

Signed-off-by: Sam Day <me@samcday.com>

* Fix sign off link in PR template.

Signed-off-by: Sam Day <me@samcday.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-24 11:10:14 +01:00
Neil Alexander
45de9dc1c0
Use room version cache in Events() 2020-09-21 16:49:37 +01:00
Neil Alexander
a06c18bb56
Soft-fail (#1364)
* Initial work on soft-fail

* Fix state block retrieval

* Copy-pasta QueryLatestEventsAndState code

* Fix state lookup

* Clean up

* Fix up failing sytest

* Linting

* Update previous events SQLite insert query

* Update SQLite InsertPreviousEvent properly

* Hopefully fix the event references updates

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-21 14:55:46 +01:00
Kegsay
18231f25b4
Implement rejected events (#1426)
* WIP Event rejection

* Still send back errors for rejected events

Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.

* Implement rejected events

Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.

* Update test to match reality

* Modify InputRoomEvents to no longer return an error

Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.

* Remove redundant returns; linting

* Update blacklist
2020-09-16 13:00:52 +01:00
Matthew Hodgson
39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Neil Alexander
35564dd73c
Process membership updates in writers (#1414) 2020-09-08 17:48:07 +01:00
Kegsay
7913759921
Remove QueryBulkStateContent from current state server (#1404)
* Remove QueryBulkStateContent from current state server

Expected fail due to db impl not existing

* Implement query bulk state content

* Fix up rejecting invites over federation

* Fix bulk content marshalling
2020-09-07 12:38:09 +01:00