Commit graph

106 commits

Author SHA1 Message Date
Matthew Hodgson
b96a31db51 track transitions more clearly 2020-09-08 11:57:49 +01:00
Matthew Hodgson
9f2bc6231f (broken) rewrite to use SelectPeeksInRange rather than MarkPeeksAsOld for idempotency 2020-09-08 11:05:54 +01:00
Matthew Hodgson
7f41f39684 remove erroneous commentary 2020-09-08 09:25:05 +01:00
Matthew Hodgson
034ff3208c move peek-cancelling to consumer 2020-09-08 01:42:27 +01:00
Matthew Hodgson
55c7f2c892 add (broken) postgres; advance streampos whenever sync output changes 2020-09-08 00:56:50 +01:00
Neil Alexander
2ccd4fa069
Clear a whole bunch of lint issues 2020-09-04 16:17:02 +01:00
Neil Alexander
ff65f0ec7b
Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"
This reverts commit 3cebd8dbfb, reversing
changes made to ed4b3a58a7.
2020-09-04 16:14:40 +01:00
Neil Alexander
56d772fafc
Cancel peeks on join again 2020-09-04 14:20:00 +01:00
Neil Alexander
2b8f0b8f59
Merge branch 'master' into matthew/peeking 2020-09-04 14:15:51 +01:00
Neil Alexander
c63abc8660
Sync bug fixes (#1394)
* Sync bug fixes

* Remove logging
2020-09-04 11:20:47 +01:00
Matthew Hodgson
b6cc4417cc re-add txn to SelectPeeks 2020-09-02 10:07:49 +01:00
Matthew Hodgson
75b91ac9e5 strip out empty roomd deltas 2020-09-01 21:59:35 +01:00
Matthew Hodgson
6424117852 use exclusive writer, and MarkPeeksAsOld more efficiently 2020-09-01 21:31:04 +01:00
Matthew Hodgson
5d7f688fa7 wrap peek storage in goid hack 2020-09-01 21:30:03 +01:00
Kegsay
82a9617659
Put redactions/filters in the writer goroutine (#1378)
* Put redactions in the writer goroutine

* Update filters on writer goroutine
2020-09-01 20:35:38 +01:00
Matthew Hodgson
3cebd8dbfb Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking 2020-09-01 21:37:22 +03:00
Matthew Hodgson
ed4b3a58a7 Merge branch 'kegan/redact-txn' into matthew/peeking 2020-09-01 21:34:01 +03:00
Kegan Dougal
6410b702ce Update filters on writer goroutine 2020-09-01 19:27:21 +01:00
Kegan Dougal
fcdb90c91b Put redactions in the writer goroutine 2020-09-01 19:21:33 +01:00
Kegan Dougal
7bf2a27319 Track partition offsets and only log unsafe for non-selects 2020-09-01 19:17:01 +01:00
Kegan Dougal
bfecc8e0e9 HACK: Track goroutine IDs to determine when we write by the wrong thread
To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`
2020-09-01 19:10:57 +01:00
Matthew Hodgson
d0d5f70105 Merge branch 'master' into matthew/peeking 2020-09-01 19:11:51 +03:00
Neil Alexander
a9f4d83d30
Fix duplicate writers (#1376)
* Fix writers

* Don't use writers in both shared and sqlite3
2020-09-01 16:58:21 +01:00
Matthew Hodgson
28219c66f5 Merge branch 'master' into matthew/peeking 2020-09-01 18:03:06 +03:00
Neil Alexander
0ab5bccd11
Storage tweaks (#1373)
* Sync API tweaks

* User API tweaks
2020-09-01 11:28:35 +01:00
Neil Alexander
b0d2b39739
Remove unused SyncStreamPosition 2020-09-01 10:26:02 +01:00
Matthew Hodgson
0bb2c2c418 remove unnecessary txn for SelectPeeks 2020-09-01 00:24:23 +03:00
Matthew Hodgson
e5899843ea fix SQL 2020-08-31 23:28:55 +03:00
Matthew Hodgson
6c3a896910 cancel any peeks when we join a room 2020-08-31 18:58:10 +03:00
Matthew Hodgson
f006b37bf0 add peeking to getResponseWithPDUsForCompleteSync 2020-08-31 18:16:07 +03:00
Matthew Hodgson
d1e4d66126 make it launch 2020-08-31 17:35:23 +03:00
Matthew Hodgson
c4e5f60d71 make it build 2020-08-31 16:12:09 +03:00
Matthew Hodgson
d343b8fb2c blind stab at adding a peek section to /sync 2020-08-31 15:28:31 +03:00
Matthew Hodgson
cfa0be544d merge master 2020-08-31 12:26:27 +03:00
Matthew Hodgson
b9342d9ee2 a very very WIP first cut of peeking via MSC2753.
doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.
2020-08-30 17:46:15 +03:00
Neil Alexander
c42c70597c
Add some error wrapping to sync API, use background context for sync (#1363)
* Add some error wrapping to sync API

* Don't use request context for BeginTx until mattn/go-sqlite3#764 is fixed
2020-08-28 13:28:40 +01:00
oliverpool
3205b9212d
[readability] use event.StateKeyEquals where relevant and minor for-loop refactoring (#1339)
* use event.StateKeyEquals where relevant

Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>

* use userID
2020-08-25 21:04:35 +01:00
Kegsay
55498c8deb
Fix 'Invited user can reject invite over federation several times' (#1341) 2020-08-25 18:59:00 +01:00
oliverpool
a4db43e096
Don't overwrite global err before return (#1293)
Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>
2020-08-25 13:11:52 +01:00
Neil Alexander
9d53351dc2
Component-wide TransactionWriters (#1290)
* Offset updates take place using TransactionWriter

* Refactor TransactionWriter in current state server

* Refactor TransactionWriter in federation sender

* Refactor TransactionWriter in key server

* Refactor TransactionWriter in media API

* Refactor TransactionWriter in server key API

* Refactor TransactionWriter in sync API

* Refactor TransactionWriter in user API

* Fix deadlocking Sync API tests

* Un-deadlock device database

* Fix appservice API

* Rename TransactionWriters to Writers

* Move writers up a layer in sync API

* Document sqlutil.Writer interface

* Add note to Writer documentation
2020-08-21 10:42:08 +01:00
Neil Alexander
5aaf32bbed
Select distinct on room memberships in sync API (#1292) 2020-08-21 09:57:52 +01:00
Neil Alexander
b24747b305
Transaction writer changes, move roomserver writers (#1285)
* Updated TransactionWriters, moved locks in roomserver, various other tweaks

* Fix redaction deadlocks

* Fix lint issue

* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter

* Fix us not sending transactions through in latest events updater
2020-08-19 15:38:27 +01:00
Neil Alexander
4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Kegsay
0fdd4f14d1
Add support for logs in StreamingToken (#1229)
* Add support for logs in StreamingToken

Tokens now end up looking like `s11_22|dl-0-123|ab-0-12224`
where `dl` and `ab` are log names, `0` is the partition and
`123` and `12224` are the offsets.

* Also test reserialisation

* s/|/./g so tokens url escape nicely
2020-07-29 19:00:04 +01:00
Henrik Sölver
83f038e12b
Don't use more than 999 variables in SQLite querys. (#1224)
Closes #1223

Signed-off-by: Henrik Sölver <henrik.solver@gmail.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-27 13:19:30 +01:00
Neil Alexander
b6bc132485
Use TransactionWriter in other component SQLite (#1209)
* Use TransactionWriter on other component SQLites

* Fix sync API tests

* Fix panic in media API

* Fix a couple of transactions

* Fix wrong query, add some logging output

* Add debug logging into StoreEvent

* Adjust InsertRoomNID

* Update logging
2020-07-21 15:48:21 +01:00
Neil Alexander
11a39fe3b5
Deduplicate FS database, EDU persistence table (#1207)
* Deduplicate FS database, add some EDU persistence groundwork

* Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes

* Fix build due to bad keyserver import

* Working EDU persistence

* gocyclo, unsurprisingly

* Remove unused

* Update copyright notices
2020-07-20 16:55:20 +01:00
Kegsay
d9648b0615
Finish implementing redactions (#1189)
* Add a bit more logging to the fedsender

* bugfix: continue sending PDUs if ones are added whilst sending another PDU

Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.

* Break if no txn is sent

* WIP syncapi work

* More debugging

* Bump GMSL version to pull in working Event.Redact

* Remove logging

* Make redactions work on v3+

* Fix more tests
2020-07-08 17:45:39 +01:00
Kegsay
1ad7219e4b
Implement /sync limited and read timeline limit from stored filters (#1168)
* Move filter table to syncapi where it is used

* Implement /sync `limited` and read timeline limit from stored filters

We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.

* Update whitelist

* Default to the default timeline limit if it's unset, also strip the extra event correctly

* Update whitelist
2020-06-26 15:34:41 +01:00
Kegsay
4897beabee
Finish implementing retiring invites (#1166)
* Pass retired invites to the syncapi with the event ID of the invite

* Implement retire invite streaming

* Update whitelist
2020-06-26 11:07:52 +01:00