Commit graph

199 commits

Author SHA1 Message Date
Neil Alexander
84d089caed
Still doesn't work 2021-01-21 13:31:03 +00:00
Neil Alexander
e254594453
Some refactoring 2021-01-21 11:18:10 +00:00
Neil Alexander
9b2b7a6e28
Merge branch 'master' into neilalexander/historyvis 2021-01-21 09:12:57 +00:00
Neil Alexander
8fe51019ad
Add sync API memberships table (#1726) 2021-01-20 20:43:20 +00:00
Neil Alexander
9ee4ca0088
Tweak messages 2021-01-20 16:24:07 +00:00
Neil Alexander
10f2e8d92f
Start his vis 2021-01-20 14:41:09 +00:00
Neil Alexander
33f8d0e278
Clean up 2021-01-20 10:36:59 +00:00
Neil Alexander
0096fc850a
Refactor removeDuplicates (might use slightly more memory but less CPU) 2021-01-20 10:29:32 +00:00
Neil Alexander
dd781907b1
Tweaks 2021-01-20 09:35:44 +00:00
Neil Alexander
9d406e1283
Merge branch 'master' into 1323-archived-rooms-sync-left-rooms 2021-01-19 18:09:00 +00:00
Neil Alexander
b70238f2d5
Basic sync filtering (#1721)
* Add some filtering (postgres only for now)

* Fix build error

* Try to use request filter

* Use default filter as a template when retrieving from the database

* Remove unused strut

* Update sytest-whitelist

* Add filtering to SelectEarlyEvents

* Fix Postgres selectEarlyEvents query

* Attempt filtering on SQLite

* Test limit, set field for limit/order in prepareWithFilters

* Remove debug logging, add comments

* Tweaks, debug logging

* Separate SQLite stream IDs

* Fix filtering in current state table

* Fix lock issues

* More tweaks

* Current state requires room ID

* Review comments
2021-01-19 18:00:42 +00:00
Neil Alexander
10b43f86ed
Include membership events 2021-01-18 16:23:37 +00:00
Neil Alexander
534c29ab02
Log event ID on consumer errors (fixes #1714) 2021-01-18 12:58:48 +00:00
Eric Eastwood
6fd8459447 Try fix unit tests 2021-01-16 01:07:52 -06:00
Eric Eastwood
fe7acc63fa Derive state events from already filtered events 2021-01-15 02:28:50 -06:00
Eric Eastwood
370b3d981e Get Sytest working back to the Archive test 2021-01-14 02:50:11 -06:00
Eric Eastwood
76e1faeaf8 Try get Sytest working again
Currently it's failing way earlier on "Newly left rooms appear in the leave section of incremental sync"
and I've commented out all of my new functionality
so need to figure out what is interfering in this branch now
2021-01-14 01:59:24 -06:00
Neil Alexander
1be95012d4
Merge branch 'master' into 1323-archived-rooms-sync-left-rooms 2021-01-13 17:37:07 +00:00
Neil Alexander
d8fba52e97
Simplify send-to-device messaging (#1702)
* Simplify send-to-device messaging

* Don't return error if there's no work to do

* Remove SQLite migrations for now

* Tweak Postgres migrations

* Tweaks

* Fixes

* Cleanup separately

* Fix SQLite migration
2021-01-13 17:29:46 +00:00
Neil Alexander
bb9e6a1281
Sync fixes (#1709)
* omitempty some fields in sync

* Add a few more

* Don't send push rules over and over again in incremental sync

* Further tweaks
2021-01-13 14:32:49 +00:00
Neil Alexander
55cfe391f7
Update /messages pagination token behaviour (#1708)
* Tweak pagination tokens

* start should be the specified from

* Don't reverse start and end

* Tweak getStartEnd again

* Update sytest-whitelist

* NOTSPEC: Re-add iOS end of topology
2021-01-13 12:59:29 +00:00
Neil Alexander
707f347f34
Less restrictive visibility filtering 2021-01-13 10:54:19 +00:00
Neil Alexander
0231199cc8
Tweaks 2021-01-13 10:24:11 +00:00
Neil Alexander
be9ee18233
Tidy filterStreamEventsAccordingToHistoryVisibility 2021-01-13 10:09:26 +00:00
Neil Alexander
2322d8f027
Merge branch 'master' into 1323-archived-rooms-sync-left-rooms 2021-01-13 10:06:21 +00:00
Neil Alexander
0fbebecd41
Tweak ApplyUpdates (#1691) 2021-01-09 11:25:09 +00:00
Neil Alexander
b5a8935042
Sync refactor — Part 1 (#1688)
* It's half-alive

* Wakeups largely working

* Other tweaks, typing works

* Fix bugs, add receipt stream

* Delete notifier, other tweaks

* Dedupe a bit, add a template for the invite stream

* Clean up, add templates for other streams

* Don't leak channels

* Bring forward some more PDU logic, clean up other places

* Add some more wakeups

* Use addRoomDeltaToResponse

* Log tweaks, typing fixed?

* Fix timed out syncs

* Don't reset next batch position on timeout

* Add account data stream/position

* End of day

* Fix complete sync for receipt, typing

* Streams package

* Clean up a bit

* Complete sync send-to-device

* Don't drop errors

* More lightweight notifications

* Fix typing positions

* Don't advance position on remove again unless needed

* Device list updates

* Advance account data position

* Use limit for incremental sync

* Limit fixes, amongst other things

* Remove some fmt.Println

* Tweaks

* Re-add notifier

* Fix invite position

* Fixes

* Notify account data without advancing PDU position in notifier

* Apply account data position

* Get initial position for account data

* Fix position update

* Fix complete sync positions

* Review comments @Kegsay

* Room consumer parameters
2021-01-08 16:59:06 +00:00
Eric Eastwood
783fba2d6a Disable IncrementalSync stuff and re-assess 2021-01-06 04:31:50 -06:00
Eric Eastwood
c7328cee41 Some WIP code for IncrementalSync but breaks other Sytests 2021-01-06 04:31:48 -06:00
Eric Eastwood
9e4fd0157f Properly cut out join and leave events 2021-01-06 04:30:59 -06:00
Eric Eastwood
810dd7842c Filter out events after the leave event 2021-01-06 04:30:59 -06:00
Eric Eastwood
88374a5cc4 Add leave rooms to /sync response 2021-01-06 04:30:59 -06:00
Eric Eastwood
4cf73a5dc2 Refactor to allow passing actual filter 2021-01-06 04:30:47 -06:00
Eric Eastwood
82afb32464 Implement archived rooms (include_leave filter on /sync)
Fix https://github.com/matrix-org/dendrite/issues/1323
2021-01-06 04:28:54 -06:00
6543
a85051ca9d
fix imports (#1665)
* fix imports

Signed-off-by: 6543 <6543@obermui.de>

* add sqlite driver import back

Signed-off-by: 6543 <6543@obermui.de>

* rm import of userapi/storage/accounts/sqlite3/storage.go
2020-12-21 14:27:01 +00:00
Neil Alexander
4fe4c180e6
Ensure we wake for our own device list updates (#1661)
* Make sure we wake up for our own key changes

* Whitelist 'Users receive device_list updates for their own devices'
2020-12-18 14:04:17 +00:00
Neil Alexander
50963b724b
More sane next batch handling, typing notification tweaks, give invites their own stream position, device list fix (#1641)
* Update sync responses

* Fix positions, add ApplyUpdates

* Fix MarshalText as non-pointer, PrevBatch is optional

* Increment by number of read receipts

* Merge branch 'master' into neilalexander/devicelist

* Tweak typing

* Include keyserver position tweak

* Fix typing next position in all cases

* Tweaks

* Fix typo

* Tweaks, restore StreamingToken.MarshalText which somehow went missing?

* Rely on positions from notifier rather than manually advancing them

* Revert "Rely on positions from notifier rather than manually advancing them"

This reverts commit 53112a62cc3bfd9989acab518e69eeb27938117a.

* Give invites their own position, fix other things

* Fix test

* Fix invites maybe

* Un-whitelist tests that look to be genuinely wrong

* Use real receipt positions

* Ensure send-to-device uses real positions too
2020-12-18 11:11:21 +00:00
Neil Alexander
e1ace7e44a
Add event ID index on current state table (helps performance) (#1649) 2020-12-16 18:16:39 +00:00
Kegsay
76becac003
Add start_stream to /messages (#1648) 2020-12-16 18:10:39 +00:00
Kegsay
42e9cbf342
NOTSPEC: Make ?from= optional in /messages (#1647) 2020-12-16 17:31:03 +00:00
Neil Alexander
56b5847c74
Add prometheus metrics for destination queues, sync requests
Squashed commit of the following:

commit 7ed1c6cfe67429dbe378a763d832c150eb0f781d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:53:27 2020 +0000

    Updates

commit 8442099d08760b8d086e6d58f9f30284e378a2cd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:43:18 2020 +0000

    Add some sync statistics

commit ffe2a11644ed3d5297d1775a680886c574143fdb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:37:00 2020 +0000

    Fix backing off display

commit 27443a93855aa60a49806ecabbf9b09f818301bd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:28:43 2020 +0000

    Add some destination queue metrics
2020-12-16 15:02:39 +00:00
Neil Alexander
38318b0f16
De-map device list positions in streaming tokens (#1642)
* De-map device list positions in streaming tokens

* Fix lint error

* Tweak toOffset
2020-12-15 15:09:10 +00:00
Neil Alexander
ebcacd1bb5
Give receipts their own stream ID in the database (#1631)
* Give read recipts their own database sequence

* Give receipts their own stream ID

* Change migration names

* Reset sequences

* Add max receipt queries, missing stream_id table entry for SQLite
2020-12-11 14:02:17 +00:00
Neil Alexander
9c03b0a4fa
Refactor sync tokens (#1628)
* Refactor sync tokens

* Comment out broken notifier test

* Update types, sytest-whitelist

* More robust token checking

* Remove New functions for streaming tokens

* Export Logs in StreamingToken

* Fix tests
2020-12-10 18:57:10 +00:00
Neil Alexander
bad81c028f
Don't recalculate event ID so often in sync (#1624)
* Don't bail so quickly in fetchMissingStateEvents

* Don't recalculate event IDs so often in sync API

* Add comments

* Fix comments

* Update to matrix-org/gomatrixserverlib@eb6a890
2020-12-09 18:07:17 +00:00
Neil Alexander
be7d8595be
Peeking updates (#1607)
* Add unpeek

* Don't allow peeks into encrypted rooms

* Fix send tests

* Update consumers
2020-12-03 11:11:46 +00:00
Neil Alexander
b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander
417c7d3569
Send client events to appservices (#1603)
* Send client events to appservices

* FormatSync instead of FormatAll
2020-12-02 15:14:12 +00:00
Ariadne Conill
d5b8260196
syncapi/requestpool: fix initial sync logic error in appendAccountData() (#1594)
* requestpool: fix initial sync logic error in appendAccountData()

In initial sync, req.since is no longer nil, but instead,
req.since.PDUPosition() and req.since.EDUPosition() returns 0.

This ensures forgotten rooms do not come back as zombies.

* syncapi/requestpool: reintroduce req.since == nil check
2020-12-01 17:01:34 +00:00
Neil Alexander
c636be5070
Update last seen on sync requests (#1593)
* Update last seen on sync requests

* Fix MSC2836 unit tests

* Only update once per minute

* Remove debug logging

* Configurable option

* Simplify updateLastSeen/cleanLastSeen
2020-11-20 11:29:02 +00:00