Commit graph

66 commits

Author SHA1 Message Date
Neil Alexander
c997461db1
Logging 2021-07-30 10:54:39 +01:00
Neil Alexander
efe01519a2
Don't give up so quickly on errors 2021-07-30 10:50:49 +01:00
Neil Alexander
ddb60859b7
Let's try this again 2021-07-30 10:34:38 +01:00
Neil Alexander
67421a9d65
Let's try this again 2021-07-30 10:28:33 +01:00
Neil Alexander
1fed507679
Check per-user 2021-07-30 10:23:36 +01:00
Neil Alexander
0571b7eb1d
Try harder to query remote keys 2021-07-30 10:18:39 +01:00
Neil Alexander
5a00ff3634
Try harder to query remote keys 2021-07-30 09:58:03 +01:00
Neil Alexander
d459ef5b16
Hopefully get keys from remote query 2021-07-30 09:55:05 +01:00
Neil Alexander
78ab127ced
Some groundwork 2021-07-29 18:22:27 +01:00
Neil Alexander
39d66ed9d6
Some refactoring 2021-07-29 15:47:30 +01:00
Neil Alexander
4802216752
Initial signature upload support maybe 2021-07-29 15:29:16 +01:00
Neil Alexander
d84883d5c4
Add insert functions for cross-signing signatures 2021-07-29 14:47:09 +01:00
Neil Alexander
f1155d26bf
Allocate the map 2021-07-29 14:39:12 +01:00
Neil Alexander
fba18e8b17
Include signatures in key query 2021-07-29 14:35:42 +01:00
Neil Alexander
870c09be03
Fix error values 2021-07-29 13:13:10 +01:00
Neil Alexander
6e56dbc566
Only the master key signature matters 2021-07-29 13:10:10 +01:00
Neil Alexander
5de569be2c
A fix 2021-07-29 12:54:03 +01:00
Neil Alexander
5654b08819
Try this 2021-07-29 12:50:37 +01:00
Neil Alexander
2fa479edca
Validate signatures 2021-07-29 12:45:29 +01:00
Neil Alexander
93bf1ffc10
Storing device keys part 1 2021-07-29 09:48:09 +01:00
Neil Alexander
dd33f2b76d
Sanity checking of uploads 2021-07-28 17:07:57 +01:00
Neil Alexander
701ffdeb36
Generate key IDs automatically 2021-07-28 16:41:53 +01:00
Neil Alexander
b638b06ec1
Allocate the maps first 2021-07-28 16:30:28 +01:00
Neil Alexander
06de424b2b
Allocate more maps 2021-07-28 16:29:08 +01:00
Neil Alexander
64647a4f87
Fix QueryKeys ordering 2021-07-28 16:09:24 +01:00
Neil Alexander
351ac51691
Logging 2021-07-28 16:07:39 +01:00
Neil Alexander
a6bb3fd0ac
Retrieve cross-signing keys sorta 2021-07-28 15:52:52 +01:00
Neil Alexander
a5bb2afbb5
Add API wiring 2021-07-28 12:31:37 +01:00
Kegsay
b769d5a25e
Optimise memory usage when calling /g_m_e (#1819)
* Optimise memory usage when calling /g_m_e

* cache more events

* refactor handling of device list update pokes

* Sigh
2021-04-08 13:50:39 +01:00
Kegsay
802f1c96f8
Add more metrics (#1802)
* Add more metrics

* Linting
2021-03-23 15:22:00 +00:00
Kegsay
a1b7e4ef3f
log less for failed key querys, add counters for incoming pdus/edus (#1801)
* log less for failed key querys, add counters for incoming pdus/edus

* use labels

* Blacklist flakey test

* Fix metrics
2021-03-23 11:33:36 +00:00
Kegsay
77fb981da5
device lists: backoff for longer if the wrong error type is returned (#1796) 2021-03-08 17:45:20 +00:00
Neil Alexander
81312b8a78
Return the current OTK count on an empty upload request (#1774)
* Always return OTK counts

* Fix parameter ordering

* Send IDs over to keyserver internal API

* Review comments

* Fix syntax error

* Fix panic, hopefully

* Require user ID to be set

* Fix user API call
2021-03-02 11:40:20 +00:00
Neil Alexander
6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Loïck Bonniot
940577cd3c
Fix integer overflow in device_list_update.go (#1717)
Fix #1511
On 32-bits systems, int(hash.Sum32()) can be negative.
This makes the computation of array indices using modulo invalid, crashing dendrite.
Signed-off-by: Loïck Bonniot <git@lesterpig.com>
2021-01-18 12:43:15 +00:00
Neil Alexander
fa65c40bae
Reduce device list GetUserDevices timeout (#1704) 2021-01-12 16:13:21 +00:00
Neil Alexander
fe5d1400bf
Update federation timeouts (#1504)
* Update to matrix-org/gomatrixserverlib#234

* Update gomatrixserverlib

* Update federation timeouts

* Fix dendritejs

* Increase /send context time in destination queue
2020-10-09 17:08:32 +01:00
Kegsay
29d6481842
Wait for 8h between device list updates for blacklisted servers (#1344) 2020-08-26 15:38:21 +01:00
Kegsay
abd16ff4a0
Modify DeviceListUpdater to retry requests according to RetryAfter (#1342)
* Modify DeviceListUpdater to retry requests according to RetryAfter

* Reduce wait time for sytest test pollution
2020-08-26 12:03:09 +01:00
Kegsay
6d6bb75137
Add FederationClient interface to federationsender (#1284)
* Add FederationClient interface to federationsender

- Use a shim struct in HTTP mode to keep the same API as `FederationClient`.
- Use `federationsender` instead of `FederationClient` in `keyserver`.

* Pointers not values

* Review comments

* Fix unit tests

* Rejig backoff

* Unbreak test

* Remove debug logs

* Review comments and linting
2020-08-20 17:03:07 +01:00
Kegsay
02a8515e99
Only emit key changes which are different from what we had before (#1279)
We did this already for local `/keys/upload` but didn't for
remote `/users/devices`. This meant any resyncs would spam produce
events, hammering disk i/o and spamming the logs.
2020-08-18 11:14:20 +01:00
Kegsay
20c8f252a7
Make 'Device list doesn't change if remote server is down' pass (#1268)
- As a last resort, query the DB when exhausting all possible remote query
  endpoints, but keep the field in `failures` so clients can detect that this
  is stale data.
- Unblock `DeviceListUpdater.Update` on failures rather than timing out.
- Use a mutex when writing directly to `res`, not just for failures.
2020-08-13 16:43:27 +01:00
Kegsay
820c56c165
Fix more E2E sytests (#1265)
* WIP: Eagerly sync device lists on /user/keys/query requests

Also notify servers when a user's device display name changes. Few
caveats:
 - sytest `Device deletion propagates over federation` fails
 - `populateResponseWithDeviceKeysFromDatabase` is called from multiple
   goroutines and hence is unsafe.

* Handle deleted devices correctly over federation
2020-08-12 22:43:02 +01:00
Kegsay
d98ec12422
Add sync mechanism to block when updating device lists (#1264)
* Add sync mechanism to block when updating device lists

With a timeout, mainly for sytest to fix the test
"Server correctly handles incoming m.device_list_update"
which is flakey because it assumes that when `/send` 200 OKs
that the server has updated the device lists in prep for
`/keys/query` which is not always true when using workers.

* Fix UT

* Add new working test
2020-08-12 13:50:54 +01:00
Kegsay
b8b854d642
Bugfixes for 'If remote user leaves room we no longer receive device updates' (#1262)
* Bugfixes for 'If remote user leaves room we no longer receive device updates'

* Update whitelist and README
2020-08-12 10:50:52 +01:00
Kegsay
befccd7d51
Reduce cooldown to make sure sytest doesn't give up (#1257)
* Reduce cooldown to make sure sytest doesn't give up

* More sytests pass weeeeeee
2020-08-11 10:44:59 +01:00
Kegan Dougal
366fd975c8 Fix unit tests 2020-08-07 17:58:44 +01:00
Kegsay
f371783da7
Finish inbound E2E device lists (#1243)
* Add tests for device list updates

* Add stale_device_lists table and use db before asking remote for device keys

* Fetch remote keys if all devices are requested

* Add display_name col to store remote device names

Few other tweaks to make `Server correctly handles incoming m.device_list_update`
pass.

* Fix sqlite otk bug

* Unbuffered channel to block /send causing sytest to not race anymore

* Linting and fix bug whereby we didn't send updated dl tokens to the client causing a tightloop on /sync sometimes

* No longer assert staleness as Update blocks on workers now

* Back out tweaks

* Bugfixes
2020-08-07 17:32:13 +01:00
Kegsay
32a4565b55
Add device list updater which manages updating remote device lists (#1242)
* Add device list updater which manages updating remote device lists

- Doesn't persist stale lists to the database yet
- Doesn't have tests yet

* Mark device lists as fresh when we persist
2020-08-06 17:48:10 +01:00
Kegsay
642f9cb964
Process inbound device list updates from federation (#1240)
* Add InputDeviceListUpdate

* Unbreak unit tests

* Process inbound device list updates from federation

- Persist the keys in the keyserver and produce key changes
- Does not currently fetch keys from the remote server if the prev IDs are missing

* Linting
2020-08-05 13:41:16 +01:00