diff --git a/CHANGES.md b/CHANGES.md index c909c571..27356b3c 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,5 +1,46 @@ # Changelog +## Dendrite 0.4.0 (2021-07-12) + +### Features + +* All-new state storage in the roomserver, which dramatically reduces disk space utilisation + * State snapshots and blocks are now aggressively deduplicated and reused wherever possible, with state blocks being reduced by up to 15x and snapshot references being reduced up to 2x + * Dendrite will upgrade to the new state storage automatically on the first run after upgrade, although this may take some time depending on the size of the state storage +* Appservice support has been improved significantly, with many bridges now working correctly with Dendrite + * Events are now correctly sent to appservices based on room memberships + * Aliases and namespaces are now handled correctly, calling the appservice to query for aliases as needed + * Appservice user registrations are no longer being subject to incorrect validation checks +* Shared secret registration has now been implemented correctly +* The roomserver input API implements a new queuing system to reduce backpressure across rooms +* Checking if the local server is in a room has been optimised substantially, reducing CPU usage +* State resolution v2 has been optimised further by improving the power level checks, reducing CPU usage +* The federation API `/send` endpoint now deduplicates missing auth and prev events more aggressively to reduce memory usage +* The federation API `/send` endpoint now uses workers to reduce backpressure across rooms +* The bcrypt cost for password storage is now configurable with the `user_api.bcrypt_cost` option +* The federation API will now use significantly less memory when calling `/get_missing_events` +* MSC2946 Spaces endpoints have been updated to stable endpoint naming +* The media API can now be configured without a maximum file size +* A new `dendrite-upgrade-test` test has been added for verifying database schema upgrades across versions +* Added Prometheus metrics for roomserver backpressure, excessive device list updates and federation API event processing summaries +* Sentry support has been added for error reporting + +### Fixes + +* Removed the legacy `/v1` register endpoint. Dendrite only implements `/r0` of the CS API, and the legacy `/v1` endpoint had implementation errors which made it possible to bypass shared secret registration (thanks to Jakob Varmose Bentzen for reporting this) +* Attempting to register an account that already exists now returns a sensible error code rather than a HTTP 500 +* Dendrite will no longer attempt to `/make_join` with itself if listed in the request `server_names` +* `/sync` will no longer return immediately if there is nothing to sync, which happened particularly with new accounts, causing high CPU usage +* Malicious media uploads can no longer exhaust all available memory (contributed by [S7evinK](https://github.com/S7evinK)) +* Selecting one-time keys from the database has been optimised (contributed by [S7evinK](https://github.com/S7evinK)) +* The return code when trying to fetch missing account data has been fixed (contributed by [adamgreig](https://github.com/adamgreig)) +* Dendrite will no longer attempt to use `/make_leave` over federation when rejecting a local invite +* A panic has been fixed in `QueryMembershipsForRoom` +* A panic on duplicate membership events has been fixed in the federation sender +* A panic has been fixed in in `IsInterestedInRoomID` (contributed by [bodqhrohro](https://github.com/bodqhrohro)) +* A panic in the roomserver has been fixed when handling empty state sets +* A panic in the federation API has been fixed when handling cached events + ## Dendrite 0.3.11 (2021-03-02) ### Fixes diff --git a/are-we-synapse-yet.list b/are-we-synapse-yet.list index 99091989..d057e8e3 100644 --- a/are-we-synapse-yet.list +++ b/are-we-synapse-yet.list @@ -17,6 +17,13 @@ reg POST /register rejects registration of usernames with '£' reg POST /register rejects registration of usernames with 'é' reg POST /register rejects registration of usernames with '\n' reg POST /register rejects registration of usernames with ''' +reg POST /register allows registration of usernames with 'q' +reg POST /register allows registration of usernames with '3' +reg POST /register allows registration of usernames with '.' +reg POST /register allows registration of usernames with '_' +reg POST /register allows registration of usernames with '=' +reg POST /register allows registration of usernames with '-' +reg POST /register allows registration of usernames with '/' reg POST /r0/admin/register with shared secret reg POST /r0/admin/register admin with shared secret reg POST /r0/admin/register with shared secret downcases capitals @@ -76,7 +83,7 @@ rst GET /rooms/:room_id/state/m.room.topic gets topic rst GET /rooms/:room_id/state fetches entire room state crm POST /createRoom with creation content ali PUT /directory/room/:room_alias creates alias -nsp GET /rooms/:room_id/aliases lists aliases +ali GET /rooms/:room_id/aliases lists aliases jon POST /rooms/:room_id/join can join a room jon POST /join/:room_alias can join a room jon POST /join/:room_id can join a room @@ -95,6 +102,7 @@ typ Typing notifications don't leak (3 subtests) rst GET /rooms/:room_id/state/m.room.power_levels can fetch levels rst PUT /rooms/:room_id/state/m.room.power_levels can set levels rst PUT power_levels should not explode if the old power levels were empty +rst Users cannot set notifications powerlevel higher than their own (2 subtests) rst Both GET and PUT work rct POST /rooms/:room_id/receipt can create receipts red POST /rooms/:room_id/read_markers can create read marker @@ -175,7 +183,7 @@ ali Users with sufficient power-level can delete other's aliases ali Can delete canonical alias ali Alias creators can delete alias with no ops ali Alias creators can delete canonical alias with no ops -ali Only room members can list aliases of a room +msc Only room members can list aliases of a room inv Can invite users to invite-only rooms inv Uninvited users cannot join the room inv Invited user can reject invite @@ -353,6 +361,7 @@ syn Syncing a new room with a large timeline limit isn't limited syn A full_state incremental update returns only recent timeline syn A prev_batch token can be used in the v1 messages API syn A next_batch token can be used in the v1 messages API +syn A prev_batch token from incremental sync can be used in the v1 messages API syn User sees their own presence in a sync syn User is offline if they set_presence=offline in their sync syn User sees updates to presence from other users in the incremental sync. @@ -574,6 +583,7 @@ fqu Outbound federation can query profile data fqu Inbound federation can query profile data fqu Outbound federation can query room alias directory fqu Inbound federation can query room alias directory +fsj Membership event with an invalid displayname in the send_join response should not cause room join to fail fsj Outbound federation can query v1 /send_join fsj Outbound federation can query v2 /send_join fmj Outbound federation passes make_join failures through to the client @@ -596,7 +606,7 @@ fsj Inbound: send_join rejects invalid JSON for room version 6 fed Outbound federation can send events fed Inbound federation can receive events fed Inbound federation can receive redacted events -fed Ephemeral messages received from servers are correctly expired +msc Ephemeral messages received from servers are correctly expired fed Events whose auth_events are in the wrong room do not mess up the room state fed Inbound federation can return events fed Inbound federation redacts events from erased users @@ -743,6 +753,10 @@ nsp Set group joinable and join it nsp Group is not joinable by default nsp Group is joinable over federation nsp Room is transitioned on local and remote groups upon room upgrade +nsp POST /_synapse/admin/v1/register with shared secret +nsp POST /_synapse/admin/v1/register admin with shared secret +nsp POST /_synapse/admin/v1/register with shared secret downcases capitals +nsp POST /_synapse/admin/v1/register with shared secret disallows symbols 3pd Can bind 3PID via home server 3pd Can bind and unbind 3PID via homeserver 3pd Can unbind 3PID via homeserver when bound out of band @@ -859,8 +873,14 @@ jso Invalid JSON special values inv Can invite users to invite-only rooms (2 subtests) plv setting 'm.room.name' respects room powerlevel (2 subtests) psh Messages that notify from another user increment notification_count -psh Messages that org.matrix.msc2625.mark_unread from another user increment org.matrix.msc2625.unread_count +msc Messages that org.matrix.msc2625.mark_unread from another user increment org.matrix.msc2625.unread_count dvk Can claim one time key using POST (2 subtests) fdk Can query remote device keys using POST (1 subtests) fdk Can claim remote one time key using POST (2 subtests) -fmj Inbound /make_join rejects attempts to join rooms where all users have left \ No newline at end of file +fmj Inbound /make_join rejects attempts to join rooms where all users have left +msc Local users can peek into world_readable rooms by room ID +msc We can't peek into rooms with shared history_visibility +msc We can't peek into rooms with invited history_visibility +msc We can't peek into rooms with joined history_visibility +msc Local users can peek by room alias +msc Peeked rooms only turn up in the sync for the device who peeked them diff --git a/are-we-synapse-yet.py b/are-we-synapse-yet.py index 3d21fa41..92c7b82b 100755 --- a/are-we-synapse-yet.py +++ b/are-we-synapse-yet.py @@ -35,6 +35,7 @@ test_mappings = { "nsp": "Non-Spec API", "unk": "Unknown API (no group specified)", "app": "Application Services API", + "msc": "MSCs", "f": "Federation", # flag to mark test involves federation "federation_apis": { @@ -223,6 +224,7 @@ def main(results_tap_path, verbose): }, "nonspec": { "nsp": {}, + "msc": {}, "unk": {} }, } @@ -237,6 +239,8 @@ def main(results_tap_path, verbose): summary["nonspec"]["unk"][name] = test_result["ok"] if group_id == "nsp": summary["nonspec"]["nsp"][name] = test_result["ok"] + elif group_id == "msc": + summary["nonspec"]["msc"][name] = test_result["ok"] elif group_id == "app": summary["appservice"]["app"][name] = test_result["ok"] elif group_id in test_mappings["federation_apis"]: diff --git a/build/docker/README.md b/build/docker/README.md index 6d3cd3db..19e4234c 100644 --- a/build/docker/README.md +++ b/build/docker/README.md @@ -28,8 +28,8 @@ There are three sample `docker-compose` files: The `docker-compose` files refer to the `/etc/dendrite` volume as where the runtime config should come from. The mounted folder must contain: -- `dendrite.yaml` configuration file (based on the sample `dendrite-config.yaml` - in the `docker/config` folder in the [Dendrite repository](https://github.com/matrix-org/dendrite) +- `dendrite.yaml` configuration file (based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml) + sample in the `build/docker/config` folder of this repository.) - `matrix_key.pem` server key, as generated using `cmd/generate-keys` - `server.crt` certificate file - `server.key` private key file for the above certificate @@ -50,8 +50,7 @@ The key files will now exist in your current working directory, and can be mount ## Starting Dendrite as a monolith deployment -Create your config based on the `dendrite.yaml` configuration file in the `docker/config` -folder in the [Dendrite repository](https://github.com/matrix-org/dendrite). +Create your config based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml) configuration file in the `build/docker/config` folder of this repository. And rename the config file to `dendrite.yml` (and put it in your `config` directory). Once in place, start the PostgreSQL dependency: @@ -67,8 +66,7 @@ docker-compose -f docker-compose.monolith.yml up ## Starting Dendrite as a polylith deployment -Create your config based on the `dendrite.yaml` configuration file in the `docker/config` -folder in the [Dendrite repository](https://github.com/matrix-org/dendrite). +Create your config based on the [`dendrite-config.yaml`](https://raw.githubusercontent.com/matrix-org/dendrite/master/dendrite-config.yaml) configuration file in the `build/docker/config` folder of this repository. And rename the config file to `dendrite.yml` (and put it in your `config` directory). Once in place, start all the dependencies: @@ -84,10 +82,10 @@ docker-compose -f docker-compose.polylith.yml up ## Building the images -The `docker/images-build.sh` script will build the base image, followed by +The `build/docker/images-build.sh` script will build the base image, followed by all of the component images. -The `docker/images-push.sh` script will push them to Docker Hub (subject +The `build/docker/images-push.sh` script will push them to Docker Hub (subject to permissions). If you wish to build and push your own images, rename `matrixdotorg/dendrite` to diff --git a/build/gobind-pinecone/monolith.go b/build/gobind-pinecone/monolith.go index 09af80f6..e30057ed 100644 --- a/build/gobind-pinecone/monolith.go +++ b/build/gobind-pinecone/monolith.go @@ -334,6 +334,7 @@ func (m *DendriteMonolith) Start() { base.PublicFederationAPIMux, base.PublicKeyAPIMux, base.PublicMediaAPIMux, + base.SynapseAdminMux, ) httpRouter := mux.NewRouter().SkipClean(true).UseEncodedPath() diff --git a/build/gobind-yggdrasil/monolith.go b/build/gobind-yggdrasil/monolith.go index 5074f6da..d5aedeac 100644 --- a/build/gobind-yggdrasil/monolith.go +++ b/build/gobind-yggdrasil/monolith.go @@ -45,10 +45,6 @@ func (m *DendriteMonolith) PeerCount() int { return m.YggdrasilNode.PeerCount() } -func (m *DendriteMonolith) SessionCount() int { - return m.YggdrasilNode.SessionCount() -} - func (m *DendriteMonolith) SetMulticastEnabled(enabled bool) { m.YggdrasilNode.SetMulticastEnabled(enabled) } @@ -78,7 +74,7 @@ func (m *DendriteMonolith) Start() { panic(err) } - ygg, err := yggconn.Setup("dendrite", m.StorageDirectory) + ygg, err := yggconn.Setup("dendrite", m.StorageDirectory, "") if err != nil { panic(err) } @@ -87,7 +83,7 @@ func (m *DendriteMonolith) Start() { cfg := &config.Dendrite{} cfg.Defaults() cfg.Global.ServerName = gomatrixserverlib.ServerName(ygg.DerivedServerName()) - cfg.Global.PrivateKey = ygg.SigningPrivateKey() + cfg.Global.PrivateKey = ygg.PrivateKey() cfg.Global.KeyID = gomatrixserverlib.KeyID(signing.KeyID) cfg.Global.Kafka.UseNaffka = true cfg.Global.Kafka.Database.ConnectionString = config.DataSource(fmt.Sprintf("file:%s/dendrite-p2p-naffka.db", m.StorageDirectory)) @@ -134,18 +130,6 @@ func (m *DendriteMonolith) Start() { asAPI := appservice.NewInternalAPI(base, userAPI, rsAPI) rsAPI.SetAppserviceAPI(asAPI) - ygg.SetSessionFunc(func(address string) { - req := &api.PerformServersAliveRequest{ - Servers: []gomatrixserverlib.ServerName{ - gomatrixserverlib.ServerName(address), - }, - } - res := &api.PerformServersAliveResponse{} - if err := fsAPI.PerformServersAlive(context.TODO(), req, res); err != nil { - logrus.WithError(err).Error("Failed to send wake-up message to newly connected node") - } - }) - // The underlying roomserver implementation needs to be able to call the fedsender. // This is different to rsAPI which can be the http client which doesn't need this dependency rsAPI.SetFederationSenderAPI(fsAPI) @@ -173,6 +157,7 @@ func (m *DendriteMonolith) Start() { base.PublicFederationAPIMux, base.PublicKeyAPIMux, base.PublicMediaAPIMux, + base.SynapseAdminMux, ) httpRouter := mux.NewRouter() diff --git a/clientapi/clientapi.go b/clientapi/clientapi.go index 2c4fa5d6..562d89d2 100644 --- a/clientapi/clientapi.go +++ b/clientapi/clientapi.go @@ -35,6 +35,7 @@ import ( // AddPublicRoutes sets up and registers HTTP handlers for the ClientAPI component. func AddPublicRoutes( router *mux.Router, + synapseAdminRouter *mux.Router, cfg *config.ClientAPI, accountsDB accounts.Database, federation *gomatrixserverlib.FederationClient, @@ -56,7 +57,7 @@ func AddPublicRoutes( } routing.Setup( - router, cfg, eduInputAPI, rsAPI, asAPI, + router, synapseAdminRouter, cfg, eduInputAPI, rsAPI, asAPI, accountsDB, userAPI, federation, syncProducer, transactionsCache, fsAPI, keyAPI, extRoomsProvider, mscCfg, ) diff --git a/clientapi/routing/register.go b/clientapi/routing/register.go index 52641866..8823a41e 100644 --- a/clientapi/routing/register.go +++ b/clientapi/routing/register.go @@ -17,10 +17,7 @@ package routing import ( "context" - "crypto/hmac" - "crypto/sha1" "encoding/json" - "errors" "fmt" "io/ioutil" "net/http" @@ -594,7 +591,6 @@ func handleRegistrationFlow( accessToken string, accessTokenErr error, ) util.JSONResponse { - // TODO: Shared secret registration (create new user scripts) // TODO: Enable registration config flag // TODO: Guest account upgrading @@ -643,20 +639,6 @@ func handleRegistrationFlow( // Add Recaptcha to the list of completed registration stages AddCompletedSessionStage(sessionID, authtypes.LoginTypeRecaptcha) - case authtypes.LoginTypeSharedSecret: - // Check shared secret against config - valid, err := isValidMacLogin(cfg, r.Username, r.Password, r.Admin, r.Auth.Mac) - - if err != nil { - util.GetLogger(req.Context()).WithError(err).Error("isValidMacLogin failed") - return jsonerror.InternalServerError() - } else if !valid { - return util.MessageResponse(http.StatusForbidden, "HMAC incorrect") - } - - // Add SharedSecret to the list of completed registration stages - AddCompletedSessionStage(sessionID, authtypes.LoginTypeSharedSecret) - case authtypes.LoginTypeDummy: // there is nothing to do // Add Dummy to the list of completed registration stages @@ -849,49 +831,6 @@ func completeRegistration( } } -// Used for shared secret registration. -// Checks if the username, password and isAdmin flag matches the given mac. -func isValidMacLogin( - cfg *config.ClientAPI, - username, password string, - isAdmin bool, - givenMac []byte, -) (bool, error) { - sharedSecret := cfg.RegistrationSharedSecret - - // Check that shared secret registration isn't disabled. - if cfg.RegistrationSharedSecret == "" { - return false, errors.New("Shared secret registration is disabled") - } - - // Double check that username/password don't contain the HMAC delimiters. We should have - // already checked this. - if strings.Contains(username, "\x00") { - return false, errors.New("Username contains invalid character") - } - if strings.Contains(password, "\x00") { - return false, errors.New("Password contains invalid character") - } - if sharedSecret == "" { - return false, errors.New("Shared secret registration is disabled") - } - - adminString := "notadmin" - if isAdmin { - adminString = "admin" - } - joined := strings.Join([]string{username, password, adminString}, "\x00") - - mac := hmac.New(sha1.New, []byte(sharedSecret)) - _, err := mac.Write([]byte(joined)) - if err != nil { - return false, err - } - expectedMAC := mac.Sum(nil) - - return hmac.Equal(givenMac, expectedMAC), nil -} - // checkFlows checks a single completed flow against another required one. If // one contains at least all of the stages that the other does, checkFlows // returns true. @@ -995,3 +934,34 @@ func RegisterAvailable( }, } } + +func handleSharedSecretRegistration(userAPI userapi.UserInternalAPI, sr *SharedSecretRegistration, req *http.Request) util.JSONResponse { + ssrr, err := NewSharedSecretRegistrationRequest(req.Body) + if err != nil { + return util.JSONResponse{ + Code: 400, + JSON: jsonerror.BadJSON(fmt.Sprintf("malformed json: %s", err)), + } + } + valid, err := sr.IsValidMacLogin(ssrr.Nonce, ssrr.User, ssrr.Password, ssrr.Admin, ssrr.MacBytes) + if err != nil { + return util.ErrorResponse(err) + } + if !valid { + return util.JSONResponse{ + Code: 403, + JSON: jsonerror.Forbidden("bad mac"), + } + } + // downcase capitals + ssrr.User = strings.ToLower(ssrr.User) + + if resErr := validateUsername(ssrr.User); resErr != nil { + return *resErr + } + if resErr := validatePassword(ssrr.Password); resErr != nil { + return *resErr + } + deviceID := "shared_secret_registration" + return completeRegistration(req.Context(), userAPI, ssrr.User, ssrr.Password, "", req.RemoteAddr, req.UserAgent(), false, &ssrr.User, &deviceID) +} diff --git a/clientapi/routing/register_secret.go b/clientapi/routing/register_secret.go new file mode 100644 index 00000000..f0436e32 --- /dev/null +++ b/clientapi/routing/register_secret.go @@ -0,0 +1,99 @@ +package routing + +import ( + "context" + "crypto/hmac" + "crypto/sha1" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "strings" + "time" + + "github.com/matrix-org/dendrite/internal" + "github.com/matrix-org/util" + cache "github.com/patrickmn/go-cache" +) + +type SharedSecretRegistrationRequest struct { + User string `json:"username"` + Password string `json:"password"` + Nonce string `json:"nonce"` + MacBytes []byte + MacStr string `json:"mac"` + Admin bool `json:"admin"` +} + +func NewSharedSecretRegistrationRequest(reader io.ReadCloser) (*SharedSecretRegistrationRequest, error) { + defer internal.CloseAndLogIfError(context.Background(), reader, "NewSharedSecretRegistrationRequest: failed to close request body") + var ssrr SharedSecretRegistrationRequest + err := json.NewDecoder(reader).Decode(&ssrr) + if err != nil { + return nil, err + } + ssrr.MacBytes, err = hex.DecodeString(ssrr.MacStr) + return &ssrr, err +} + +type SharedSecretRegistration struct { + sharedSecret string + nonces *cache.Cache +} + +func NewSharedSecretRegistration(sharedSecret string) *SharedSecretRegistration { + return &SharedSecretRegistration{ + sharedSecret: sharedSecret, + // nonces live for 5mins, purge every 10mins + nonces: cache.New(5*time.Minute, 10*time.Minute), + } +} + +func (r *SharedSecretRegistration) GenerateNonce() string { + nonce := util.RandomString(16) + r.nonces.Set(nonce, true, cache.DefaultExpiration) + return nonce +} + +func (r *SharedSecretRegistration) validNonce(nonce string) bool { + _, exists := r.nonces.Get(nonce) + return exists +} + +func (r *SharedSecretRegistration) IsValidMacLogin( + nonce, username, password string, + isAdmin bool, + givenMac []byte, +) (bool, error) { + // Check that shared secret registration isn't disabled. + if r.sharedSecret == "" { + return false, errors.New("Shared secret registration is disabled") + } + if !r.validNonce(nonce) { + return false, fmt.Errorf("Incorrect or expired nonce: %s", nonce) + } + + // Check that username/password don't contain the HMAC delimiters. + if strings.Contains(username, "\x00") { + return false, errors.New("Username contains invalid character") + } + if strings.Contains(password, "\x00") { + return false, errors.New("Password contains invalid character") + } + + adminString := "notadmin" + if isAdmin { + adminString = "admin" + } + joined := strings.Join([]string{nonce, username, password, adminString}, "\x00") + + mac := hmac.New(sha1.New, []byte(r.sharedSecret)) + _, err := mac.Write([]byte(joined)) + if err != nil { + return false, err + } + expectedMAC := mac.Sum(nil) + + return hmac.Equal(givenMac, expectedMAC), nil +} diff --git a/clientapi/routing/register_secret_test.go b/clientapi/routing/register_secret_test.go new file mode 100644 index 00000000..e702b215 --- /dev/null +++ b/clientapi/routing/register_secret_test.go @@ -0,0 +1,43 @@ +package routing + +import ( + "bytes" + "io/ioutil" + "testing" + + "github.com/patrickmn/go-cache" +) + +func TestSharedSecretRegister(t *testing.T) { + // these values have come from a local synapse instance to ensure compatibility + jsonStr := []byte(`{"admin":false,"mac":"f1ba8d37123866fd659b40de4bad9b0f8965c565","nonce":"759f047f312b99ff428b21d581256f8592b8976e58bc1b543972dc6147e529a79657605b52d7becd160ff5137f3de11975684319187e06901955f79e5a6c5a79","password":"wonderland","username":"alice"}`) + sharedSecret := "dendritetest" + + req, err := NewSharedSecretRegistrationRequest(ioutil.NopCloser(bytes.NewBuffer(jsonStr))) + if err != nil { + t.Fatalf("failed to read request: %s", err) + } + + r := NewSharedSecretRegistration(sharedSecret) + + // force the nonce to be known + r.nonces.Set(req.Nonce, true, cache.DefaultExpiration) + + valid, err := r.IsValidMacLogin(req.Nonce, req.User, req.Password, req.Admin, req.MacBytes) + if err != nil { + t.Fatalf("failed to check for valid mac: %s", err) + } + if !valid { + t.Errorf("mac login failed, wanted success") + } + + // modify the mac so it fails + req.MacBytes[0] = 0xff + valid, err = r.IsValidMacLogin(req.Nonce, req.User, req.Password, req.Admin, req.MacBytes) + if err != nil { + t.Fatalf("failed to check for valid mac: %s", err) + } + if valid { + t.Errorf("mac login succeeded, wanted failure") + } +} diff --git a/clientapi/routing/routing.go b/clientapi/routing/routing.go index 9f980e0a..37279e8e 100644 --- a/clientapi/routing/routing.go +++ b/clientapi/routing/routing.go @@ -37,6 +37,7 @@ import ( "github.com/matrix-org/dendrite/userapi/storage/accounts" "github.com/matrix-org/gomatrixserverlib" "github.com/matrix-org/util" + "github.com/sirupsen/logrus" ) // Setup registers HTTP handlers with the given ServeMux. It also supplies the given http.Client @@ -46,7 +47,7 @@ import ( // applied: // nolint: gocyclo func Setup( - publicAPIMux *mux.Router, cfg *config.ClientAPI, + publicAPIMux, synapseAdminRouter *mux.Router, cfg *config.ClientAPI, eduAPI eduServerAPI.EDUServerInputAPI, rsAPI roomserverAPI.RoomserverInternalAPI, asAPI appserviceAPI.AppServiceQueryAPI, @@ -88,6 +89,32 @@ func Setup( }), ).Methods(http.MethodGet, http.MethodOptions) + if cfg.RegistrationSharedSecret != "" { + logrus.Info("Enabling shared secret registration at /_synapse/admin/v1/register") + sr := NewSharedSecretRegistration(cfg.RegistrationSharedSecret) + synapseAdminRouter.Handle("/admin/v1/register", + httputil.MakeExternalAPI("shared_secret_registration", func(req *http.Request) util.JSONResponse { + if req.Method == http.MethodGet { + return util.JSONResponse{ + Code: 200, + JSON: struct { + Nonce string `json:"nonce"` + }{ + Nonce: sr.GenerateNonce(), + }, + } + } + if req.Method == http.MethodPost { + return handleSharedSecretRegistration(userAPI, sr, req) + } + return util.JSONResponse{ + Code: http.StatusMethodNotAllowed, + JSON: jsonerror.NotFound("unknown method"), + } + }), + ).Methods(http.MethodGet, http.MethodPost, http.MethodOptions) + } + r0mux := publicAPIMux.PathPrefix("/r0").Subrouter() unstableMux := publicAPIMux.PathPrefix("/unstable").Subrouter() diff --git a/cmd/create-room-events/main.go b/cmd/create-room-events/main.go deleted file mode 100644 index 23b44193..00000000 --- a/cmd/create-room-events/main.go +++ /dev/null @@ -1,147 +0,0 @@ -// Copyright 2017 Vector Creations Ltd -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// Generate a list of matrix room events for load testing. -// Writes the events to stdout by default. -package main - -import ( - "encoding/base64" - "encoding/json" - "flag" - "fmt" - "os" - "strings" - "time" - - "github.com/matrix-org/dendrite/roomserver/api" - "github.com/matrix-org/gomatrixserverlib" - "golang.org/x/crypto/ed25519" -) - -const usage = `Usage: %s - -Generate a list of matrix room events for load testing. -Writes the events to stdout separated by new lines - -Arguments: - -` - -var ( - serverName = flag.String("server-name", "localhost", "The name of the matrix server to generate events for") - keyID = flag.String("key-id", "ed25519:auto", "The ID of the key used to sign the events") - privateKeyString = flag.String("private-key", defaultKey, "Base64 encoded private key to sign events with") - roomID = flag.String("room-id", "!roomid:$SERVER_NAME", "The room ID to generate events in") - userID = flag.String("user-id", "@userid:$SERVER_NAME", "The user ID to use as the event sender") - messageCount = flag.Int("message-count", 10, "The number of m.room.messsage events to generate") - format = flag.String("Format", "InputRoomEvent", "The output format to use for the messages: InputRoomEvent or Event") - ver = flag.String("version", string(gomatrixserverlib.RoomVersionV1), "Room version to generate events as") -) - -// By default we use a private key of 0. -const defaultKey = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" - -var privateKey ed25519.PrivateKey -var emptyString = "" -var now time.Time -var b gomatrixserverlib.EventBuilder -var eventID int - -func main() { - flag.Usage = func() { - fmt.Fprintf(os.Stderr, usage, os.Args[0]) - flag.PrintDefaults() - } - - flag.Parse() - *userID = strings.Replace(*userID, "$SERVER_NAME", *serverName, 1) - *roomID = strings.Replace(*roomID, "$SERVER_NAME", *serverName, 1) - - // Decode the ed25519 private key. - privateKeyBytes, err := base64.RawStdEncoding.DecodeString(*privateKeyString) - if err != nil { - panic(err) - } - privateKey = ed25519.PrivateKey(privateKeyBytes) - - // Build a m.room.create event. - b.Sender = *userID - b.RoomID = *roomID - b.Type = "m.room.create" - b.StateKey = &emptyString - b.SetContent(map[string]string{"creator": *userID}) // nolint: errcheck - create := buildAndOutput() - - // Build a m.room.member event. - b.Type = "m.room.member" - b.StateKey = userID - b.SetContent(map[string]string{"membership": gomatrixserverlib.Join}) // nolint: errcheck - b.AuthEvents = []gomatrixserverlib.EventReference{create} - member := buildAndOutput() - - // Build a number of m.room.message events. - b.Type = "m.room.message" - b.StateKey = nil - b.SetContent(map[string]string{"body": "Test Message"}) // nolint: errcheck - b.AuthEvents = []gomatrixserverlib.EventReference{create, member} - for i := 0; i < *messageCount; i++ { - buildAndOutput() - } -} - -// Build an event and write the event to the output. -func buildAndOutput() gomatrixserverlib.EventReference { - eventID++ - now = time.Unix(0, 0) - name := gomatrixserverlib.ServerName(*serverName) - key := gomatrixserverlib.KeyID(*keyID) - - event, err := b.Build( - now, name, key, privateKey, - gomatrixserverlib.RoomVersion(*ver), - ) - if err != nil { - panic(err) - } - writeEvent(event) - reference := event.EventReference() - b.PrevEvents = []gomatrixserverlib.EventReference{reference} - b.Depth++ - return reference -} - -// Write an event to the output. -func writeEvent(event *gomatrixserverlib.Event) { - encoder := json.NewEncoder(os.Stdout) - if *format == "InputRoomEvent" { - var ire api.InputRoomEvent - ire.Kind = api.KindNew - ire.Event = event.Headered(gomatrixserverlib.RoomVersion(*ver)) - authEventIDs := []string{} - for _, ref := range b.AuthEvents.([]gomatrixserverlib.EventReference) { - authEventIDs = append(authEventIDs, ref.EventID) - } - ire.AuthEventIDs = authEventIDs - if err := encoder.Encode(ire); err != nil { - panic(err) - } - } else if *format == "Event" { - if err := encoder.Encode(event); err != nil { - panic(err) - } - } else { - panic(fmt.Errorf("Format %q is not valid, must be %q or %q", *format, "InputRoomEvent", "Event")) - } -} diff --git a/cmd/dendrite-demo-libp2p/main.go b/cmd/dendrite-demo-libp2p/main.go index cc7dcf02..6b0e57d8 100644 --- a/cmd/dendrite-demo-libp2p/main.go +++ b/cmd/dendrite-demo-libp2p/main.go @@ -197,6 +197,7 @@ func main() { base.Base.PublicFederationAPIMux, base.Base.PublicKeyAPIMux, base.Base.PublicMediaAPIMux, + base.Base.SynapseAdminMux, ) if err := mscs.Enable(&base.Base, &monolith); err != nil { logrus.WithError(err).Fatalf("Failed to enable MSCs") diff --git a/cmd/dendrite-demo-pinecone/main.go b/cmd/dendrite-demo-pinecone/main.go index 72936e42..2712ed4a 100644 --- a/cmd/dendrite-demo-pinecone/main.go +++ b/cmd/dendrite-demo-pinecone/main.go @@ -210,6 +210,7 @@ func main() { base.PublicFederationAPIMux, base.PublicKeyAPIMux, base.PublicMediaAPIMux, + base.SynapseAdminMux, ) wsUpgrader := websocket.Upgrader{ diff --git a/cmd/dendrite-demo-yggdrasil/convert/25519.go b/cmd/dendrite-demo-yggdrasil/convert/25519.go deleted file mode 100644 index 97f053ec..00000000 --- a/cmd/dendrite-demo-yggdrasil/convert/25519.go +++ /dev/null @@ -1,53 +0,0 @@ -// Copyright 2019 Google LLC -// -// Use of this source code is governed by a BSD-style -// license that can be found in the LICENSE file or at -// https://developers.google.com/open-source/licenses/bsd -// -// Original code from https://github.com/FiloSottile/age/blob/bbab440e198a4d67ba78591176c7853e62d29e04/internal/age/ssh.go - -package convert - -import ( - "crypto/ed25519" - "crypto/sha512" - "math/big" - - "golang.org/x/crypto/curve25519" -) - -var curve25519P, _ = new(big.Int).SetString("57896044618658097711785492504343953926634992332820282019728792003956564819949", 10) - -func Ed25519PrivateKeyToCurve25519(pk ed25519.PrivateKey) []byte { - h := sha512.New() - _, _ = h.Write(pk.Seed()) - out := h.Sum(nil) - return out[:curve25519.ScalarSize] -} - -func Ed25519PublicKeyToCurve25519(pk ed25519.PublicKey) []byte { - // ed25519.PublicKey is a little endian representation of the y-coordinate, - // with the most significant bit set based on the sign of the x-coordinate. - bigEndianY := make([]byte, ed25519.PublicKeySize) - for i, b := range pk { - bigEndianY[ed25519.PublicKeySize-i-1] = b - } - bigEndianY[0] &= 0b0111_1111 - - // The Montgomery u-coordinate is derived through the bilinear map - // u = (1 + y) / (1 - y) - // See https://blog.filippo.io/using-ed25519-keys-for-encryption. - y := new(big.Int).SetBytes(bigEndianY) - denom := big.NewInt(1) - denom.ModInverse(denom.Sub(denom, y), curve25519P) // 1 / (1 - y) - u := y.Mul(y.Add(y, big.NewInt(1)), denom) - u.Mod(u, curve25519P) - - out := make([]byte, curve25519.PointSize) - uBytes := u.Bytes() - for i, b := range uBytes { - out[len(uBytes)-i-1] = b - } - - return out -} diff --git a/cmd/dendrite-demo-yggdrasil/convert/25519_test.go b/cmd/dendrite-demo-yggdrasil/convert/25519_test.go deleted file mode 100644 index 22177b8b..00000000 --- a/cmd/dendrite-demo-yggdrasil/convert/25519_test.go +++ /dev/null @@ -1,51 +0,0 @@ -// Copyright 2020 The Matrix.org Foundation C.I.C. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package convert - -import ( - "bytes" - "crypto/ed25519" - "encoding/hex" - "testing" - - "golang.org/x/crypto/curve25519" -) - -func TestKeyConversion(t *testing.T) { - edPub, edPriv, err := ed25519.GenerateKey(nil) - if err != nil { - t.Fatal(err) - } - t.Log("Signing public:", hex.EncodeToString(edPub)) - t.Log("Signing private:", hex.EncodeToString(edPriv)) - - cuPriv := Ed25519PrivateKeyToCurve25519(edPriv) - t.Log("Encryption private:", hex.EncodeToString(cuPriv)) - - cuPub := Ed25519PublicKeyToCurve25519(edPub) - t.Log("Converted encryption public:", hex.EncodeToString(cuPub)) - - var realPub, realPriv [32]byte - copy(realPriv[:32], cuPriv[:32]) - curve25519.ScalarBaseMult(&realPub, &realPriv) - t.Log("Scalar-multed encryption public:", hex.EncodeToString(realPub[:])) - - if !bytes.Equal(realPriv[:], cuPriv[:]) { - t.Fatal("Private keys should be equal (this means the test is broken)") - } - if !bytes.Equal(realPub[:], cuPub[:]) { - t.Fatal("Public keys should be equal") - } -} diff --git a/cmd/dendrite-demo-yggdrasil/main.go b/cmd/dendrite-demo-yggdrasil/main.go index 2d710ae7..95207a60 100644 --- a/cmd/dendrite-demo-yggdrasil/main.go +++ b/cmd/dendrite-demo-yggdrasil/main.go @@ -56,21 +56,23 @@ func main() { flag.Parse() internal.SetupPprof() - ygg, err := yggconn.Setup(*instanceName, ".") + ygg, err := yggconn.Setup(*instanceName, ".", *instancePeer) if err != nil { panic(err) } - ygg.SetMulticastEnabled(true) - if instancePeer != nil && *instancePeer != "" { - if err = ygg.SetStaticPeer(*instancePeer); err != nil { - logrus.WithError(err).Error("Failed to set static peer") + /* + ygg.SetMulticastEnabled(true) + if instancePeer != nil && *instancePeer != "" { + if err = ygg.SetStaticPeer(*instancePeer); err != nil { + logrus.WithError(err).Error("Failed to set static peer") + } } - } + */ cfg := &config.Dendrite{} cfg.Defaults() cfg.Global.ServerName = gomatrixserverlib.ServerName(ygg.DerivedServerName()) - cfg.Global.PrivateKey = ygg.SigningPrivateKey() + cfg.Global.PrivateKey = ygg.PrivateKey() cfg.Global.KeyID = gomatrixserverlib.KeyID(signing.KeyID) cfg.Global.Kafka.UseNaffka = true cfg.UserAPI.AccountDatabase.ConnectionString = config.DataSource(fmt.Sprintf("file:%s-account.db", *instanceName)) @@ -117,18 +119,6 @@ func main() { base, federation, rsAPI, keyRing, true, ) - ygg.SetSessionFunc(func(address string) { - req := &api.PerformServersAliveRequest{ - Servers: []gomatrixserverlib.ServerName{ - gomatrixserverlib.ServerName(address), - }, - } - res := &api.PerformServersAliveResponse{} - if err := fsAPI.PerformServersAlive(context.TODO(), req, res); err != nil { - logrus.WithError(err).Error("Failed to send wake-up message to newly connected node") - } - }) - rsComponent.SetFederationSenderAPI(fsAPI) monolith := setup.Monolith{ @@ -154,6 +144,7 @@ func main() { base.PublicFederationAPIMux, base.PublicKeyAPIMux, base.PublicMediaAPIMux, + base.SynapseAdminMux, ) if err := mscs.Enable(base, &monolith); err != nil { logrus.WithError(err).Fatalf("Failed to enable MSCs") diff --git a/cmd/dendrite-demo-yggdrasil/yggconn/client.go b/cmd/dendrite-demo-yggdrasil/yggconn/client.go index 157a9bf2..c7409c21 100644 --- a/cmd/dendrite-demo-yggdrasil/yggconn/client.go +++ b/cmd/dendrite-demo-yggdrasil/yggconn/client.go @@ -51,7 +51,6 @@ func (n *Node) CreateFederationClient( ResponseHeaderTimeout: 10 * time.Second, IdleConnTimeout: 30 * time.Second, DialContext: n.DialerContext, - TLSClientConfig: n.tlsConfig, }, }, ) diff --git a/cmd/dendrite-demo-yggdrasil/yggconn/node.go b/cmd/dendrite-demo-yggdrasil/yggconn/node.go index 9c286dfb..002e8071 100644 --- a/cmd/dendrite-demo-yggdrasil/yggconn/node.go +++ b/cmd/dendrite-demo-yggdrasil/yggconn/node.go @@ -17,7 +17,6 @@ package yggconn import ( "context" "crypto/ed25519" - "crypto/tls" "encoding/hex" "encoding/json" "fmt" @@ -26,60 +25,48 @@ import ( "net" "os" "strings" - "sync" - "time" "github.com/lucas-clemente/quic-go" - "github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/convert" "github.com/matrix-org/gomatrixserverlib" - "go.uber.org/atomic" + "github.com/neilalexander/utp" + ironwoodtypes "github.com/Arceliar/ironwood/types" yggdrasilconfig "github.com/yggdrasil-network/yggdrasil-go/src/config" + yggdrasilcore "github.com/yggdrasil-network/yggdrasil-go/src/core" + yggdrasildefaults "github.com/yggdrasil-network/yggdrasil-go/src/defaults" yggdrasilmulticast "github.com/yggdrasil-network/yggdrasil-go/src/multicast" - "github.com/yggdrasil-network/yggdrasil-go/src/yggdrasil" gologme "github.com/gologme/log" ) type Node struct { - core *yggdrasil.Core - config *yggdrasilconfig.NodeConfig - state *yggdrasilconfig.NodeState - multicast *yggdrasilmulticast.Multicast - log *gologme.Logger - listener quic.Listener - tlsConfig *tls.Config - quicConfig *quic.Config - sessions sync.Map // string -> *session - sessionCount atomic.Uint32 - sessionFunc func(address string) - coords sync.Map // string -> yggdrasil.Coords - incoming chan QUICStream - NewSession func(remote gomatrixserverlib.ServerName) + core *yggdrasilcore.Core + config *yggdrasilconfig.NodeConfig + multicast *yggdrasilmulticast.Multicast + log *gologme.Logger + listener quic.Listener + utpSocket *utp.Socket + incoming chan net.Conn } -func (n *Node) Dialer(_, address string) (net.Conn, error) { +func (n *Node) DialerContext(ctx context.Context, _, address string) (net.Conn, error) { tokens := strings.Split(address, ":") raw, err := hex.DecodeString(tokens[0]) if err != nil { return nil, fmt.Errorf("hex.DecodeString: %w", err) } - converted := convert.Ed25519PublicKeyToCurve25519(ed25519.PublicKey(raw)) - convhex := hex.EncodeToString(converted) - return n.Dial("curve25519", convhex) + pk := make(ironwoodtypes.Addr, ed25519.PublicKeySize) + copy(pk, raw[:]) + return n.utpSocket.DialAddrContext(ctx, pk) } -func (n *Node) DialerContext(ctx context.Context, network, address string) (net.Conn, error) { - return n.Dialer(network, address) -} - -func Setup(instanceName, storageDirectory string) (*Node, error) { +func Setup(instanceName, storageDirectory, peerURI string) (*Node, error) { n := &Node{ - core: &yggdrasil.Core{}, - config: yggdrasilconfig.GenerateConfig(), + core: &yggdrasilcore.Core{}, + config: yggdrasildefaults.GenerateConfig(), multicast: &yggdrasilmulticast.Multicast{}, log: gologme.New(os.Stdout, "YGG ", log.Flags()), - incoming: make(chan QUICStream), + incoming: make(chan net.Conn), } yggfile := fmt.Sprintf("%s/%s-yggdrasil.conf", storageDirectory, instanceName) @@ -93,24 +80,11 @@ func Setup(instanceName, storageDirectory string) (*Node, error) { } } - n.core.SetCoordChangeCallback(func(old, new yggdrasil.Coords) { - fmt.Println("COORDINATE CHANGE!") - fmt.Println("Old:", old) - fmt.Println("New:", new) - n.sessions.Range(func(k, v interface{}) bool { - if s, ok := v.(*session); ok { - fmt.Println("Killing session", k) - s.kill() - } - return true - }) - }) - n.config.Peers = []string{} + if peerURI != "" { + n.config.Peers = append(n.config.Peers, peerURI) + } n.config.AdminListen = "none" - n.config.MulticastInterfaces = []string{} - n.config.EncryptionPrivateKey = hex.EncodeToString(n.EncryptionPrivateKey()) - n.config.EncryptionPublicKey = hex.EncodeToString(n.EncryptionPublicKey()) j, err := json.MarshalIndent(n.config, "", " ") if err != nil { @@ -123,34 +97,22 @@ func Setup(instanceName, storageDirectory string) (*Node, error) { n.log.EnableLevel("error") n.log.EnableLevel("warn") n.log.EnableLevel("info") - n.state, err = n.core.Start(n.config, n.log) + if err = n.core.Start(n.config, n.log); err != nil { + panic(err) + } + n.utpSocket, err = utp.NewSocketFromPacketConnNoClose(n.core) if err != nil { panic(err) } - if err = n.multicast.Init(n.core, n.state, n.log, nil); err != nil { + if err = n.multicast.Init(n.core, n.config, n.log, nil); err != nil { panic(err) } if err = n.multicast.Start(); err != nil { panic(err) } - n.tlsConfig = n.generateTLSConfig() - n.quicConfig = &quic.Config{ - MaxIncomingStreams: 0, - MaxIncomingUniStreams: 0, - KeepAlive: true, - MaxIdleTimeout: time.Minute * 30, - HandshakeTimeout: time.Second * 15, - } - copy(n.quicConfig.StatelessResetKey, n.EncryptionPublicKey()) - - n.log.Println("Public curve25519:", n.core.EncryptionPublicKey()) - n.log.Println("Public ed25519:", n.core.SigningPublicKey()) - - go func() { - time.Sleep(time.Second) - n.listenFromYgg() - }() + n.log.Println("Public key:", n.core.PublicKey()) + go n.listenFromYgg() return n, nil } @@ -163,64 +125,33 @@ func (n *Node) Stop() { } func (n *Node) DerivedServerName() string { - return hex.EncodeToString(n.SigningPublicKey()) + return hex.EncodeToString(n.PublicKey()) } -func (n *Node) DerivedSessionName() string { - return hex.EncodeToString(n.EncryptionPublicKey()) +func (n *Node) PrivateKey() ed25519.PrivateKey { + sk := make(ed25519.PrivateKey, ed25519.PrivateKeySize) + sb, err := hex.DecodeString(n.config.PrivateKey) + if err == nil { + copy(sk, sb[:]) + } else { + panic(err) + } + return sk } -func (n *Node) EncryptionPublicKey() []byte { - edkey := n.SigningPublicKey() - return convert.Ed25519PublicKeyToCurve25519(edkey) -} - -func (n *Node) EncryptionPrivateKey() []byte { - edkey := n.SigningPrivateKey() - return convert.Ed25519PrivateKeyToCurve25519(edkey) -} - -func (n *Node) SigningPublicKey() ed25519.PublicKey { - pubBytes, _ := hex.DecodeString(n.config.SigningPublicKey) - return ed25519.PublicKey(pubBytes) -} - -func (n *Node) SigningPrivateKey() ed25519.PrivateKey { - privBytes, _ := hex.DecodeString(n.config.SigningPrivateKey) - return ed25519.PrivateKey(privBytes) -} - -func (n *Node) SetSessionFunc(f func(address string)) { - n.sessionFunc = f +func (n *Node) PublicKey() ed25519.PublicKey { + return n.core.PublicKey() } func (n *Node) PeerCount() int { - return len(n.core.GetPeers()) - 1 -} - -func (n *Node) SessionCount() int { - return int(n.sessionCount.Load()) + return len(n.core.GetPeers()) } func (n *Node) KnownNodes() []gomatrixserverlib.ServerName { - nodemap := map[string]struct{}{ - //"b5ae50589e50991dd9dd7d59c5c5f7a4521e8da5b603b7f57076272abc58b374": {}, + nodemap := map[string]struct{}{} + for _, peer := range n.core.GetPeers() { + nodemap[hex.EncodeToString(peer.Key)] = struct{}{} } - for _, peer := range n.core.GetSwitchPeers() { - nodemap[hex.EncodeToString(peer.SigPublicKey[:])] = struct{}{} - } - n.sessions.Range(func(_, v interface{}) bool { - session, ok := v.(quic.Session) - if !ok { - return true - } - if len(session.ConnectionState().PeerCertificates) != 1 { - return true - } - subjectName := session.ConnectionState().PeerCertificates[0].Subject.CommonName - nodemap[subjectName] = struct{}{} - return true - }) var nodes []gomatrixserverlib.ServerName for node := range nodemap { nodes = append(nodes, gomatrixserverlib.ServerName(node)) @@ -229,53 +160,22 @@ func (n *Node) KnownNodes() []gomatrixserverlib.ServerName { } func (n *Node) SetMulticastEnabled(enabled bool) { - if enabled { - n.config.MulticastInterfaces = []string{".*"} - } else { - n.config.MulticastInterfaces = []string{} - } - n.multicast.UpdateConfig(n.config) - if !enabled { - n.DisconnectMulticastPeers() - } + // TODO: There's no dynamic reconfiguration in Yggdrasil v0.4 + // so we need a solution for this. } func (n *Node) DisconnectMulticastPeers() { - for _, sp := range n.core.GetSwitchPeers() { - if !strings.HasPrefix(sp.Endpoint, "fe80") { - continue - } - if err := n.core.DisconnectPeer(sp.Port); err != nil { - n.log.Printf("Failed to disconnect port %d: %s", sp.Port, err) - } - } + // TODO: There's no dynamic reconfiguration in Yggdrasil v0.4 + // so we need a solution for this. } func (n *Node) DisconnectNonMulticastPeers() { - for _, sp := range n.core.GetSwitchPeers() { - if strings.HasPrefix(sp.Endpoint, "fe80") { - continue - } - if err := n.core.DisconnectPeer(sp.Port); err != nil { - n.log.Printf("Failed to disconnect port %d: %s", sp.Port, err) - } - } + // TODO: There's no dynamic reconfiguration in Yggdrasil v0.4 + // so we need a solution for this. } func (n *Node) SetStaticPeer(uri string) error { - n.config.Peers = []string{} - n.core.UpdateConfig(n.config) - n.DisconnectNonMulticastPeers() - if uri != "" { - n.log.Infoln("Adding static peer", uri) - if err := n.core.AddPeer(uri, ""); err != nil { - n.log.Warnln("Adding static peer failed:", err) - return err - } - if err := n.core.CallPeer(uri, ""); err != nil { - n.log.Warnln("Calling static peer failed:", err) - return err - } - } + // TODO: There's no dynamic reconfiguration in Yggdrasil v0.4 + // so we need a solution for this. return nil } diff --git a/cmd/dendrite-demo-yggdrasil/yggconn/session.go b/cmd/dendrite-demo-yggdrasil/yggconn/session.go index 7b56e736..b9a523dd 100644 --- a/cmd/dendrite-demo-yggdrasil/yggconn/session.go +++ b/cmd/dendrite-demo-yggdrasil/yggconn/session.go @@ -16,94 +16,17 @@ package yggconn import ( "context" - "crypto/rand" - "crypto/rsa" - "crypto/tls" - "crypto/x509" - "crypto/x509/pkix" - "encoding/hex" - "encoding/pem" - "errors" - "fmt" - "math/big" "net" - "time" - - "github.com/lucas-clemente/quic-go" - "github.com/yggdrasil-network/yggdrasil-go/src/crypto" - "github.com/yggdrasil-network/yggdrasil-go/src/yggdrasil" ) -type session struct { - node *Node - session quic.Session - address string - context context.Context - cancel context.CancelFunc -} - -func (n *Node) newSession(sess quic.Session, address string) *session { - ctx, cancel := context.WithCancel(context.TODO()) - return &session{ - node: n, - session: sess, - address: address, - context: ctx, - cancel: cancel, - } -} - -func (s *session) kill() { - s.cancel() -} - func (n *Node) listenFromYgg() { - var err error - n.listener, err = quic.Listen( - n.core, // yggdrasil.PacketConn - n.tlsConfig, // TLS config - n.quicConfig, // QUIC config - ) - if err != nil { - panic(err) - } - for { - n.log.Infoln("Waiting to accept QUIC sessions") - session, err := n.listener.Accept(context.TODO()) + conn, err := n.utpSocket.Accept() if err != nil { - n.log.Println("n.listener.Accept:", err) + n.log.Println("n.utpSocket.Accept:", err) return } - if len(session.ConnectionState().PeerCertificates) != 1 { - _ = session.CloseWithError(0, "expected a peer certificate") - continue - } - address := session.ConnectionState().PeerCertificates[0].DNSNames[0] - n.log.Infoln("Accepted connection from", address) - go n.newSession(session, address).listenFromQUIC() - go n.sessionFunc(address) - } -} - -func (s *session) listenFromQUIC() { - if existing, ok := s.node.sessions.Load(s.address); ok { - if existingSession, ok := existing.(*session); ok { - fmt.Println("Killing existing session to replace", s.address) - existingSession.kill() - } - } - s.node.sessionCount.Inc() - s.node.sessions.Store(s.address, s) - defer s.node.sessions.Delete(s.address) - defer s.node.sessionCount.Dec() - for { - st, err := s.session.AcceptStream(s.context) - if err != nil { - s.node.log.Println("session.AcceptStream:", err) - return - } - s.node.incoming <- QUICStream{st, s.session} + n.incoming <- conn } } @@ -129,155 +52,5 @@ func (n *Node) Dial(network, address string) (net.Conn, error) { // Implements http.Transport.DialContext func (n *Node) DialContext(ctx context.Context, network, address string) (net.Conn, error) { - s, ok1 := n.sessions.Load(address) - session, ok2 := s.(*session) - if !ok1 || !ok2 { - // First of all, check if we think we know the coords of this - // node. If we do then we'll try to dial to it directly. This - // will either succeed or fail. - if v, ok := n.coords.Load(address); ok { - coords, ok := v.(yggdrasil.Coords) - if !ok { - n.coords.Delete(address) - return nil, errors.New("should have found yggdrasil.Coords but didn't") - } - n.log.Infof("Coords %s for %q cached, trying to dial", coords.String(), address) - var err error - // We think we know the coords. Try to dial the node. - if session, err = n.tryDial(address, coords); err != nil { - // We thought we knew the coords but it didn't result - // in a successful dial. Nuke them from the cache. - n.coords.Delete(address) - n.log.Infof("Cached coords %s for %q failed", coords.String(), address) - } - } - - // We either don't know the coords for the node, or we failed - // to dial it before, in which case try to resolve the coords. - if _, ok := n.coords.Load(address); !ok { - var coords yggdrasil.Coords - var err error - - // First look and see if the node is something that we already - // know about from our direct switch peers. - for _, peer := range n.core.GetSwitchPeers() { - if peer.PublicKey.String() == address { - coords = peer.Coords - n.log.Infof("%q is a direct peer, coords are %s", address, coords.String()) - n.coords.Store(address, coords) - break - } - } - - // If it isn' a node that we know directly then try to search - // the network. - if coords == nil { - n.log.Infof("Searching for coords for %q", address) - dest, derr := hex.DecodeString(address) - if derr != nil { - return nil, derr - } - if len(dest) != crypto.BoxPubKeyLen { - return nil, errors.New("invalid key length supplied") - } - var pubKey crypto.BoxPubKey - copy(pubKey[:], dest) - nodeID := crypto.GetNodeID(&pubKey) - nodeMask := &crypto.NodeID{} - for i := range nodeMask { - nodeMask[i] = 0xFF - } - - fmt.Println("Resolving coords") - coords, err = n.core.Resolve(nodeID, nodeMask) - if err != nil { - return nil, fmt.Errorf("n.core.Resolve: %w", err) - } - fmt.Println("Found coords:", coords) - n.coords.Store(address, coords) - } - - // We now know the coords in theory. Let's try dialling the - // node again. - if session, err = n.tryDial(address, coords); err != nil { - return nil, fmt.Errorf("n.tryDial: %w", err) - } - } - } - - if session == nil { - return nil, fmt.Errorf("should have found session but didn't") - } - - st, err := session.session.OpenStream() - if err != nil { - n.log.Println("session.OpenStream:", err) - _ = session.session.CloseWithError(0, "expected to be able to open session") - return nil, err - } - return QUICStream{st, session.session}, nil -} - -func (n *Node) tryDial(address string, coords yggdrasil.Coords) (*session, error) { - quicSession, err := quic.Dial( - n.core, // yggdrasil.PacketConn - coords, // dial address - address, // dial SNI - n.tlsConfig, // TLS config - n.quicConfig, // QUIC config - ) - if err != nil { - return nil, err - } - if len(quicSession.ConnectionState().PeerCertificates) != 1 { - _ = quicSession.CloseWithError(0, "expected a peer certificate") - return nil, errors.New("didn't receive a peer certificate") - } - if len(quicSession.ConnectionState().PeerCertificates[0].DNSNames) != 1 { - _ = quicSession.CloseWithError(0, "expected a DNS name") - return nil, errors.New("didn't receive a DNS name") - } - if gotAddress := quicSession.ConnectionState().PeerCertificates[0].DNSNames[0]; address != gotAddress { - _ = quicSession.CloseWithError(0, "you aren't the host I was hoping for") - return nil, fmt.Errorf("expected %q but dialled %q", address, gotAddress) - } - session := n.newSession(quicSession, address) - go session.listenFromQUIC() - go n.sessionFunc(address) - return session, nil -} - -func (n *Node) generateTLSConfig() *tls.Config { - key, err := rsa.GenerateKey(rand.Reader, 1024) - if err != nil { - panic(err) - } - template := x509.Certificate{ - Subject: pkix.Name{ - CommonName: n.DerivedServerName(), - }, - SerialNumber: big.NewInt(1), - NotAfter: time.Now().Add(time.Hour * 24 * 365), - DNSNames: []string{n.DerivedSessionName()}, - } - certDER, err := x509.CreateCertificate(rand.Reader, &template, &template, &key.PublicKey, key) - if err != nil { - panic(err) - } - keyPEM := pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(key)}) - certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER}) - - tlsCert, err := tls.X509KeyPair(certPEM, keyPEM) - if err != nil { - panic(err) - } - return &tls.Config{ - Certificates: []tls.Certificate{tlsCert}, - NextProtos: []string{"quic-matrix-ygg"}, - InsecureSkipVerify: true, - ClientAuth: tls.RequireAnyClientCert, - GetClientCertificate: func(info *tls.CertificateRequestInfo) (*tls.Certificate, error) { - return &tlsCert, nil - }, - } + return n.utpSocket.DialContext(ctx, network, address) } diff --git a/cmd/dendrite-demo-yggdrasil/yggconn/stream.go b/cmd/dendrite-demo-yggdrasil/yggconn/stream.go deleted file mode 100644 index dac7447e..00000000 --- a/cmd/dendrite-demo-yggdrasil/yggconn/stream.go +++ /dev/null @@ -1,20 +0,0 @@ -package yggconn - -import ( - "net" - - "github.com/lucas-clemente/quic-go" -) - -type QUICStream struct { - quic.Stream - session quic.Session -} - -func (s QUICStream) LocalAddr() net.Addr { - return s.session.LocalAddr() -} - -func (s QUICStream) RemoteAddr() net.Addr { - return s.session.RemoteAddr() -} diff --git a/cmd/dendrite-monolith-server/main.go b/cmd/dendrite-monolith-server/main.go index ef349505..5efbe856 100644 --- a/cmd/dendrite-monolith-server/main.go +++ b/cmd/dendrite-monolith-server/main.go @@ -149,6 +149,7 @@ func main() { base.PublicFederationAPIMux, base.PublicKeyAPIMux, base.PublicMediaAPIMux, + base.SynapseAdminMux, ) if len(base.Cfg.MSCs.MSCs) > 0 { diff --git a/cmd/dendrite-polylith-multi/personalities/clientapi.go b/cmd/dendrite-polylith-multi/personalities/clientapi.go index ec445ceb..5e0c4354 100644 --- a/cmd/dendrite-polylith-multi/personalities/clientapi.go +++ b/cmd/dendrite-polylith-multi/personalities/clientapi.go @@ -33,7 +33,7 @@ func ClientAPI(base *setup.BaseDendrite, cfg *config.Dendrite) { keyAPI := base.KeyServerHTTPClient() clientapi.AddPublicRoutes( - base.PublicClientAPIMux, &base.Cfg.ClientAPI, accountDB, federation, + base.PublicClientAPIMux, base.SynapseAdminMux, &base.Cfg.ClientAPI, accountDB, federation, rsAPI, eduInputAPI, asQuery, transactions.New(), fsAPI, userAPI, keyAPI, nil, &cfg.MSCs, ) diff --git a/cmd/dendritejs-pinecone/main.go b/cmd/dendritejs-pinecone/main.go index 433e9bf8..25e49690 100644 --- a/cmd/dendritejs-pinecone/main.go +++ b/cmd/dendritejs-pinecone/main.go @@ -215,6 +215,7 @@ func main() { base.PublicFederationAPIMux, base.PublicKeyAPIMux, base.PublicMediaAPIMux, + base.SynapseAdminMux, ) httpRouter := mux.NewRouter().SkipClean(true).UseEncodedPath() diff --git a/cmd/dendritejs/main.go b/cmd/dendritejs/main.go index 7ece94ff..d5a845ae 100644 --- a/cmd/dendritejs/main.go +++ b/cmd/dendritejs/main.go @@ -236,6 +236,7 @@ func main() { base.PublicFederationAPIMux, base.PublicKeyAPIMux, base.PublicMediaAPIMux, + base.SynapseAdminMux, ) httpRouter := mux.NewRouter().SkipClean(true).UseEncodedPath() diff --git a/cmd/kafka-producer/main.go b/cmd/kafka-producer/main.go deleted file mode 100644 index 18ee3cdf..00000000 --- a/cmd/kafka-producer/main.go +++ /dev/null @@ -1,93 +0,0 @@ -// Copyright 2017 Vector Creations Ltd -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package main - -import ( - "bufio" - "flag" - "fmt" - "os" - "strings" - - sarama "github.com/Shopify/sarama" -) - -const usage = `Usage: %s - -Reads a list of newline separated messages from stdin and writes them to a single partition in kafka. - -Arguments: - -` - -var ( - brokerList = flag.String("brokers", os.Getenv("KAFKA_PEERS"), "The comma separated list of brokers in the Kafka cluster. You can also set the KAFKA_PEERS environment variable") - topic = flag.String("topic", "", "REQUIRED: the topic to produce to") - partition = flag.Int("partition", 0, "The partition to produce to. All the messages will be written to this partition.") -) - -func main() { - flag.Usage = func() { - fmt.Fprintf(os.Stderr, usage, os.Args[0]) - flag.PrintDefaults() - } - - flag.Parse() - - if *brokerList == "" { - fmt.Fprintln(os.Stderr, "no -brokers specified. Alternatively, set the KAFKA_PEERS environment variable") - os.Exit(1) - } - - if *topic == "" { - fmt.Fprintln(os.Stderr, "no -topic specified") - os.Exit(1) - } - - config := sarama.NewConfig() - config.Producer.RequiredAcks = sarama.WaitForAll - config.Producer.Return.Successes = true - config.Producer.Partitioner = sarama.NewManualPartitioner - - producer, err := sarama.NewSyncProducer(strings.Split(*brokerList, ","), config) - if err != nil { - fmt.Fprintln(os.Stderr, "Failed to open Kafka producer:", err) - os.Exit(1) - } - defer func() { - if err := producer.Close(); err != nil { - fmt.Fprintln(os.Stderr, "Failed to close Kafka producer cleanly:", err) - } - }() - - scanner := bufio.NewScanner(os.Stdin) - for scanner.Scan() { - line := scanner.Bytes() - message := &sarama.ProducerMessage{ - Topic: *topic, - Partition: int32(*partition), - Value: sarama.ByteEncoder(line), - } - if _, _, err := producer.SendMessage(message); err != nil { - fmt.Fprintln(os.Stderr, "Failed to send message:", err) - os.Exit(1) - } - - } - if err := scanner.Err(); err != nil { - fmt.Fprintln(os.Stderr, "reading standard input:", err) - } - -} diff --git a/cmd/mediaapi-integration-tests/TESTS.md b/cmd/mediaapi-integration-tests/TESTS.md deleted file mode 100644 index 82777f45..00000000 --- a/cmd/mediaapi-integration-tests/TESTS.md +++ /dev/null @@ -1,69 +0,0 @@ -# Media API Tests - -## Implemented - -* functional - * upload - * normal case - * download - * local file - * existing - * non-existing - * remote file - * existing - * thumbnail - * original file formats - * JPEG - * local file - * existing - * remote file - * existing - * cache - * cold - * hot - * pre-generation according to configuration - * scale - * crop - * dynamic generation - * cold cache - * larger than original - * scale - -## TODO - -* functional - * upload - * file too large - * 0-byte file? - * invalid filename - * invalid content-type - * download - * invalid origin - * invalid media id - * thumbnail - * original file formats - * GIF - * PNG - * BMP - * SVG - * PDF - * TIFF - * WEBP - * local file - * non-existing - * remote file - * non-existing - * pre-generation according to configuration - * manual verification + hash check for regressions? - * dynamic generation - * hot cache - * limit on dimensions? - * 0x0 - * crop -* load - * 100 parallel requests - * same file - * different local files - * different remote files - * pre-generated thumbnails - * non-pre-generated thumbnails diff --git a/cmd/mediaapi-integration-tests/main.go b/cmd/mediaapi-integration-tests/main.go deleted file mode 100644 index 8a5a0d54..00000000 --- a/cmd/mediaapi-integration-tests/main.go +++ /dev/null @@ -1,269 +0,0 @@ -// Copyright 2017 Vector Creations Ltd -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package main - -import ( - "fmt" - "io/ioutil" - "net/http" - "os" - "os/exec" - "path" - "path/filepath" - "time" - - "github.com/matrix-org/dendrite/internal/test" - "github.com/matrix-org/gomatrixserverlib" - "gopkg.in/yaml.v2" -) - -var ( - // How long to wait for the server to write the expected output messages. - // This needs to be high enough to account for the time it takes to create - // the postgres database tables which can take a while on travis. - timeoutString = test.Defaulting(os.Getenv("TIMEOUT"), "10s") - // The name of maintenance database to connect to in order to create the test database. - postgresDatabase = test.Defaulting(os.Getenv("POSTGRES_DATABASE"), "postgres") - // The name of the test database to create. - testDatabaseName = test.Defaulting(os.Getenv("DATABASE_NAME"), "mediaapi_test") - // Postgres docker container name (for running psql). If not set, psql must be in PATH. - postgresContainerName = os.Getenv("POSTGRES_CONTAINER") - // Test image to be uploaded/downloaded - testJPEG = test.Defaulting(os.Getenv("TEST_JPEG_PATH"), "cmd/mediaapi-integration-tests/totem.jpg") - kafkaURI = test.Defaulting(os.Getenv("KAFKA_URIS"), "localhost:9092") -) - -var thumbnailSizes = (` -- width: 32 - height: 32 - method: crop -- width: 96 - height: 96 - method: crop -- width: 320 - height: 240 - method: scale -- width: 640 - height: 480 - method: scale -- width: 800 - height: 600 - method: scale -`) - -const serverType = "media-api" - -const testMediaID = "1VuVy8u_hmDllD8BrcY0deM34Bl7SPJeY9J6BkMmpx0" -const testContentType = "image/jpeg" -const testOrigin = "localhost:18001" - -var testDatabaseTemplate = "dbname=%s sslmode=disable binary_parameters=yes" - -var timeout time.Duration - -var port = 10000 - -func startMediaAPI(suffix string, dynamicThumbnails bool) (*exec.Cmd, chan error, *exec.Cmd, string, string) { - dir, err := ioutil.TempDir("", serverType+"-server-test"+suffix) - if err != nil { - panic(err) - } - - proxyAddr := "localhost:1800" + suffix - - database := fmt.Sprintf(testDatabaseTemplate, testDatabaseName+suffix) - cfg, nextPort, err := test.MakeConfig(dir, kafkaURI, database, "localhost", port) - if err != nil { - panic(err) - } - cfg.Global.ServerName = gomatrixserverlib.ServerName(proxyAddr) - cfg.MediaAPI.DynamicThumbnails = dynamicThumbnails - if err = yaml.Unmarshal([]byte(thumbnailSizes), &cfg.MediaAPI.ThumbnailSizes); err != nil { - panic(err) - } - - port = nextPort - if err = test.WriteConfig(cfg, dir); err != nil { - panic(err) - } - - serverArgs := []string{ - "--config", filepath.Join(dir, test.ConfigFile), - } - - databases := []string{ - testDatabaseName + suffix, - } - - proxyCmd, _ := test.StartProxy(proxyAddr, cfg) - - test.InitDatabase( - postgresDatabase, - postgresContainerName, - databases, - ) - - cmd, cmdChan := test.CreateBackgroundCommand( - filepath.Join(filepath.Dir(os.Args[0]), "dendrite-"+serverType+"-server"), - serverArgs, - ) - - fmt.Printf("==TESTSERVER== STARTED %v -> %v : %v\n", proxyAddr, cfg.MediaAPI.InternalAPI.Listen, dir) - return cmd, cmdChan, proxyCmd, proxyAddr, dir -} - -func cleanUpServer(cmd *exec.Cmd, dir string) { - // ensure server is dead, only cleaning up so don't care about errors this returns - cmd.Process.Kill() // nolint: errcheck - if err := os.RemoveAll(dir); err != nil { - fmt.Printf("WARNING: Failed to remove temporary directory %v: %q\n", dir, err) - } -} - -// Runs a battery of media API server tests -// The tests will pause at various points in this list to conduct tests on the HTTP responses before continuing. -func main() { - fmt.Println("==TESTING==", os.Args[0]) - - var err error - timeout, err = time.ParseDuration(timeoutString) - if err != nil { - fmt.Printf("ERROR: Invalid timeout string %v: %q\n", timeoutString, err) - return - } - - // create server1 with only pre-generated thumbnails allowed - server1Cmd, server1CmdChan, server1ProxyCmd, server1ProxyAddr, server1Dir := startMediaAPI("1", false) - defer cleanUpServer(server1Cmd, server1Dir) - defer server1ProxyCmd.Process.Kill() // nolint: errcheck - testDownload(server1ProxyAddr, server1ProxyAddr, "doesnotexist", 404, server1CmdChan) - - // upload a JPEG file - testUpload( - server1ProxyAddr, testJPEG, - ) - - // download that JPEG file - testDownload(server1ProxyAddr, testOrigin, testMediaID, 200, server1CmdChan) - - // thumbnail that JPEG file - testThumbnail(64, 64, "crop", server1ProxyAddr, server1CmdChan) - - // create server2 with dynamic thumbnail generation - server2Cmd, server2CmdChan, server2ProxyCmd, server2ProxyAddr, server2Dir := startMediaAPI("2", true) - defer cleanUpServer(server2Cmd, server2Dir) - defer server2ProxyCmd.Process.Kill() // nolint: errcheck - testDownload(server2ProxyAddr, server2ProxyAddr, "doesnotexist", 404, server2CmdChan) - - // pre-generated thumbnail that JPEG file via server2 - testThumbnail(800, 600, "scale", server2ProxyAddr, server2CmdChan) - - // download that JPEG file via server2 - testDownload(server2ProxyAddr, testOrigin, testMediaID, 200, server2CmdChan) - - // dynamic thumbnail that JPEG file via server2 - testThumbnail(1920, 1080, "scale", server2ProxyAddr, server2CmdChan) - - // thumbnail that JPEG file via server2 - testThumbnail(10000, 10000, "scale", server2ProxyAddr, server2CmdChan) - -} - -func getMediaURI(host, endpoint, query string, components []string) string { - pathComponents := []string{host, "_matrix/media/v1", endpoint} - pathComponents = append(pathComponents, components...) - return "https://" + path.Join(pathComponents...) + query -} - -func testUpload(host, filePath string) { - fmt.Printf("==TESTING== upload %v to %v\n", filePath, host) - file, err := os.Open(filePath) - defer file.Close() // nolint: errcheck, staticcheck, megacheck - if err != nil { - panic(err) - } - filename := filepath.Base(filePath) - stat, err := file.Stat() - if os.IsNotExist(err) { - panic(err) - } - fileSize := stat.Size() - - req, err := http.NewRequest( - "POST", - getMediaURI(host, "upload", "?filename="+filename, nil), - file, - ) - if err != nil { - panic(err) - } - req.ContentLength = fileSize - req.Header.Set("Content-Type", testContentType) - - wantedBody := `{"content_uri": "mxc://localhost:18001/` + testMediaID + `"}` - testReq := &test.Request{ - Req: req, - WantedStatusCode: 200, - WantedBody: test.CanonicalJSONInput([]string{wantedBody})[0], - } - if err := testReq.Do(); err != nil { - panic(err) - } - fmt.Printf("==TESTING== upload %v to %v PASSED\n", filePath, host) -} - -func testDownload(host, origin, mediaID string, wantedStatusCode int, serverCmdChan chan error) { - req, err := http.NewRequest( - "GET", - getMediaURI(host, "download", "", []string{ - origin, - mediaID, - }), - nil, - ) - if err != nil { - panic(err) - } - testReq := &test.Request{ - Req: req, - WantedStatusCode: wantedStatusCode, - WantedBody: "", - } - testReq.Run(fmt.Sprintf("download mxc://%v/%v from %v", origin, mediaID, host), timeout, serverCmdChan) -} - -func testThumbnail(width, height int, resizeMethod, host string, serverCmdChan chan error) { - query := fmt.Sprintf("?width=%v&height=%v", width, height) - if resizeMethod != "" { - query += "&method=" + resizeMethod - } - req, err := http.NewRequest( - "GET", - getMediaURI(host, "thumbnail", query, []string{ - testOrigin, - testMediaID, - }), - nil, - ) - if err != nil { - panic(err) - } - testReq := &test.Request{ - Req: req, - WantedStatusCode: 200, - WantedBody: "", - } - testReq.Run(fmt.Sprintf("thumbnail mxc://%v/%v%v from %v", testOrigin, testMediaID, query, host), timeout, serverCmdChan) -} diff --git a/cmd/mediaapi-integration-tests/totem.jpg b/cmd/mediaapi-integration-tests/totem.jpg deleted file mode 100644 index 9cc2d424..00000000 Binary files a/cmd/mediaapi-integration-tests/totem.jpg and /dev/null differ diff --git a/cmd/roomserver-integration-tests/main.go b/cmd/roomserver-integration-tests/main.go deleted file mode 100644 index ff3f06b6..00000000 --- a/cmd/roomserver-integration-tests/main.go +++ /dev/null @@ -1,442 +0,0 @@ -// Copyright 2017 Vector Creations Ltd -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package main - -import ( - "context" - "fmt" - "io/ioutil" - "os" - "os/exec" - "path/filepath" - "strings" - "time" - - "encoding/json" - - "net/http" - - "github.com/matrix-org/dendrite/internal/caching" - "github.com/matrix-org/dendrite/internal/test" - "github.com/matrix-org/dendrite/roomserver/api" - "github.com/matrix-org/dendrite/roomserver/inthttp" - "github.com/matrix-org/dendrite/setup/config" - "github.com/matrix-org/gomatrixserverlib" -) - -var ( - // Path to where kafka is installed. - kafkaDir = defaulting(os.Getenv("KAFKA_DIR"), "kafka") - // The URI the kafka zookeeper is listening on. - zookeeperURI = defaulting(os.Getenv("ZOOKEEPER_URI"), "localhost:2181") - // The URI the kafka server is listening on. - kafkaURI = defaulting(os.Getenv("KAFKA_URIS"), "localhost:9092") - // How long to wait for the roomserver to write the expected output messages. - // This needs to be high enough to account for the time it takes to create - // the postgres database tables which can take a while on travis. - timeoutString = defaulting(os.Getenv("TIMEOUT"), "60s") - // Timeout for http client - timeoutHTTPClient = defaulting(os.Getenv("TIMEOUT_HTTP"), "30s") - // The name of maintenance database to connect to in order to create the test database. - postgresDatabase = defaulting(os.Getenv("POSTGRES_DATABASE"), "postgres") - // The name of the test database to create. - testDatabaseName = defaulting(os.Getenv("DATABASE_NAME"), "roomserver_test") - // The postgres connection config for connecting to the test database. - testDatabase = defaulting(os.Getenv("DATABASE"), fmt.Sprintf("dbname=%s binary_parameters=yes", testDatabaseName)) -) - -var exe = test.KafkaExecutor{ - ZookeeperURI: zookeeperURI, - KafkaDirectory: kafkaDir, - KafkaURI: kafkaURI, - // Send stdout and stderr to our stderr so that we see error messages from - // the kafka process. - OutputWriter: os.Stderr, -} - -func defaulting(value, defaultValue string) string { - if value == "" { - value = defaultValue - } - return value -} - -var ( - timeout time.Duration - timeoutHTTP time.Duration -) - -func init() { - var err error - timeout, err = time.ParseDuration(timeoutString) - if err != nil { - panic(err) - } - timeoutHTTP, err = time.ParseDuration(timeoutHTTPClient) - if err != nil { - panic(err) - } -} - -func createDatabase(database string) error { - cmd := exec.Command("psql", postgresDatabase) - cmd.Stdin = strings.NewReader( - fmt.Sprintf("DROP DATABASE IF EXISTS %s; CREATE DATABASE %s;", database, database), - ) - // Send stdout and stderr to our stderr so that we see error messages from - // the psql process - cmd.Stdout = os.Stderr - cmd.Stderr = os.Stderr - return cmd.Run() -} - -// runAndReadFromTopic runs a command and waits for a number of messages to be -// written to a kafka topic. It returns if the command exits, the number of -// messages is reached or after a timeout. It kills the command before it returns. -// It returns a list of the messages read from the command on success or an error -// on failure. -func runAndReadFromTopic(runCmd *exec.Cmd, readyURL string, doInput func(), topic string, count int, checkQueryAPI func()) ([]string, error) { - type result struct { - // data holds all of stdout on success. - data []byte - // err is set on failure. - err error - } - done := make(chan result) - readCmd := exec.Command( - filepath.Join(kafkaDir, "bin", "kafka-console-consumer.sh"), - "--bootstrap-server", kafkaURI, - "--topic", topic, - "--from-beginning", - "--max-messages", fmt.Sprintf("%d", count), - ) - // Send stderr to our stderr so the user can see any error messages. - readCmd.Stderr = os.Stderr - - // Kill both processes before we exit. - defer func() { runCmd.Process.Kill() }() // nolint: errcheck - defer func() { readCmd.Process.Kill() }() // nolint: errcheck - - // Run the command, read the messages and wait for a timeout in parallel. - go func() { - // Read all of stdout. - defer func() { - if err := recover(); err != nil { - if errv, ok := err.(error); ok { - done <- result{nil, errv} - } else { - panic(err) - } - } - }() - data, err := readCmd.Output() - checkQueryAPI() - done <- result{data, err} - }() - go func() { - err := runCmd.Run() - done <- result{nil, err} - }() - go func() { - time.Sleep(timeout) - done <- result{nil, fmt.Errorf("Timeout reading %d messages from topic %q", count, topic)} - }() - - // Poll the HTTP listener of the process waiting for it to be ready to receive requests. - ready := make(chan struct{}) - go func() { - delay := 10 * time.Millisecond - for { - time.Sleep(delay) - if delay < 100*time.Millisecond { - delay *= 2 - } - resp, err := http.Get(readyURL) - if err != nil { - continue - } - if resp.StatusCode == 200 { - break - } - } - ready <- struct{}{} - }() - - // Wait for the roomserver to be ready to receive input or for it to crash. - select { - case <-ready: - case r := <-done: - return nil, r.err - } - - // Write the input now that the server is running. - doInput() - - // Wait for one of the tasks to finsh. - r := <-done - - if r.err != nil { - return nil, r.err - } - - // The kafka console consumer writes a newline character after each message. - // So we split on newline characters - lines := strings.Split(string(r.data), "\n") - if len(lines) > 0 { - // Remove the blank line at the end of the data. - lines = lines[:len(lines)-1] - } - return lines, nil -} - -func writeToRoomServer(input []string, roomserverURL string) error { - var request api.InputRoomEventsRequest - var response api.InputRoomEventsResponse - var err error - request.InputRoomEvents = make([]api.InputRoomEvent, len(input)) - for i := range input { - if err = json.Unmarshal([]byte(input[i]), &request.InputRoomEvents[i]); err != nil { - return err - } - } - x, err := inthttp.NewRoomserverClient(roomserverURL, &http.Client{Timeout: timeoutHTTP}, nil) - if err != nil { - return err - } - x.InputRoomEvents(context.Background(), &request, &response) - return response.Err() -} - -// testRoomserver is used to run integration tests against a single roomserver. -// It creates new kafka topics for the input and output of the roomserver. -// It writes the input messages to the input kafka topic, formatting each message -// as canonical JSON so that it fits on a single line. -// It then runs the roomserver and waits for a number of messages to be written -// to the output topic. -// Once those messages have been written it runs the checkQueries function passing -// a api.RoomserverQueryAPI client. The caller can use this function to check the -// behaviour of the query API. -func testRoomserver(input []string, wantOutput []string, checkQueries func(api.RoomserverInternalAPI)) { - dir, err := ioutil.TempDir("", "room-server-test") - if err != nil { - panic(err) - } - - cfg, _, err := test.MakeConfig(dir, kafkaURI, testDatabase, "localhost", 10000) - if err != nil { - panic(err) - } - if err = test.WriteConfig(cfg, dir); err != nil { - panic(err) - } - - outputTopic := cfg.Global.Kafka.TopicFor(config.TopicOutputRoomEvent) - - err = exe.DeleteTopic(outputTopic) - if err != nil { - panic(err) - } - - if err = exe.CreateTopic(outputTopic); err != nil { - panic(err) - } - - if err = createDatabase(testDatabaseName); err != nil { - panic(err) - } - - cache, err := caching.NewInMemoryLRUCache(false) - if err != nil { - panic(err) - } - - doInput := func() { - fmt.Printf("Roomserver is ready to receive input, sending %d events\n", len(input)) - if err = writeToRoomServer(input, cfg.RoomServerURL()); err != nil { - panic(err) - } - } - - cmd := exec.Command(filepath.Join(filepath.Dir(os.Args[0]), "dendrite-room-server")) - - // Append the roomserver config to the existing environment. - // We append to the environment rather than replacing so that any additional - // postgres and golang environment variables such as PGHOST are passed to - // the roomserver process. - cmd.Stderr = os.Stderr - cmd.Args = []string{"dendrite-room-server", "--config", filepath.Join(dir, test.ConfigFile)} - - gotOutput, err := runAndReadFromTopic(cmd, cfg.RoomServerURL()+"/metrics", doInput, outputTopic, len(wantOutput), func() { - queryAPI, _ := inthttp.NewRoomserverClient("http://"+string(cfg.RoomServer.InternalAPI.Connect), &http.Client{Timeout: timeoutHTTP}, cache) - checkQueries(queryAPI) - }) - if err != nil { - panic(err) - } - - if len(wantOutput) != len(gotOutput) { - panic(fmt.Errorf("Wanted %d lines of output got %d lines", len(wantOutput), len(gotOutput))) - } - - for i := range wantOutput { - if !equalJSON(wantOutput[i], gotOutput[i]) { - panic(fmt.Errorf("Wanted %q at index %d got %q", wantOutput[i], i, gotOutput[i])) - } - } -} - -func equalJSON(a, b string) bool { - canonicalA, err := gomatrixserverlib.CanonicalJSON([]byte(a)) - if err != nil { - panic(err) - } - canonicalB, err := gomatrixserverlib.CanonicalJSON([]byte(b)) - if err != nil { - panic(err) - } - return string(canonicalA) == string(canonicalB) -} - -func main() { - fmt.Println("==TESTING==", os.Args[0]) - - input := []string{ - `{ - "auth_event_ids": [], - "kind": 1, - "event": { - "origin": "matrix.org", - "signatures": { - "matrix.org": { - "ed25519:auto": "3kXGwNtdj+zqEXlI8PWLiB76xtrQ7SxcvPuXAEVCTo+QPoBoUvLi1RkHs6O5mDz7UzIowK5bi1seAN4vOh0OBA" - } - }, - "origin_server_ts": 1463671337837, - "sender": "@richvdh:matrix.org", - "event_id": "$1463671337126266wrSBX:matrix.org", - "prev_events": [], - "state_key": "", - "content": {"creator": "@richvdh:matrix.org"}, - "depth": 1, - "prev_state": [], - "room_id": "!HCXfdvrfksxuYnIFiJ:matrix.org", - "auth_events": [], - "hashes": {"sha256": "Q05VLC8nztN2tguy+KnHxxhitI95wK9NelnsDaXRqeo"}, - "type": "m.room.create"} - }`, `{ - "auth_event_ids": ["$1463671337126266wrSBX:matrix.org"], - "kind": 2, - "state_event_ids": ["$1463671337126266wrSBX:matrix.org"], - "event": { - "origin": "matrix.org", - "signatures": { - "matrix.org": { - "ed25519:auto": "a2b3xXYVPPFeG1sHCU3hmZnAaKqZFgzGZozijRGblG5Y//ewRPAn1A2mCrI2UM5I+0zqr70cNpHgF8bmNFu4BA" - } - }, - "origin_server_ts": 1463671339844, - "sender": "@richvdh:matrix.org", - "event_id": "$1463671339126270PnVwC:matrix.org", - "prev_events": [[ - "$1463671337126266wrSBX:matrix.org", {"sha256": "h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"} - ]], - "membership": "join", - "state_key": "@richvdh:matrix.org", - "content": { - "membership": "join", - "avatar_url": "mxc://matrix.org/ZafPzsxMJtLaSaJXloBEKiws", - "displayname": "richvdh" - }, - "depth": 2, - "prev_state": [], - "room_id": "!HCXfdvrfksxuYnIFiJ:matrix.org", - "auth_events": [[ - "$1463671337126266wrSBX:matrix.org", {"sha256": "h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"} - ]], - "hashes": {"sha256": "t9t3sZV1Eu0P9Jyrs7pge6UTa1zuTbRdVxeUHnrQVH0"}, - "type": "m.room.member"}, - "has_state": true - }`, - } - - want := []string{ - `{"type":"new_room_event","new_room_event":{ - "event":{ - "auth_events":[[ - "$1463671337126266wrSBX:matrix.org",{"sha256":"h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"} - ]], - "content":{ - "avatar_url":"mxc://matrix.org/ZafPzsxMJtLaSaJXloBEKiws", - "displayname":"richvdh", - "membership":"join" - }, - "depth": 2, - "event_id": "$1463671339126270PnVwC:matrix.org", - "hashes": {"sha256":"t9t3sZV1Eu0P9Jyrs7pge6UTa1zuTbRdVxeUHnrQVH0"}, - "membership": "join", - "origin": "matrix.org", - "origin_server_ts": 1463671339844, - "prev_events": [[ - "$1463671337126266wrSBX:matrix.org",{"sha256":"h/VS07u8KlMwT3Ee8JhpkC7sa1WUs0Srgs+l3iBv6c0"} - ]], - "prev_state":[], - "room_id":"!HCXfdvrfksxuYnIFiJ:matrix.org", - "sender":"@richvdh:matrix.org", - "signatures":{ - "matrix.org":{ - "ed25519:auto":"a2b3xXYVPPFeG1sHCU3hmZnAaKqZFgzGZozijRGblG5Y//ewRPAn1A2mCrI2UM5I+0zqr70cNpHgF8bmNFu4BA" - } - }, - "state_key":"@richvdh:matrix.org", - "type":"m.room.member" - }, - "state_before_removes_event_ids":["$1463671339126270PnVwC:matrix.org"], - "state_before_adds_event_ids":null, - "latest_event_ids":["$1463671339126270PnVwC:matrix.org"], - "adds_state_event_ids":["$1463671337126266wrSBX:matrix.org", "$1463671339126270PnVwC:matrix.org"], - "removes_state_event_ids":null, - "last_sent_event_id":"", - "send_as_server":"", - "transaction_id": null - }}`, - } - - testRoomserver(input, want, func(q api.RoomserverInternalAPI) { - var response api.QueryLatestEventsAndStateResponse - if err := q.QueryLatestEventsAndState( - context.Background(), - &api.QueryLatestEventsAndStateRequest{ - RoomID: "!HCXfdvrfksxuYnIFiJ:matrix.org", - StateToFetch: []gomatrixserverlib.StateKeyTuple{ - {EventType: "m.room.member", StateKey: "@richvdh:matrix.org"}, - }, - }, - &response, - ); err != nil { - panic(err) - } - if !response.RoomExists { - panic(fmt.Errorf(`Wanted room "!HCXfdvrfksxuYnIFiJ:matrix.org" to exist`)) - } - if len(response.LatestEvents) != 1 || response.LatestEvents[0].EventID != "$1463671339126270PnVwC:matrix.org" { - panic(fmt.Errorf(`Wanted "$1463671339126270PnVwC:matrix.org" to be the latest event got %#v`, response.LatestEvents)) - } - if len(response.StateEvents) != 1 || response.StateEvents[0].EventID() != "$1463671339126270PnVwC:matrix.org" { - panic(fmt.Errorf(`Wanted "$1463671339126270PnVwC:matrix.org" to be the state event got %#v`, response.StateEvents)) - } - }) - - fmt.Println("==PASSED==", os.Args[0]) -} diff --git a/cmd/syncserver-integration-tests/main.go b/cmd/syncserver-integration-tests/main.go deleted file mode 100644 index 332bde10..00000000 --- a/cmd/syncserver-integration-tests/main.go +++ /dev/null @@ -1,563 +0,0 @@ -// Copyright 2017 Vector Creations Ltd -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package main - -import ( - "encoding/json" - "fmt" - "io/ioutil" - "net/http" - "os" - "os/exec" - "path/filepath" - "time" - - "github.com/matrix-org/dendrite/internal/test" - "github.com/matrix-org/dendrite/roomserver/api" - "github.com/matrix-org/dendrite/setup/config" - "github.com/matrix-org/gomatrixserverlib" -) - -var ( - // Path to where kafka is installed. - kafkaDir = test.Defaulting(os.Getenv("KAFKA_DIR"), "kafka") - // The URI the kafka zookeeper is listening on. - zookeeperURI = test.Defaulting(os.Getenv("ZOOKEEPER_URI"), "localhost:2181") - // The URI the kafka server is listening on. - kafkaURI = test.Defaulting(os.Getenv("KAFKA_URIS"), "localhost:9092") - // The address the syncserver should listen on. - syncserverAddr = test.Defaulting(os.Getenv("SYNCSERVER_URI"), "localhost:9876") - // How long to wait for the syncserver to write the expected output messages. - // This needs to be high enough to account for the time it takes to create - // the postgres database tables which can take a while on travis. - timeoutString = test.Defaulting(os.Getenv("TIMEOUT"), "10s") - // The name of maintenance database to connect to in order to create the test database. - postgresDatabase = test.Defaulting(os.Getenv("POSTGRES_DATABASE"), "postgres") - // Postgres docker container name (for running psql). If not set, psql must be in PATH. - postgresContainerName = os.Getenv("POSTGRES_CONTAINER") - // The name of the test database to create. - testDatabaseName = test.Defaulting(os.Getenv("DATABASE_NAME"), "syncserver_test") - // The postgres connection config for connecting to the test database. - testDatabase = test.Defaulting(os.Getenv("DATABASE"), fmt.Sprintf("dbname=%s sslmode=disable binary_parameters=yes", testDatabaseName)) -) - -const inputTopic = "syncserverInput" -const clientTopic = "clientapiserverOutput" - -var exe = test.KafkaExecutor{ - ZookeeperURI: zookeeperURI, - KafkaDirectory: kafkaDir, - KafkaURI: kafkaURI, - // Send stdout and stderr to our stderr so that we see error messages from - // the kafka process. - OutputWriter: os.Stderr, -} - -var timeout time.Duration -var clientEventTestData []string - -func init() { - var err error - timeout, err = time.ParseDuration(timeoutString) - if err != nil { - panic(err) - } - - for _, s := range outputRoomEventTestData { - clientEventTestData = append(clientEventTestData, clientEventJSONForOutputRoomEvent(s)) - } -} - -func createTestUser(database, username, token string) error { - cmd := exec.Command( - filepath.Join(filepath.Dir(os.Args[0]), "create-account"), - "--database", database, - "--username", username, - "--token", token, - ) - - // Send stdout and stderr to our stderr so that we see error messages from - // the create-account process - cmd.Stdout = os.Stderr - cmd.Stderr = os.Stderr - return cmd.Run() -} - -// clientEventJSONForOutputRoomEvent parses the given output room event and extracts the 'Event' JSON. It is -// trimmed to the client format and then canonicalised and returned as a string. -// Panics if there are any problems. -func clientEventJSONForOutputRoomEvent(outputRoomEvent string) string { - var out api.OutputEvent - if err := json.Unmarshal([]byte(outputRoomEvent), &out); err != nil { - panic("failed to unmarshal output room event: " + err.Error()) - } - clientEvs := gomatrixserverlib.ToClientEvents([]*gomatrixserverlib.Event{ - out.NewRoomEvent.Event.Event, - }, gomatrixserverlib.FormatSync) - b, err := json.Marshal(clientEvs[0]) - if err != nil { - panic("failed to marshal client event as json: " + err.Error()) - } - jsonBytes, err := gomatrixserverlib.CanonicalJSON(b) - if err != nil { - panic("failed to turn event json into canonical json: " + err.Error()) - } - return string(jsonBytes) -} - -// startSyncServer creates the database and config file needed for the sync server to run and -// then starts the sync server. The Cmd being executed is returned. A channel is also returned, -// which will have any termination errors sent down it, followed immediately by the channel being closed. -func startSyncServer() (*exec.Cmd, chan error) { - - dir, err := ioutil.TempDir("", "syncapi-server-test") - if err != nil { - panic(err) - } - - cfg, _, err := test.MakeConfig(dir, kafkaURI, testDatabase, "localhost", 10000) - if err != nil { - panic(err) - } - // TODO use the address assigned by the config generator rather than clobbering. - cfg.Global.ServerName = "localhost" - cfg.SyncAPI.InternalAPI.Listen = config.HTTPAddress("http://" + syncserverAddr) - cfg.SyncAPI.InternalAPI.Connect = cfg.SyncAPI.InternalAPI.Listen - - if err := test.WriteConfig(cfg, dir); err != nil { - panic(err) - } - - serverArgs := []string{ - "--config", filepath.Join(dir, test.ConfigFile), - } - - databases := []string{ - testDatabaseName, - } - - test.InitDatabase( - postgresDatabase, - postgresContainerName, - databases, - ) - - if err := createTestUser(testDatabase, "alice", "@alice:localhost"); err != nil { - panic(err) - } - if err := createTestUser(testDatabase, "bob", "@bob:localhost"); err != nil { - panic(err) - } - if err := createTestUser(testDatabase, "charlie", "@charlie:localhost"); err != nil { - panic(err) - } - - cmd, cmdChan := test.CreateBackgroundCommand( - filepath.Join(filepath.Dir(os.Args[0]), "dendrite-sync-api-server"), - serverArgs, - ) - - return cmd, cmdChan -} - -// prepareKafka creates the topics which will be written to by the tests. -func prepareKafka() { - err := exe.DeleteTopic(inputTopic) - if err != nil { - panic(err) - } - - if err = exe.CreateTopic(inputTopic); err != nil { - panic(err) - } - - err = exe.DeleteTopic(clientTopic) - if err != nil { - panic(err) - } - - if err = exe.CreateTopic(clientTopic); err != nil { - panic(err) - } -} - -func testSyncServer(syncServerCmdChan chan error, userID, since, want string) { - fmt.Printf("==TESTING== testSyncServer(%s,%s)\n", userID, since) - sinceQuery := "" - if since != "" { - sinceQuery = "&since=" + since - } - req, err := http.NewRequest( - "GET", - "http://"+syncserverAddr+"/api/_matrix/client/r0/sync?timeout=100&access_token="+userID+sinceQuery, - nil, - ) - if err != nil { - panic(err) - } - testReq := &test.Request{ - Req: req, - WantedStatusCode: 200, - WantedBody: test.CanonicalJSONInput([]string{want})[0], - } - testReq.Run("sync-api", timeout, syncServerCmdChan) -} - -func writeToRoomServerLog(indexes ...int) { - var roomEvents []string - for _, i := range indexes { - roomEvents = append(roomEvents, outputRoomEventTestData[i]) - } - if err := exe.WriteToTopic(inputTopic, test.CanonicalJSONInput(roomEvents)); err != nil { - panic(err) - } -} - -// Runs a battery of sync server tests against test data in testdata.go -// testdata.go has a list of OutputRoomEvents which will be fed into the kafka log which the sync server will consume. -// The tests will pause at various points in this list to conduct tests on the /sync responses before continuing. -// For ease of understanding, the curl commands used to create the OutputRoomEvents are listed along with each write to kafka. -func main() { - fmt.Println("==TESTING==", os.Args[0]) - prepareKafka() - cmd, syncServerCmdChan := startSyncServer() - // ensure server is dead, only cleaning up so don't care about errors this returns. - defer cmd.Process.Kill() // nolint: errcheck - - // $ curl -XPOST -d '{}' "http://localhost:8009/_matrix/client/r0/createRoom?access_token=@alice:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@alice:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 2"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 3"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - // $ curl -XPUT -d '{"name":"Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - writeToRoomServerLog( - i0StateRoomCreate, i1StateAliceJoin, i2StatePowerLevels, i3StateJoinRules, i4StateHistoryVisibility, - i5AliceMsg, i6AliceMsg, i7AliceMsg, i8StateAliceRoomName, - ) - - // Make sure initial sync works TODO: prev_batch - testSyncServer(syncServerCmdChan, "@alice:localhost", "", `{ - "account_data": { - "events": [] - }, - "next_batch": "9", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": { - "!PjrbIMW2cIiaYF4t:localhost": { - "account_data": { - "events": [] - }, - "ephemeral": { - "events": [] - }, - "state": { - "events": [] - }, - "timeline": { - "events": [`+ - clientEventTestData[i0StateRoomCreate]+","+ - clientEventTestData[i1StateAliceJoin]+","+ - clientEventTestData[i2StatePowerLevels]+","+ - clientEventTestData[i3StateJoinRules]+","+ - clientEventTestData[i4StateHistoryVisibility]+","+ - clientEventTestData[i5AliceMsg]+","+ - clientEventTestData[i6AliceMsg]+","+ - clientEventTestData[i7AliceMsg]+","+ - clientEventTestData[i8StateAliceRoomName]+`], - "limited": true, - "prev_batch": "" - } - } - }, - "leave": {} - } - }`) - // Make sure alice's rooms don't leak to bob - testSyncServer(syncServerCmdChan, "@bob:localhost", "", `{ - "account_data": { - "events": [] - }, - "next_batch": "9", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": {}, - "leave": {} - } - }`) - // Make sure polling with an up-to-date token returns nothing new - testSyncServer(syncServerCmdChan, "@alice:localhost", "9", `{ - "account_data": { - "events": [] - }, - "next_batch": "9", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": {}, - "leave": {} - } - }`) - - // $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost" - writeToRoomServerLog(i9StateBobJoin) - - // Make sure alice sees it TODO: prev_batch - testSyncServer(syncServerCmdChan, "@alice:localhost", "9", `{ - "account_data": { - "events": [] - }, - "next_batch": "10", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": { - "!PjrbIMW2cIiaYF4t:localhost": { - "account_data": { - "events": [] - }, - "ephemeral": { - "events": [] - }, - "state": { - "events": [] - }, - "timeline": { - "limited": false, - "prev_batch": "", - "events": [`+clientEventTestData[i9StateBobJoin]+`] - } - } - }, - "leave": {} - } - }`) - - // Make sure bob sees the room AND all the current room state TODO: history visibility - testSyncServer(syncServerCmdChan, "@bob:localhost", "9", `{ - "account_data": { - "events": [] - }, - "next_batch": "10", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": { - "!PjrbIMW2cIiaYF4t:localhost": { - "account_data": { - "events": [] - }, - "ephemeral": { - "events": [] - }, - "state": { - "events": [`+ - clientEventTestData[i0StateRoomCreate]+","+ - clientEventTestData[i1StateAliceJoin]+","+ - clientEventTestData[i2StatePowerLevels]+","+ - clientEventTestData[i3StateJoinRules]+","+ - clientEventTestData[i4StateHistoryVisibility]+","+ - clientEventTestData[i8StateAliceRoomName]+`] - }, - "timeline": { - "limited": false, - "prev_batch": "", - "events": [`+ - clientEventTestData[i9StateBobJoin]+`] - } - } - }, - "leave": {} - } - }`) - - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello alice"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@bob:localhost" - writeToRoomServerLog(i10BobMsg) - - // Make sure alice can see everything around the join point for bob TODO: prev_batch - testSyncServer(syncServerCmdChan, "@alice:localhost", "7", `{ - "account_data": { - "events": [] - }, - "next_batch": "11", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": { - "!PjrbIMW2cIiaYF4t:localhost": { - "account_data": { - "events": [] - }, - "ephemeral": { - "events": [] - }, - "state": { - "events": [] - }, - "timeline": { - "limited": false, - "prev_batch": "", - "events": [`+ - clientEventTestData[i7AliceMsg]+","+ - clientEventTestData[i8StateAliceRoomName]+","+ - clientEventTestData[i9StateBobJoin]+","+ - clientEventTestData[i10BobMsg]+`] - } - } - }, - "leave": {} - } - }`) - - // $ curl -XPUT -d '{"name":"A Different Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello bob"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost" - // $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@bob:localhost" - writeToRoomServerLog(i11StateAliceRoomName, i12AliceMsg, i13StateBobInviteCharlie) - - // Make sure charlie sees the invite both with and without a ?since= token - // TODO: Invite state should include the invite event and the room name. - charlieInviteData := `{ - "account_data": { - "events": [] - }, - "next_batch": "14", - "presence": { - "events": [] - }, - "rooms": { - "invite": { - "!PjrbIMW2cIiaYF4t:localhost": { - "invite_state": { - "events": [] - } - } - }, - "join": {}, - "leave": {} - } - }` - testSyncServer(syncServerCmdChan, "@charlie:localhost", "7", charlieInviteData) - testSyncServer(syncServerCmdChan, "@charlie:localhost", "", charlieInviteData) - - // $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"not charlie..."}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - // $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@alice:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"why did you kick charlie"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost" - writeToRoomServerLog(i14StateCharlieJoin, i15AliceMsg, i16StateAliceKickCharlie, i17BobMsg) - - // Check transitions to leave work - testSyncServer(syncServerCmdChan, "@charlie:localhost", "15", `{ - "account_data": { - "events": [] - }, - "next_batch": "18", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": {}, - "leave": { - "!PjrbIMW2cIiaYF4t:localhost": { - "state": { - "events": [] - }, - "timeline": { - "limited": false, - "prev_batch": "", - "events": [`+ - clientEventTestData[i15AliceMsg]+","+ - clientEventTestData[i16StateAliceKickCharlie]+`] - } - } - } - } - }`) - - // Test joining and leaving the same room in a single /sync request puts the room in the 'leave' section. - // TODO: Use an earlier since value to assert that the /sync response doesn't leak messages - // from before charlie was joined to the room. Currently it does leak because RecentEvents doesn't - // take membership into account. - testSyncServer(syncServerCmdChan, "@charlie:localhost", "14", `{ - "account_data": { - "events": [] - }, - "next_batch": "18", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": {}, - "leave": { - "!PjrbIMW2cIiaYF4t:localhost": { - "state": { - "events": [] - }, - "timeline": { - "limited": false, - "prev_batch": "", - "events": [`+ - clientEventTestData[i14StateCharlieJoin]+","+ - clientEventTestData[i15AliceMsg]+","+ - clientEventTestData[i16StateAliceKickCharlie]+`] - } - } - } - } - }`) - - // $ curl -XPUT -d '{"name":"No Charlies"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - writeToRoomServerLog(i18StateAliceRoomName) - - // Check that users don't see state changes in rooms after they have left - testSyncServer(syncServerCmdChan, "@charlie:localhost", "17", `{ - "account_data": { - "events": [] - }, - "next_batch": "19", - "presence": { - "events": [] - }, - "rooms": { - "invite": {}, - "join": {}, - "leave": {} - } - }`) - - // $ curl -XPUT -d '{"msgtype":"m.text","body":"whatever"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost" - // $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"im alone now"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - // $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@alice:localhost" - // $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"so alone"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - // $ curl -XPUT -d '{"name":"Everyone welcome"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - // $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost" - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hiiiii"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@charlie:localhost" -} diff --git a/cmd/syncserver-integration-tests/testdata.go b/cmd/syncserver-integration-tests/testdata.go deleted file mode 100644 index 4ff5d1ee..00000000 --- a/cmd/syncserver-integration-tests/testdata.go +++ /dev/null @@ -1,102 +0,0 @@ -// Copyright 2017 Vector Creations Ltd -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package main - -// nolint: varcheck, deadcode, unused, megacheck -const ( - i0StateRoomCreate = iota - i1StateAliceJoin - i2StatePowerLevels - i3StateJoinRules - i4StateHistoryVisibility - i5AliceMsg - i6AliceMsg - i7AliceMsg - i8StateAliceRoomName - i9StateBobJoin - i10BobMsg - i11StateAliceRoomName - i12AliceMsg - i13StateBobInviteCharlie - i14StateCharlieJoin - i15AliceMsg - i16StateAliceKickCharlie - i17BobMsg - i18StateAliceRoomName - i19BobMsg - i20StateBobLeave - i21AliceMsg - i22StateAliceInviteBob - i23StateBobRejectInvite - i24AliceMsg - i25StateAliceRoomName - i26StateCharlieJoin - i27CharlieMsg -) - -var outputRoomEventTestData = []string{ - // $ curl -XPOST -d '{}' "http://localhost:8009/_matrix/client/r0/createRoom?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[],"content":{"creator":"@alice:localhost"},"depth":1,"event_id":"$xz0fUB8zNMTGFh1W:localhost","hashes":{"sha256":"KKkpxS8NoH0igBbL3J+nJ39MRlmA7QgW4BGL7Fv4ASI"},"origin":"localhost","origin_server_ts":1494411218382,"prev_events":[],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"uZG5Q/Hs2Z611gFlZPdwomomRJKf70xV2FQV+gLWM1XgzkLDRlRF3cBZc9y3CnHKnV/upTcXs7Op2/GmgD3UBw"}},"state_key":"","type":"m.room.create"},"latest_event_ids":["$xz0fUB8zNMTGFh1W:localhost"],"adds_state_event_ids":["$xz0fUB8zNMTGFh1W:localhost"],"last_sent_event_id":""}}`, - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}]],"content":{"membership":"join"},"depth":2,"event_id":"$QTen1vksfcRTpUCk:localhost","hashes":{"sha256":"tTukc9ab1fJfzgc5EMA/UD3swqfl/ic9Y9Zkt4fJo0Q"},"origin":"localhost","origin_server_ts":1494411218385,"prev_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"OPysDn/wT7yHeALXLTcEgR+iaKjv0p7VPuR/Mzvyg2IMAwPUjSOw8SQZlhSioWRtVPUp9VHbhIhJxQaPUg9yBQ"}},"state_key":"@alice:localhost","type":"m.room.member"},"latest_event_ids":["$QTen1vksfcRTpUCk:localhost"],"adds_state_event_ids":["$QTen1vksfcRTpUCk:localhost"],"last_sent_event_id":"$xz0fUB8zNMTGFh1W:localhost"}}`, - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"content":{"ban":50,"events":{"m.room.avatar":50,"m.room.canonical_alias":50,"m.room.history_visibility":100,"m.room.name":50,"m.room.power_levels":100},"events_default":0,"invite":0,"kick":50,"redact":50,"state_default":50,"users":{"@alice:localhost":100},"users_default":0},"depth":3,"event_id":"$RWsxGlfPHAcijTgu:localhost","hashes":{"sha256":"ueZWiL/Q8bagRQGFktpnYJAJV6V6U3QKcUEmWYeyaaM"},"origin":"localhost","origin_server_ts":1494411218385,"prev_events":[["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"hZwWx3lyW61zMYmqLOxLTlfW2CnbjJQsZPLjZFa97TVG4ISz8CixMPsnVAIu5is29UCmiHyP8RvLecJjbLCtAQ"}},"state_key":"","type":"m.room.power_levels"},"latest_event_ids":["$RWsxGlfPHAcijTgu:localhost"],"adds_state_event_ids":["$RWsxGlfPHAcijTgu:localhost"],"last_sent_event_id":"$QTen1vksfcRTpUCk:localhost"}}`, - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"content":{"join_rule":"public"},"depth":4,"event_id":"$2O2DpHB37CuwwJOe:localhost","hashes":{"sha256":"3P3HxAXI8gc094i020EoV/gissYiMVWv8+JAbrakM4E"},"origin":"localhost","origin_server_ts":1494411218386,"prev_events":[["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"L2yZoBbG/6TNsRHz+UtHY0SK4FgrdAYPR1l7RBWaNFbm+k/7kVhnoGlJ9yptpdLJjPMR2InqKXH8BBxRC83BCg"}},"state_key":"","type":"m.room.join_rules"},"latest_event_ids":["$2O2DpHB37CuwwJOe:localhost"],"adds_state_event_ids":["$2O2DpHB37CuwwJOe:localhost"],"last_sent_event_id":"$RWsxGlfPHAcijTgu:localhost"}}`, - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}]],"content":{"history_visibility":"joined"},"depth":5,"event_id":"$5LRiBskVCROnL5WY:localhost","hashes":{"sha256":"341alVufcKSVKLPr9WsJNTnW33QkBTn9eTfVWbyoa0o"},"origin":"localhost","origin_server_ts":1494411218387,"prev_events":[["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"kRyt68cstwYgK8NtYzf0V5CnAbqUO47ixCCWYzRCi0WNstEwUw4XW1GHc8BllQsXwSj+nNv9g/66zZgG0DtxCA"}},"state_key":"","type":"m.room.history_visibility"},"latest_event_ids":["$5LRiBskVCROnL5WY:localhost"],"adds_state_event_ids":["$5LRiBskVCROnL5WY:localhost"],"last_sent_event_id":"$2O2DpHB37CuwwJOe:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello world","msgtype":"m.text"},"depth":0,"event_id":"$Z8ZJik7ghwzSYTH9:localhost","hashes":{"sha256":"ahN1T5aiSZCzllf0pqNWJkF+x2h2S3kic+40pQ1X6BE"},"origin":"localhost","origin_server_ts":1494411339207,"prev_events":[["$5LRiBskVCROnL5WY:localhost",{"sha256":"3jULNC9b9Q0AhvnDQqpjhbtYwmkioHzPzdTJZvn8vOI"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"ylEpahRwEfGpqk+UCv0IF8YAxmut7w7udgHy3sVDfdJhs/4uJ6EkFEsKLknpXRc1vTIy1etKCBQ63QbCmRC2Bw"}},"type":"m.room.message"},"latest_event_ids":["$Z8ZJik7ghwzSYTH9:localhost"],"last_sent_event_id":"$5LRiBskVCROnL5WY:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 2"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello world 2","msgtype":"m.text"},"depth":0,"event_id":"$8382Ah682eL4hxjN:localhost","hashes":{"sha256":"hQElDGSYc6KOdylrbMMm3+LlvUiCKo6S9G9n58/qtns"},"origin":"localhost","origin_server_ts":1494411380282,"prev_events":[["$Z8ZJik7ghwzSYTH9:localhost",{"sha256":"FBDwP+2FeqDENe7AEa3iAFAVKl1/IVq43mCH0uPRn90"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"LFXi6jTG7qn9xzi4rhIiHbkLD+4AZ9Yg7UTS2gqm1gt2lXQsgTYH1wE4Fol2fq4lvGlQVpxhtEr2huAYSbT7DA"}},"type":"m.room.message"},"latest_event_ids":["$8382Ah682eL4hxjN:localhost"],"last_sent_event_id":"$Z8ZJik7ghwzSYTH9:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello world 3"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello world 3","msgtype":"m.text"},"depth":0,"event_id":"$17SfHsvSeTQthSWF:localhost","hashes":{"sha256":"eS6VFQI0l2U8rA8U17jgSHr9lQ73SNSnlnZu+HD0IjE"},"origin":"localhost","origin_server_ts":1494411396560,"prev_events":[["$8382Ah682eL4hxjN:localhost",{"sha256":"c6I/PUY7WnvxQ+oUEp/w2HEEuD3g8Vq7QwPUOSUjuc8"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"dvu9bSHZmX+yZoEqHioK7YDMtLH9kol0DdFqc5aHsbhZe/fKRZpfJMrlf1iXQdXSCMhikvnboPAXN3guiZCUBQ"}},"type":"m.room.message"},"latest_event_ids":["$17SfHsvSeTQthSWF:localhost"],"last_sent_event_id":"$8382Ah682eL4hxjN:localhost"}}`, - // $ curl -XPUT -d '{"name":"Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"Custom Room Name"},"depth":0,"event_id":"$j7KtuOzM0K15h3Kr:localhost","hashes":{"sha256":"QIKj5Klr50ugll4EjaNUATJmrru4CDp6TvGPv0v15bo"},"origin":"localhost","origin_server_ts":1494411482625,"prev_events":[["$17SfHsvSeTQthSWF:localhost",{"sha256":"iMTefewJ4W5sKQy7osQv4ilJAi7X0NsK791kqEUmYX0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"WU7lwSWUAk7bsyDnBs128PyXxPZZoD1sN4AiDcvk+W1mDezJbFvWHDWymclxWESlP7TDrFTZEumRWGGCakjyAg"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$j7KtuOzM0K15h3Kr:localhost"],"adds_state_event_ids":["$j7KtuOzM0K15h3Kr:localhost"],"last_sent_event_id":"$17SfHsvSeTQthSWF:localhost"}}`, - // $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}]],"content":{"membership":"join"},"depth":0,"event_id":"$wPepDhIla765Odre:localhost","hashes":{"sha256":"KeKqWLvM+LTvyFbwx6y3Y4W5Pj6nBSFUQ6jpkSf1oTE"},"origin":"localhost","origin_server_ts":1494411534290,"prev_events":[["$j7KtuOzM0K15h3Kr:localhost",{"sha256":"oDrWG5/sy1Ea3hYDOSJZRuGKCcjaHQlDYPDn2gB0/L0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"oVtvjZbWFe+iJhoDvLcQKnFpSYQ94dOodM4gGsx26P6fs2sFJissYwSIqpoxlElCJnmBAgy5iv4JK/5x21R2CQ"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$wPepDhIla765Odre:localhost"],"adds_state_event_ids":["$wPepDhIla765Odre:localhost"],"last_sent_event_id":"$j7KtuOzM0K15h3Kr:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello alice"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/1?access_token=@bob:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"body":"hello alice","msgtype":"m.text"},"depth":0,"event_id":"$RHNjeYUvXVZfb93t:localhost","hashes":{"sha256":"Ic1QLxTWFrWt1o31DS93ftrNHkunf4O6ubFvdD4ydNI"},"origin":"localhost","origin_server_ts":1494411593196,"prev_events":[["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"8BHHkiThWwiIZbXCegRjIKNVGIa2kqrZW8VuL7nASfJBORhZ9R9p34UsmhsxVwTs/2/dX7M2ogMB28gIGdLQCg"}},"type":"m.room.message"},"latest_event_ids":["$RHNjeYUvXVZfb93t:localhost"],"last_sent_event_id":"$wPepDhIla765Odre:localhost"}}`, - // $ curl -XPUT -d '{"name":"A Different Custom Room Name"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"A Different Custom Room Name"},"depth":0,"event_id":"$1xoUuqOFjFFJgwA5:localhost","hashes":{"sha256":"2pNnLhoHxNeSUpqxrd3c0kZUA4I+cdWZgYcJ8V3e2tk"},"origin":"localhost","origin_server_ts":1494411643348,"prev_events":[["$RHNjeYUvXVZfb93t:localhost",{"sha256":"LqFmTIzULgUDSf5xM3REObvnsRGLQliWBUf1hEDT4+w"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"gsY4B6TIBdVvLyFAaXw0xez9N5/Cn/ZaJ4z+j9gJU/ZR8j1t3OYlcVQN6uln9JwEU1k20AsGnIqvOaayd+bfCg"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$1xoUuqOFjFFJgwA5:localhost"],"adds_state_event_ids":["$1xoUuqOFjFFJgwA5:localhost"],"removes_state_event_ids":["$j7KtuOzM0K15h3Kr:localhost"],"last_sent_event_id":"$RHNjeYUvXVZfb93t:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hello bob"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/2?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"hello bob","msgtype":"m.text"},"depth":0,"event_id":"$4NBTdIwDxq5fDGpv:localhost","hashes":{"sha256":"msCIESAya8kD7nLCopxkEqrgVuGfrlr9YBIADH5czTA"},"origin":"localhost","origin_server_ts":1494411674630,"prev_events":[["$1xoUuqOFjFFJgwA5:localhost",{"sha256":"ZXj+kY6sqQpf5vsNqvCMSvNoXXKDKxRE4R7+gZD9Tkk"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"bZRT3NxVlfBWw1PxSlKlgfnJixG+NI5H9QmUK2AjECg+l887BZJNCvAK0eD27N8e9V+c2glyXWYje2wexP2CBw"}},"type":"m.room.message"},"latest_event_ids":["$4NBTdIwDxq5fDGpv:localhost"],"last_sent_event_id":"$1xoUuqOFjFFJgwA5:localhost"}}`, - // $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@bob:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"membership":"invite"},"depth":0,"event_id":"$zzLHVlHIWPrnE7DI:localhost","hashes":{"sha256":"LKk7tnYJAHsyffbi9CzfdP+TU4KQ5g6YTgYGKjJ7NxU"},"origin":"localhost","origin_server_ts":1494411709192,"prev_events":[["$4NBTdIwDxq5fDGpv:localhost",{"sha256":"EpqmxEoJP93Zb2Nt2fS95SJWTqqIutHm/Ne8OHqp6Ps"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"GdUzkC+7YKl1XDi7kYuD39yi2L/+nv+YrecIQHS+0BLDQqnEj+iRXfNBuZfTk6lUBCJCHXZlk7MnEIjvWDlZCg"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$zzLHVlHIWPrnE7DI:localhost"],"adds_state_event_ids":["$zzLHVlHIWPrnE7DI:localhost"],"last_sent_event_id":"$4NBTdIwDxq5fDGpv:localhost"}}`, - // $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}],["$zzLHVlHIWPrnE7DI:localhost",{"sha256":"Jw28x9W+GoZYw7sEynsi1fcRzqRQiLddolOa/p26PV0"}]],"content":{"membership":"join"},"unsigned":{"prev_content":{"membership":"invite"},"prev_sender":"@bob:localhost","replaces_state":"$zzLHVlHIWPrnE7DI:localhost"},"depth":0,"event_id":"$uJVKyzZi8ZX0kOd9:localhost","hashes":{"sha256":"9ZZs/Cg0ewpBiCB6iFXXYlmW8koFiesCNGFrOLDTolE"},"origin":"localhost","origin_server_ts":1494411745015,"prev_events":[["$zzLHVlHIWPrnE7DI:localhost",{"sha256":"Jw28x9W+GoZYw7sEynsi1fcRzqRQiLddolOa/p26PV0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@charlie:localhost","signatures":{"localhost":{"ed25519:something":"+TM0gFPM/M3Ji2BjYuTUTgDyCOWlOq8aTMCxLg7EBvS62yPxJ558f13OWWTczUO5aRAt+PvXsMVM/bp8u6c8DQ"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$uJVKyzZi8ZX0kOd9:localhost"],"adds_state_event_ids":["$uJVKyzZi8ZX0kOd9:localhost"],"removes_state_event_ids":["$zzLHVlHIWPrnE7DI:localhost"],"last_sent_event_id":"$zzLHVlHIWPrnE7DI:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"not charlie..."}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"not charlie...","msgtype":"m.text"},"depth":0,"event_id":"$Ixfn5WT9ocWTYxfy:localhost","hashes":{"sha256":"hRChdyMQ3AY4jvrPpI8PEX6Taux83Qo5hdSeHlhPxGo"},"origin":"localhost","origin_server_ts":1494411792737,"prev_events":[["$uJVKyzZi8ZX0kOd9:localhost",{"sha256":"BtesLFnHZOREQCeilFM+xvDU/Wdj+nyHMw7IGTh/9gU"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"LC/Zqwu/XdqjmLdTOp/NQaFaE0niSAGgEpa39gCxsnsqEX80P7P5WDn/Kzx6rjWTnhIszrLsnoycqkXQT0Z4DQ"}},"type":"m.room.message"},"latest_event_ids":["$Ixfn5WT9ocWTYxfy:localhost"],"last_sent_event_id":"$uJVKyzZi8ZX0kOd9:localhost"}}`, - // $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$uJVKyzZi8ZX0kOd9:localhost",{"sha256":"BtesLFnHZOREQCeilFM+xvDU/Wdj+nyHMw7IGTh/9gU"}]],"content":{"membership":"leave"},"unsigned":{"prev_content":{"membership":"join"},"prev_sender":"@charlie:localhost","replaces_state":"$uJVKyzZi8ZX0kOd9:localhost"},"depth":0,"event_id":"$om1F4AI8tCYlHUSp:localhost","hashes":{"sha256":"7JVI0uCxSUyEqDJ+o36/zUIlIZkXVK/R6wkrZGvQXDE"},"origin":"localhost","origin_server_ts":1494411855278,"prev_events":[["$Ixfn5WT9ocWTYxfy:localhost",{"sha256":"hOoPIDQFvvNqQJzA5ggjoQi4v1BOELnhnmwU4UArDOY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"3sxoDLUPnKuDJgFgS3C647BbiXrozxhhxrZOlFP3KgJKzBYv/ht+Jd2V2iSZOvsv94wgRBf0A/lEcJRIqeLgDA"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$om1F4AI8tCYlHUSp:localhost"],"adds_state_event_ids":["$om1F4AI8tCYlHUSp:localhost"],"removes_state_event_ids":["$uJVKyzZi8ZX0kOd9:localhost"],"last_sent_event_id":"$Ixfn5WT9ocWTYxfy:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"why did you kick charlie"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"body":"why did you kick charlie","msgtype":"m.text"},"depth":0,"event_id":"$hgao5gTmr3r9TtK2:localhost","hashes":{"sha256":"Aa2ZCrvwjX5xhvkVqIOFUeEGqrnrQZjjNFiZRybjsPY"},"origin":"localhost","origin_server_ts":1494411912809,"prev_events":[["$om1F4AI8tCYlHUSp:localhost",{"sha256":"yVs+CW7AiJrJOYouL8xPIBrtIHAhnbxaegna8MxeCto"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"sGkpbEXGsvAuCvE3wb5E9H5fjCVKpRdWNt6csj1bCB9Fmg4Rg4mvj3TAJ+91DjO8IPsgSxDKdqqRYF0OtcynBA"}},"type":"m.room.message"},"latest_event_ids":["$hgao5gTmr3r9TtK2:localhost"],"last_sent_event_id":"$om1F4AI8tCYlHUSp:localhost"}}`, - // $ curl -XPUT -d '{"name":"No Charlies"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"No Charlies"},"depth":0,"event_id":"$CY4XDoxjbns3a4Pc:localhost","hashes":{"sha256":"chk72pVkp3AGR2FtdC0mORBWS1b9ePnRN4WK3BP0BiI"},"origin":"localhost","origin_server_ts":1494411959114,"prev_events":[["$hgao5gTmr3r9TtK2:localhost",{"sha256":"/4/OG4Q2YalIeBtN76BEPIieBKA/3UFshR9T+WJip4o"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"mapvA3KJYgw5FmzJMhSFa/+JSuNyv2eKAkiGomAeBB7LQ1e9nK9XhW/Fp7a5Z2Sy2ENwHyd3ij7FEGiLOnSIAw"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$CY4XDoxjbns3a4Pc:localhost"],"adds_state_event_ids":["$CY4XDoxjbns3a4Pc:localhost"],"removes_state_event_ids":["$1xoUuqOFjFFJgwA5:localhost"],"last_sent_event_id":"$hgao5gTmr3r9TtK2:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"whatever"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@bob:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"body":"whatever","msgtype":"m.text"},"depth":0,"event_id":"$pl8VBHRPYDmsnDh4:localhost","hashes":{"sha256":"FYqY9+/cepwIxxjfFV3AjOFBXkTlyEI2jep87dUc+SU"},"origin":"localhost","origin_server_ts":1494411988548,"prev_events":[["$CY4XDoxjbns3a4Pc:localhost",{"sha256":"hCoV63fp8eiquVdEefsOqJtLmJhw4wTlRv+wNTS20Ac"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"sQKwRzE59eZyb8rDySo/pVwZXBh0nA5zx+kjEyXglxIQrTre+8Gj3R7Prni+RE3Dq7oWfKYV7QklTLURAaSICQ"}},"type":"m.room.message"},"latest_event_ids":["$pl8VBHRPYDmsnDh4:localhost"],"last_sent_event_id":"$CY4XDoxjbns3a4Pc:localhost"}}`, - // $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$wPepDhIla765Odre:localhost",{"sha256":"GqUhRiAkRvPrNBDyUxj+emRfK2P8j6iWtvsXDOUltiI"}]],"content":{"membership":"leave"},"depth":0,"event_id":"$acCW4IgnBo8YD3jw:localhost","hashes":{"sha256":"porP+E2yftBGjfS381+WpZeDM9gZHsM3UydlBcRKBLw"},"origin":"localhost","origin_server_ts":1494412037042,"prev_events":[["$pl8VBHRPYDmsnDh4:localhost",{"sha256":"b+qQ380JDFq7quVU9EbIJ2sbpUKM1LAUNX0ZZUoVMZw"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"kxbjTIC0/UR4cOYUAOTNiUc0SSVIF4BY6Rq6IEgYJemq4jcU2fYqum4mFxIQTDKKXMSRHEoNPDmYMFIJwkrsCg"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$acCW4IgnBo8YD3jw:localhost"],"adds_state_event_ids":["$acCW4IgnBo8YD3jw:localhost"],"removes_state_event_ids":["$wPepDhIla765Odre:localhost"],"last_sent_event_id":"$pl8VBHRPYDmsnDh4:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"im alone now"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"im alone now","msgtype":"m.text"},"depth":0,"event_id":"$nYdEXrvTDeb7DfkC:localhost","hashes":{"sha256":"qibC5NmlJpSRMBWSWxy1pv73FXymhPDXQFMmGosfsV0"},"origin":"localhost","origin_server_ts":1494412084668,"prev_events":[["$acCW4IgnBo8YD3jw:localhost",{"sha256":"8h3uXoE6pnI9iLnXI6493qJ0HeuRQfenRIu9PcgH72g"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"EHRoZznhXywhYeIn83o4FSFm3No/aOdLQPHQ68YGtNgESWwpuWLkkGVjoISjz3QgXQ06Fl3cHt7nlTaAHpCNAg"}},"type":"m.room.message"},"latest_event_ids":["$nYdEXrvTDeb7DfkC:localhost"],"last_sent_event_id":"$acCW4IgnBo8YD3jw:localhost"}}`, - // $ curl -XPUT -d '{"membership":"invite"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$acCW4IgnBo8YD3jw:localhost",{"sha256":"8h3uXoE6pnI9iLnXI6493qJ0HeuRQfenRIu9PcgH72g"}]],"content":{"membership":"invite"},"depth":0,"event_id":"$gKNfcXLlWvs2cFad:localhost","hashes":{"sha256":"iYDOUjYkaGSFbVp7TRVFvGJyGMEuBHMQrJ9XqwhzmPI"},"origin":"localhost","origin_server_ts":1494412135845,"prev_events":[["$nYdEXrvTDeb7DfkC:localhost",{"sha256":"83T5Q3+nDvtS0oJTEhHxIw02twBDa1A7QR2bHtnxv1Y"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"ofw009aMJMqVjww9eDXgeTjOQqSlJl/GN/AAb+6mZAPcUI8aVgRlXOSESfhu1ONEuV/yNUycxNXWfMwuvoWsDg"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$gKNfcXLlWvs2cFad:localhost"],"adds_state_event_ids":["$gKNfcXLlWvs2cFad:localhost"],"removes_state_event_ids":["$acCW4IgnBo8YD3jw:localhost"],"last_sent_event_id":"$nYdEXrvTDeb7DfkC:localhost"}}`, - // $ curl -XPUT -d '{"membership":"leave"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@bob:localhost?access_token=@bob:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$gKNfcXLlWvs2cFad:localhost",{"sha256":"/TYIY+L9qjg516Bzl8sadu+Np21KkxE4KdPXALeJ9eE"}]],"content":{"membership":"leave"},"depth":0,"event_id":"$B2q9Tepb6Xc1Rku0:localhost","hashes":{"sha256":"RbHTVdceAEfTALQDZdGrOmakKeTYnChaKjlVuoNUdSY"},"origin":"localhost","origin_server_ts":1494412187614,"prev_events":[["$gKNfcXLlWvs2cFad:localhost",{"sha256":"/TYIY+L9qjg516Bzl8sadu+Np21KkxE4KdPXALeJ9eE"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@bob:localhost","signatures":{"localhost":{"ed25519:something":"dNtUL86j2zUe5+DkfOkil5VujvFZg4FeTjbtcpeF+3E4SUChCAG3lyR6YOAIYBnjtD0/kqT7OcP3pM6vMEp1Aw"}},"state_key":"@bob:localhost","type":"m.room.member"},"latest_event_ids":["$B2q9Tepb6Xc1Rku0:localhost"],"adds_state_event_ids":["$B2q9Tepb6Xc1Rku0:localhost"],"removes_state_event_ids":["$gKNfcXLlWvs2cFad:localhost"],"last_sent_event_id":"$gKNfcXLlWvs2cFad:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"so alone"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"body":"so alone","msgtype":"m.text"},"depth":0,"event_id":"$W1nrYHQIbCTTSJOV:localhost","hashes":{"sha256":"uUKSa4U1coDoT3LUcNF25dt+UpUa2pLXzRJ3ljgxXZs"},"origin":"localhost","origin_server_ts":1494412229742,"prev_events":[["$B2q9Tepb6Xc1Rku0:localhost",{"sha256":"0CLru7nGPgyF9AWlZnarCElscSVrXl2MMY2atrz80Uc"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"YlBJyDnE34UhaCB9hirQN5OySfTDoqiBDnNvxomXjU94z4a8g2CLWKjApwd/q/j4HamCUtjgkjJ2um6hNjsVBA"}},"type":"m.room.message"},"latest_event_ids":["$W1nrYHQIbCTTSJOV:localhost"],"last_sent_event_id":"$B2q9Tepb6Xc1Rku0:localhost"}}`, - // $ curl -XPUT -d '{"name":"Everyone welcome"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.name?access_token=@alice:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$QTen1vksfcRTpUCk:localhost",{"sha256":"znwhbYzdueh0grYkUX4jgXmP9AjKphzyesMZWMiF4IY"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}]],"content":{"name":"Everyone welcome"},"depth":0,"event_id":"$nLzxoBC4A0QRvJ1k:localhost","hashes":{"sha256":"PExCybjaMW1TfgFr57MdIRYJ642FY2jnrdW/tpPOf1Y"},"origin":"localhost","origin_server_ts":1494412294551,"prev_events":[["$W1nrYHQIbCTTSJOV:localhost",{"sha256":"HXk/ACcsiaZ/z1f2aZSIhJF8Ih3BWeh1vp+cV/fwoE0"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@alice:localhost","signatures":{"localhost":{"ed25519:something":"RK09L8sQv78y69PNbOLaX8asq5kp51mbqUuct5gd7ZNmaHKnVds6ew06QEn+gHSDAxqQo2tpcfoajp+yMj1HBw"}},"state_key":"","type":"m.room.name"},"latest_event_ids":["$nLzxoBC4A0QRvJ1k:localhost"],"adds_state_event_ids":["$nLzxoBC4A0QRvJ1k:localhost"],"removes_state_event_ids":["$CY4XDoxjbns3a4Pc:localhost"],"last_sent_event_id":"$W1nrYHQIbCTTSJOV:localhost"}}`, - // $ curl -XPUT -d '{"membership":"join"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/state/m.room.member/@charlie:localhost?access_token=@charlie:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$2O2DpHB37CuwwJOe:localhost",{"sha256":"ulaRD63dbCyolLTwvInIQpcrtU2c7ex/BHmhpLXAUoE"}],["$om1F4AI8tCYlHUSp:localhost",{"sha256":"yVs+CW7AiJrJOYouL8xPIBrtIHAhnbxaegna8MxeCto"}]],"content":{"membership":"join"},"depth":0,"event_id":"$Zo6P8r9bczF6kctV:localhost","hashes":{"sha256":"R3J2iUWnGxVdmly8ah+Dgb5VbJ2i/e8BLaWM0z9eZKU"},"origin":"localhost","origin_server_ts":1494412338689,"prev_events":[["$nLzxoBC4A0QRvJ1k:localhost",{"sha256":"TDcFaArAXpxIJ1noSubcFqkLXiQTrc1Dw1+kgCtx3XY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@charlie:localhost","signatures":{"localhost":{"ed25519:something":"tVnjLVoJ9SLlMQIJSK/6zANWaEu8tVVkx3AEJiC3y5JmhPORb3PyG8eE+e/9hC4aJSQL8LGLaJNWXukMpb2SBg"}},"state_key":"@charlie:localhost","type":"m.room.member"},"latest_event_ids":["$Zo6P8r9bczF6kctV:localhost"],"adds_state_event_ids":["$Zo6P8r9bczF6kctV:localhost"],"removes_state_event_ids":["$om1F4AI8tCYlHUSp:localhost"],"last_sent_event_id":"$nLzxoBC4A0QRvJ1k:localhost"}}`, - // $ curl -XPUT -d '{"msgtype":"m.text","body":"hiiiii"}' "http://localhost:8009/_matrix/client/r0/rooms/%21PjrbIMW2cIiaYF4t:localhost/send/m.room.message/3?access_token=@charlie:localhost" - `{"type":"new_room_event","new_room_event":{"event":{"auth_events":[["$xz0fUB8zNMTGFh1W:localhost",{"sha256":"F4tTLtltC6f2XKeXq4ZKpMZ5EpditaW+RYQSnYzq3lI"}],["$RWsxGlfPHAcijTgu:localhost",{"sha256":"1zc+86U9vLK1BvTJbeLuYpw9dZqvX2fr8rc3pOF69f8"}],["$Zo6P8r9bczF6kctV:localhost",{"sha256":"mnjt3WTYqwtuyl2Fca+0cgm6moHaNL+W9BqRJTQzdEY"}]],"content":{"body":"hiiiii","msgtype":"m.text"},"depth":0,"event_id":"$YAEvK8u2zkTsjf5P:localhost","hashes":{"sha256":"6hKy61h1tuHjYdfpq2MnaPtGEBAZOUz8FLTtxLwjK5A"},"origin":"localhost","origin_server_ts":1494412375465,"prev_events":[["$Zo6P8r9bczF6kctV:localhost",{"sha256":"mnjt3WTYqwtuyl2Fca+0cgm6moHaNL+W9BqRJTQzdEY"}]],"room_id":"!PjrbIMW2cIiaYF4t:localhost","sender":"@charlie:localhost","signatures":{"localhost":{"ed25519:something":"BsSLaMM5U/YkyvBZ00J/+si9My+wAJZOcBhBeato0oHayiag7FW77ZpSTfADazPdNH62kjB0sdP9CN6vQA7yDg"}},"type":"m.room.message"},"latest_event_ids":["$YAEvK8u2zkTsjf5P:localhost"],"last_sent_event_id":"$Zo6P8r9bczF6kctV:localhost"}}`, -} diff --git a/federationapi/routing/join.go b/federationapi/routing/join.go index e9414033..a8f850fb 100644 --- a/federationapi/routing/join.go +++ b/federationapi/routing/join.go @@ -271,17 +271,27 @@ func SendJoin( // Check if the user is already in the room. If they're already in then // there isn't much point in sending another join event into the room. + // Also check to see if they are banned: if they are then we reject them. alreadyJoined := false + isBanned := false for _, se := range stateAndAuthChainResponse.StateEvents { if !se.StateKeyEquals(*event.StateKey()) { continue } if membership, merr := se.Membership(); merr == nil { alreadyJoined = (membership == gomatrixserverlib.Join) + isBanned = (membership == gomatrixserverlib.Ban) break } } + if isBanned { + return util.JSONResponse{ + Code: http.StatusForbidden, + JSON: jsonerror.Forbidden("user is banned"), + } + } + // Send the events to the room server. // We are responsible for notifying other servers that the user has joined // the room, so set SendAsServer to cfg.Matrix.ServerName diff --git a/federationapi/routing/keys.go b/federationapi/routing/keys.go index 1f39094b..d73161e9 100644 --- a/federationapi/routing/keys.go +++ b/federationapi/routing/keys.go @@ -188,40 +188,46 @@ func NotaryKeys( } response.ServerKeys = []json.RawMessage{} - for serverName := range req.ServerKeys { - var keys *gomatrixserverlib.ServerKeys + for serverName, kidToCriteria := range req.ServerKeys { + var keyList []gomatrixserverlib.ServerKeys if serverName == cfg.Matrix.ServerName { if k, err := localKeys(cfg, time.Now().Add(cfg.Matrix.KeyValidityPeriod)); err == nil { - keys = k + keyList = append(keyList, *k) } else { return util.ErrorResponse(err) } } else { - if k, err := fsAPI.GetServerKeys(httpReq.Context(), serverName); err == nil { - keys = &k - } else { + var resp federationSenderAPI.QueryServerKeysResponse + err := fsAPI.QueryServerKeys(httpReq.Context(), &federationSenderAPI.QueryServerKeysRequest{ + ServerName: serverName, + KeyIDToCriteria: kidToCriteria, + }, &resp) + if err != nil { return util.ErrorResponse(err) } + keyList = append(keyList, resp.ServerKeys...) } - if keys == nil { + if len(keyList) == 0 { continue } - j, err := json.Marshal(keys) - if err != nil { - logrus.WithError(err).Errorf("Failed to marshal %q response", serverName) - return jsonerror.InternalServerError() - } + for _, keys := range keyList { + j, err := json.Marshal(keys) + if err != nil { + logrus.WithError(err).Errorf("Failed to marshal %q response", serverName) + return jsonerror.InternalServerError() + } - js, err := gomatrixserverlib.SignJSON( - string(cfg.Matrix.ServerName), cfg.Matrix.KeyID, cfg.Matrix.PrivateKey, j, - ) - if err != nil { - logrus.WithError(err).Errorf("Failed to sign %q response", serverName) - return jsonerror.InternalServerError() - } + js, err := gomatrixserverlib.SignJSON( + string(cfg.Matrix.ServerName), cfg.Matrix.KeyID, cfg.Matrix.PrivateKey, j, + ) + if err != nil { + logrus.WithError(err).Errorf("Failed to sign %q response", serverName) + return jsonerror.InternalServerError() + } - response.ServerKeys = append(response.ServerKeys, js) + response.ServerKeys = append(response.ServerKeys, js) + } } return util.JSONResponse{ diff --git a/federationapi/routing/send.go b/federationapi/routing/send.go index 1c9e72bf..5f214e0f 100644 --- a/federationapi/routing/send.go +++ b/federationapi/routing/send.go @@ -573,6 +573,23 @@ func (t *txnReq) processEvent(ctx context.Context, e *gomatrixserverlib.Event) e logger := util.GetLogger(ctx).WithField("event_id", e.EventID()).WithField("room_id", e.RoomID()) t.work = "" // reset from previous event + // Ask the roomserver if we know about the room and/or if we're joined + // to it. If we aren't then we won't bother processing the event. + joinedReq := api.QueryServerJoinedToRoomRequest{ + RoomID: e.RoomID(), + } + var joinedRes api.QueryServerJoinedToRoomResponse + if err := t.rsAPI.QueryServerJoinedToRoom(ctx, &joinedReq, &joinedRes); err != nil { + return fmt.Errorf("t.rsAPI.QueryServerJoinedToRoom: %w", err) + } + + if !joinedRes.RoomExists || !joinedRes.IsInRoom { + // We don't believe we're a member of this room, therefore there's + // no point in wasting work trying to figure out what to do with + // missing auth or prev events. Drop the event. + return roomNotFoundError{e.RoomID()} + } + // Work out if the roomserver knows everything it needs to know to auth // the event. This includes the prev_events and auth_events. // NOTE! This is going to include prev_events that have an empty state @@ -589,16 +606,6 @@ func (t *txnReq) processEvent(ctx context.Context, e *gomatrixserverlib.Event) e return fmt.Errorf("t.rsAPI.QueryMissingAuthPrevEvents: %w", err) } - if !stateResp.RoomExists { - // TODO: When synapse receives a message for a room it is not in it - // asks the remote server for the state of the room so that it can - // check if the remote server knows of a join "m.room.member" event - // that this server is unaware of. - // However generally speaking we should reject events for rooms we - // aren't a member of. - return roomNotFoundError{e.RoomID()} - } - // Prepare a map of all the events we already had before this point, so // that we don't send them to the roomserver again. for _, eventID := range append(e.AuthEventIDs(), e.PrevEventIDs()...) { diff --git a/federationapi/routing/send_test.go b/federationapi/routing/send_test.go index 98ff1a0a..0da06aa9 100644 --- a/federationapi/routing/send_test.go +++ b/federationapi/routing/send_test.go @@ -190,7 +190,9 @@ func (t *testRoomserverAPI) QueryServerJoinedToRoom( request *api.QueryServerJoinedToRoomRequest, response *api.QueryServerJoinedToRoomResponse, ) error { - return fmt.Errorf("not implemented") + response.RoomExists = true + response.IsInRoom = true + return nil } // Query whether a server is allowed to see an event diff --git a/federationsender/api/api.go b/federationsender/api/api.go index a9ebedaf..82cdf9d8 100644 --- a/federationsender/api/api.go +++ b/federationsender/api/api.go @@ -20,7 +20,6 @@ type FederationClient interface { ClaimKeys(ctx context.Context, s gomatrixserverlib.ServerName, oneTimeKeys map[string]map[string]string) (res gomatrixserverlib.RespClaimKeys, err error) QueryKeys(ctx context.Context, s gomatrixserverlib.ServerName, keys map[string][]string) (res gomatrixserverlib.RespQueryKeys, err error) GetEvent(ctx context.Context, s gomatrixserverlib.ServerName, eventID string) (res gomatrixserverlib.Transaction, err error) - GetServerKeys(ctx context.Context, matrixServer gomatrixserverlib.ServerName) (gomatrixserverlib.ServerKeys, error) MSC2836EventRelationships(ctx context.Context, dst gomatrixserverlib.ServerName, r gomatrixserverlib.MSC2836EventRelationshipsRequest, roomVersion gomatrixserverlib.RoomVersion) (res gomatrixserverlib.MSC2836EventRelationshipsResponse, err error) MSC2946Spaces(ctx context.Context, dst gomatrixserverlib.ServerName, roomID string, r gomatrixserverlib.MSC2946SpacesRequest) (res gomatrixserverlib.MSC2946SpacesResponse, err error) LookupServerKeys(ctx context.Context, s gomatrixserverlib.ServerName, keyRequests map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.Timestamp) ([]gomatrixserverlib.ServerKeys, error) @@ -41,6 +40,8 @@ func (e *FederationClientError) Error() string { type FederationSenderInternalAPI interface { FederationClient + QueryServerKeys(ctx context.Context, request *QueryServerKeysRequest, response *QueryServerKeysResponse) error + // PerformDirectoryLookup looks up a remote room ID from a room alias. PerformDirectoryLookup( ctx context.Context, @@ -94,6 +95,25 @@ type FederationSenderInternalAPI interface { ) error } +type QueryServerKeysRequest struct { + ServerName gomatrixserverlib.ServerName + KeyIDToCriteria map[gomatrixserverlib.KeyID]gomatrixserverlib.PublicKeyNotaryQueryCriteria +} + +func (q *QueryServerKeysRequest) KeyIDs() []gomatrixserverlib.KeyID { + kids := make([]gomatrixserverlib.KeyID, len(q.KeyIDToCriteria)) + i := 0 + for keyID := range q.KeyIDToCriteria { + kids[i] = keyID + i++ + } + return kids +} + +type QueryServerKeysResponse struct { + ServerKeys []gomatrixserverlib.ServerKeys +} + type PerformDirectoryLookupRequest struct { RoomAlias string `json:"room_alias"` ServerName gomatrixserverlib.ServerName `json:"server_name"` diff --git a/federationsender/internal/api.go b/federationsender/internal/api.go index 1de774ef..11032eda 100644 --- a/federationsender/internal/api.go +++ b/federationsender/internal/api.go @@ -202,20 +202,6 @@ func (a *FederationSenderInternalAPI) GetEvent( return ires.(gomatrixserverlib.Transaction), nil } -func (a *FederationSenderInternalAPI) GetServerKeys( - ctx context.Context, s gomatrixserverlib.ServerName, -) (gomatrixserverlib.ServerKeys, error) { - ctx, cancel := context.WithTimeout(ctx, time.Second*30) - defer cancel() - ires, err := a.doRequest(s, func() (interface{}, error) { - return a.federation.GetServerKeys(ctx, s) - }) - if err != nil { - return gomatrixserverlib.ServerKeys{}, err - } - return ires.(gomatrixserverlib.ServerKeys), nil -} - func (a *FederationSenderInternalAPI) LookupServerKeys( ctx context.Context, s gomatrixserverlib.ServerName, keyRequests map[gomatrixserverlib.PublicKeyLookupRequest]gomatrixserverlib.Timestamp, ) ([]gomatrixserverlib.ServerKeys, error) { diff --git a/federationsender/internal/query.go b/federationsender/internal/query.go index 8ba228d1..af531f7d 100644 --- a/federationsender/internal/query.go +++ b/federationsender/internal/query.go @@ -2,8 +2,12 @@ package internal import ( "context" + "fmt" + "time" "github.com/matrix-org/dendrite/federationsender/api" + "github.com/matrix-org/gomatrixserverlib" + "github.com/matrix-org/util" ) // QueryJoinedHostServerNamesInRoom implements api.FederationSenderInternalAPI @@ -20,3 +24,74 @@ func (f *FederationSenderInternalAPI) QueryJoinedHostServerNamesInRoom( return } + +func (a *FederationSenderInternalAPI) fetchServerKeysDirectly(ctx context.Context, serverName gomatrixserverlib.ServerName) (*gomatrixserverlib.ServerKeys, error) { + ctx, cancel := context.WithTimeout(ctx, time.Second*30) + defer cancel() + ires, err := a.doRequest(serverName, func() (interface{}, error) { + return a.federation.GetServerKeys(ctx, serverName) + }) + if err != nil { + return nil, err + } + sks := ires.(gomatrixserverlib.ServerKeys) + return &sks, nil +} + +func (a *FederationSenderInternalAPI) fetchServerKeysFromCache( + ctx context.Context, req *api.QueryServerKeysRequest, +) ([]gomatrixserverlib.ServerKeys, error) { + var results []gomatrixserverlib.ServerKeys + for keyID, criteria := range req.KeyIDToCriteria { + serverKeysResponses, _ := a.db.GetNotaryKeys(ctx, req.ServerName, []gomatrixserverlib.KeyID{keyID}) + if len(serverKeysResponses) == 0 { + return nil, fmt.Errorf("failed to find server key response for key ID %s", keyID) + } + // we should only get 1 result as we only gave 1 key ID + sk := serverKeysResponses[0] + util.GetLogger(ctx).Infof("fetchServerKeysFromCache: minvalid:%v keys: %+v", criteria.MinimumValidUntilTS, sk) + if criteria.MinimumValidUntilTS != 0 { + // check if it's still valid. if they have the same value that's also valid + if sk.ValidUntilTS < criteria.MinimumValidUntilTS { + return nil, fmt.Errorf( + "found server response for key ID %s but it is no longer valid, min: %v valid_until: %v", + keyID, criteria.MinimumValidUntilTS, sk.ValidUntilTS, + ) + } + } + results = append(results, sk) + } + return results, nil +} + +func (a *FederationSenderInternalAPI) QueryServerKeys( + ctx context.Context, req *api.QueryServerKeysRequest, res *api.QueryServerKeysResponse, +) error { + // attempt to satisfy the entire request from the cache first + results, err := a.fetchServerKeysFromCache(ctx, req) + if err == nil { + // satisfied entirely from cache, return it + res.ServerKeys = results + return nil + } + util.GetLogger(ctx).WithField("server", req.ServerName).WithError(err).Warn("notary: failed to satisfy keys request entirely from cache, hitting direct") + + serverKeys, err := a.fetchServerKeysDirectly(ctx, req.ServerName) + if err != nil { + // try to load as much as we can from the cache in a best effort basis + util.GetLogger(ctx).WithField("server", req.ServerName).WithError(err).Warn("notary: failed to ask server for keys, returning best effort keys") + serverKeysResponses, dbErr := a.db.GetNotaryKeys(ctx, req.ServerName, req.KeyIDs()) + if dbErr != nil { + return fmt.Errorf("notary: server returned %s, and db returned %s", err, dbErr) + } + res.ServerKeys = serverKeysResponses + return nil + } + // cache it! + if err = a.db.UpdateNotaryKeys(context.Background(), req.ServerName, *serverKeys); err != nil { + // non-fatal, still return the response + util.GetLogger(ctx).WithError(err).Warn("failed to UpdateNotaryKeys") + } + res.ServerKeys = []gomatrixserverlib.ServerKeys{*serverKeys} + return nil +} diff --git a/federationsender/inthttp/client.go b/federationsender/inthttp/client.go index 3f86a2d0..f08e610a 100644 --- a/federationsender/inthttp/client.go +++ b/federationsender/inthttp/client.go @@ -15,6 +15,7 @@ import ( // HTTP paths for the internal HTTP API const ( FederationSenderQueryJoinedHostServerNamesInRoomPath = "/federationsender/queryJoinedHostServerNamesInRoom" + FederationSenderQueryServerKeysPath = "/federationsender/queryServerKeys" FederationSenderPerformDirectoryLookupRequestPath = "/federationsender/performDirectoryLookup" FederationSenderPerformJoinRequestPath = "/federationsender/performJoinRequest" @@ -31,7 +32,6 @@ const ( FederationSenderLookupStatePath = "/federationsender/client/lookupState" FederationSenderLookupStateIDsPath = "/federationsender/client/lookupStateIDs" FederationSenderGetEventPath = "/federationsender/client/getEvent" - FederationSenderGetServerKeysPath = "/federationsender/client/getServerKeys" FederationSenderLookupServerKeysPath = "/federationsender/client/lookupServerKeys" FederationSenderEventRelationshipsPath = "/federationsender/client/msc2836eventRelationships" FederationSenderSpacesSummaryPath = "/federationsender/client/msc2946spacesSummary" @@ -377,31 +377,14 @@ func (h *httpFederationSenderInternalAPI) GetEvent( return *response.Res, nil } -type getServerKeys struct { - S gomatrixserverlib.ServerName - ServerKeys gomatrixserverlib.ServerKeys - Err *api.FederationClientError -} - -func (h *httpFederationSenderInternalAPI) GetServerKeys( - ctx context.Context, s gomatrixserverlib.ServerName, -) (gomatrixserverlib.ServerKeys, error) { - span, ctx := opentracing.StartSpanFromContext(ctx, "GetServerKeys") +func (h *httpFederationSenderInternalAPI) QueryServerKeys( + ctx context.Context, req *api.QueryServerKeysRequest, res *api.QueryServerKeysResponse, +) error { + span, ctx := opentracing.StartSpanFromContext(ctx, "QueryServerKeys") defer span.Finish() - request := getServerKeys{ - S: s, - } - var response getServerKeys - apiURL := h.federationSenderURL + FederationSenderGetServerKeysPath - err := httputil.PostJSON(ctx, span, h.httpClient, apiURL, &request, &response) - if err != nil { - return gomatrixserverlib.ServerKeys{}, err - } - if response.Err != nil { - return gomatrixserverlib.ServerKeys{}, response.Err - } - return response.ServerKeys, nil + apiURL := h.federationSenderURL + FederationSenderQueryServerKeysPath + return httputil.PostJSON(ctx, span, h.httpClient, apiURL, req, res) } type lookupServerKeys struct { diff --git a/federationsender/inthttp/server.go b/federationsender/inthttp/server.go index be995111..a7fbc4ed 100644 --- a/federationsender/inthttp/server.go +++ b/federationsender/inthttp/server.go @@ -264,25 +264,17 @@ func AddRoutes(intAPI api.FederationSenderInternalAPI, internalAPIMux *mux.Route }), ) internalAPIMux.Handle( - FederationSenderGetServerKeysPath, - httputil.MakeInternalAPI("GetServerKeys", func(req *http.Request) util.JSONResponse { - var request getServerKeys + FederationSenderQueryServerKeysPath, + httputil.MakeInternalAPI("QueryServerKeys", func(req *http.Request) util.JSONResponse { + var request api.QueryServerKeysRequest + var response api.QueryServerKeysResponse if err := json.NewDecoder(req.Body).Decode(&request); err != nil { return util.MessageResponse(http.StatusBadRequest, err.Error()) } - res, err := intAPI.GetServerKeys(req.Context(), request.S) - if err != nil { - ferr, ok := err.(*api.FederationClientError) - if ok { - request.Err = ferr - } else { - request.Err = &api.FederationClientError{ - Err: err.Error(), - } - } + if err := intAPI.QueryServerKeys(req.Context(), &request, &response); err != nil { + return util.ErrorResponse(err) } - request.ServerKeys = res - return util.JSONResponse{Code: http.StatusOK, JSON: request} + return util.JSONResponse{Code: http.StatusOK, JSON: &response} }), ) internalAPIMux.Handle( diff --git a/federationsender/storage/interface.go b/federationsender/storage/interface.go index 9c5ac004..58c8a7cf 100644 --- a/federationsender/storage/interface.go +++ b/federationsender/storage/interface.go @@ -66,4 +66,10 @@ type Database interface { RenewInboundPeek(ctx context.Context, serverName gomatrixserverlib.ServerName, roomID, peekID string, renewalInterval int64) error GetInboundPeek(ctx context.Context, serverName gomatrixserverlib.ServerName, roomID, peekID string) (*types.InboundPeek, error) GetInboundPeeks(ctx context.Context, roomID string) ([]types.InboundPeek, error) + + // Update the notary with the given server keys from the given server name. + UpdateNotaryKeys(ctx context.Context, serverName gomatrixserverlib.ServerName, serverKeys gomatrixserverlib.ServerKeys) error + // Query the notary for the server keys for the given server. If `optKeyIDs` is not empty, multiple server keys may be returned (between 1 - len(optKeyIDs)) + // such that the combination of all server keys will include all the `optKeyIDs`. + GetNotaryKeys(ctx context.Context, serverName gomatrixserverlib.ServerName, optKeyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error) } diff --git a/federationsender/storage/postgres/notary_server_keys_json_table.go b/federationsender/storage/postgres/notary_server_keys_json_table.go new file mode 100644 index 00000000..42e58ba7 --- /dev/null +++ b/federationsender/storage/postgres/notary_server_keys_json_table.go @@ -0,0 +1,64 @@ +// Copyright 2021 The Matrix.org Foundation C.I.C. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package postgres + +import ( + "context" + "database/sql" + + "github.com/matrix-org/dendrite/federationsender/storage/tables" + "github.com/matrix-org/gomatrixserverlib" +) + +const notaryServerKeysJSONSchema = ` +CREATE SEQUENCE IF NOT EXISTS federationsender_notary_server_keys_json_pkey; +CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_json ( + notary_id BIGINT PRIMARY KEY NOT NULL DEFAULT nextval('federationsender_notary_server_keys_json_pkey'), + response_json TEXT NOT NULL, + server_name TEXT NOT NULL, + valid_until BIGINT NOT NULL +); +` + +const insertServerKeysJSONSQL = "" + + "INSERT INTO federationsender_notary_server_keys_json (response_json, server_name, valid_until) VALUES ($1, $2, $3)" + + " RETURNING notary_id" + +type notaryServerKeysStatements struct { + db *sql.DB + insertServerKeysJSONStmt *sql.Stmt +} + +func NewPostgresNotaryServerKeysTable(db *sql.DB) (s *notaryServerKeysStatements, err error) { + s = ¬aryServerKeysStatements{ + db: db, + } + _, err = db.Exec(notaryServerKeysJSONSchema) + if err != nil { + return + } + + if s.insertServerKeysJSONStmt, err = db.Prepare(insertServerKeysJSONSQL); err != nil { + return + } + return +} + +func (s *notaryServerKeysStatements) InsertJSONResponse( + ctx context.Context, txn *sql.Tx, keyQueryResponseJSON gomatrixserverlib.ServerKeys, serverName gomatrixserverlib.ServerName, validUntil gomatrixserverlib.Timestamp, +) (tables.NotaryID, error) { + var notaryID tables.NotaryID + return notaryID, txn.Stmt(s.insertServerKeysJSONStmt).QueryRowContext(ctx, string(keyQueryResponseJSON.Raw), serverName, validUntil).Scan(¬aryID) +} diff --git a/federationsender/storage/postgres/notary_server_keys_metadata_table.go b/federationsender/storage/postgres/notary_server_keys_metadata_table.go new file mode 100644 index 00000000..b460dcd8 --- /dev/null +++ b/federationsender/storage/postgres/notary_server_keys_metadata_table.go @@ -0,0 +1,167 @@ +// Copyright 2021 The Matrix.org Foundation C.I.C. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package postgres + +import ( + "context" + "database/sql" + "encoding/json" + + "github.com/lib/pq" + "github.com/matrix-org/dendrite/federationsender/storage/tables" + "github.com/matrix-org/dendrite/internal" + "github.com/matrix-org/gomatrixserverlib" +) + +const notaryServerKeysMetadataSchema = ` +CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_metadata ( + notary_id BIGINT NOT NULL, + server_name TEXT NOT NULL, + key_id TEXT NOT NULL, + UNIQUE (server_name, key_id) +); +` + +const upsertServerKeysSQL = "" + + "INSERT INTO federationsender_notary_server_keys_metadata (notary_id, server_name, key_id) VALUES ($1, $2, $3)" + + " ON CONFLICT (server_name, key_id) DO UPDATE SET notary_id = $1" + +// for a given (server_name, key_id), find the existing notary ID and valid until. Used to check if we will replace it +// JOINs with the json table +const selectNotaryKeyMetadataSQL = ` + SELECT federationsender_notary_server_keys_metadata.notary_id, valid_until FROM federationsender_notary_server_keys_json + JOIN federationsender_notary_server_keys_metadata ON + federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id + WHERE federationsender_notary_server_keys_metadata.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id = $2 +` + +// select the response which has the highest valid_until value +// JOINs with the json table +const selectNotaryKeyResponsesSQL = ` + SELECT response_json FROM federationsender_notary_server_keys_json + WHERE server_name = $1 AND valid_until = ( + SELECT MAX(valid_until) FROM federationsender_notary_server_keys_json WHERE server_name = $1 + ) +` + +// select the responses which have the given key IDs +// JOINs with the json table +const selectNotaryKeyResponsesWithKeyIDsSQL = ` + SELECT response_json FROM federationsender_notary_server_keys_json + JOIN federationsender_notary_server_keys_metadata ON + federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id + WHERE federationsender_notary_server_keys_json.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id = ANY ($2) + GROUP BY federationsender_notary_server_keys_json.notary_id +` + +// JOINs with the metadata table +const deleteUnusedServerKeysJSONSQL = ` + DELETE FROM federationsender_notary_server_keys_json WHERE federationsender_notary_server_keys_json.notary_id NOT IN ( + SELECT DISTINCT notary_id FROM federationsender_notary_server_keys_metadata + ) +` + +type notaryServerKeysMetadataStatements struct { + db *sql.DB + upsertServerKeysStmt *sql.Stmt + selectNotaryKeyResponsesStmt *sql.Stmt + selectNotaryKeyResponsesWithKeyIDsStmt *sql.Stmt + selectNotaryKeyMetadataStmt *sql.Stmt + deleteUnusedServerKeysJSONStmt *sql.Stmt +} + +func NewPostgresNotaryServerKeysMetadataTable(db *sql.DB) (s *notaryServerKeysMetadataStatements, err error) { + s = ¬aryServerKeysMetadataStatements{ + db: db, + } + _, err = db.Exec(notaryServerKeysMetadataSchema) + if err != nil { + return + } + + if s.upsertServerKeysStmt, err = db.Prepare(upsertServerKeysSQL); err != nil { + return + } + if s.selectNotaryKeyResponsesStmt, err = db.Prepare(selectNotaryKeyResponsesSQL); err != nil { + return + } + if s.selectNotaryKeyResponsesWithKeyIDsStmt, err = db.Prepare(selectNotaryKeyResponsesWithKeyIDsSQL); err != nil { + return + } + if s.selectNotaryKeyMetadataStmt, err = db.Prepare(selectNotaryKeyMetadataSQL); err != nil { + return + } + if s.deleteUnusedServerKeysJSONStmt, err = db.Prepare(deleteUnusedServerKeysJSONSQL); err != nil { + return + } + return +} + +func (s *notaryServerKeysMetadataStatements) UpsertKey( + ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyID gomatrixserverlib.KeyID, newNotaryID tables.NotaryID, newValidUntil gomatrixserverlib.Timestamp, +) (tables.NotaryID, error) { + notaryID := newNotaryID + // see if the existing notary ID a) exists, b) has a longer valid_until + var existingNotaryID tables.NotaryID + var existingValidUntil gomatrixserverlib.Timestamp + if err := txn.Stmt(s.selectNotaryKeyMetadataStmt).QueryRowContext(ctx, serverName, keyID).Scan(&existingNotaryID, &existingValidUntil); err != nil { + if err != sql.ErrNoRows { + return 0, err + } + } + if existingValidUntil.Time().After(newValidUntil.Time()) { + // the existing valid_until is valid longer, so use that. + return existingNotaryID, nil + } + // overwrite the notary_id for this (server_name, key_id) tuple + _, err := txn.Stmt(s.upsertServerKeysStmt).ExecContext(ctx, notaryID, serverName, keyID) + return notaryID, err +} + +func (s *notaryServerKeysMetadataStatements) SelectKeys(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error) { + var rows *sql.Rows + var err error + if len(keyIDs) == 0 { + rows, err = txn.Stmt(s.selectNotaryKeyResponsesStmt).QueryContext(ctx, string(serverName)) + } else { + keyIDstr := make([]string, len(keyIDs)) + for i := range keyIDs { + keyIDstr[i] = string(keyIDs[i]) + } + rows, err = txn.Stmt(s.selectNotaryKeyResponsesWithKeyIDsStmt).QueryContext(ctx, string(serverName), pq.StringArray(keyIDstr)) + } + if err != nil { + return nil, err + } + defer internal.CloseAndLogIfError(ctx, rows, "selectNotaryKeyResponsesStmt close failed") + var results []gomatrixserverlib.ServerKeys + for rows.Next() { + var sk gomatrixserverlib.ServerKeys + var raw string + if err = rows.Scan(&raw); err != nil { + return nil, err + } + if err = json.Unmarshal([]byte(raw), &sk); err != nil { + return nil, err + } + results = append(results, sk) + } + return results, nil +} + +func (s *notaryServerKeysMetadataStatements) DeleteOldJSONResponses(ctx context.Context, txn *sql.Tx) error { + _, err := txn.Stmt(s.deleteUnusedServerKeysJSONStmt).ExecContext(ctx) + return err +} diff --git a/federationsender/storage/postgres/storage.go b/federationsender/storage/postgres/storage.go index 5edc08ad..5507bad7 100644 --- a/federationsender/storage/postgres/storage.go +++ b/federationsender/storage/postgres/storage.go @@ -17,6 +17,7 @@ package postgres import ( "database/sql" + "fmt" "github.com/matrix-org/dendrite/federationsender/storage/postgres/deltas" "github.com/matrix-org/dendrite/federationsender/storage/shared" @@ -69,6 +70,14 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS if err != nil { return nil, err } + notaryJSON, err := NewPostgresNotaryServerKeysTable(d.db) + if err != nil { + return nil, fmt.Errorf("NewPostgresNotaryServerKeysTable: %s", err) + } + notaryMetadata, err := NewPostgresNotaryServerKeysMetadataTable(d.db) + if err != nil { + return nil, fmt.Errorf("NewPostgresNotaryServerKeysMetadataTable: %s", err) + } m := sqlutil.NewMigrations() deltas.LoadRemoveRoomsTable(m) if err = m.RunDeltas(d.db, dbProperties); err != nil { @@ -85,6 +94,8 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS FederationSenderBlacklist: blacklist, FederationSenderInboundPeeks: inboundPeeks, FederationSenderOutboundPeeks: outboundPeeks, + NotaryServerKeysJSON: notaryJSON, + NotaryServerKeysMetadata: notaryMetadata, } if err = d.PartitionOffsetStatements.Prepare(d.db, d.writer, "federationsender"); err != nil { return nil, err diff --git a/federationsender/storage/shared/storage.go b/federationsender/storage/shared/storage.go index 2d409922..45c9febd 100644 --- a/federationsender/storage/shared/storage.go +++ b/federationsender/storage/shared/storage.go @@ -18,6 +18,7 @@ import ( "context" "database/sql" "fmt" + "time" "github.com/matrix-org/dendrite/federationsender/storage/tables" "github.com/matrix-org/dendrite/federationsender/types" @@ -37,6 +38,8 @@ type Database struct { FederationSenderBlacklist tables.FederationSenderBlacklist FederationSenderOutboundPeeks tables.FederationSenderOutboundPeeks FederationSenderInboundPeeks tables.FederationSenderInboundPeeks + NotaryServerKeysJSON tables.FederationSenderNotaryServerKeysJSON + NotaryServerKeysMetadata tables.FederationSenderNotaryServerKeysMetadata } // An Receipt contains the NIDs of a call to GetNextTransactionPDUs/EDUs. @@ -197,3 +200,47 @@ func (d *Database) GetInboundPeek(ctx context.Context, serverName gomatrixserver func (d *Database) GetInboundPeeks(ctx context.Context, roomID string) ([]types.InboundPeek, error) { return d.FederationSenderInboundPeeks.SelectInboundPeeks(ctx, nil, roomID) } + +func (d *Database) UpdateNotaryKeys(ctx context.Context, serverName gomatrixserverlib.ServerName, serverKeys gomatrixserverlib.ServerKeys) error { + return d.Writer.Do(d.DB, nil, func(txn *sql.Tx) error { + validUntil := serverKeys.ValidUntilTS + // Servers MUST use the lesser of this field and 7 days into the future when determining if a key is valid. + // This is to avoid a situation where an attacker publishes a key which is valid for a significant amount of + // time without a way for the homeserver owner to revoke it. + // https://spec.matrix.org/unstable/server-server-api/#querying-keys-through-another-server + weekIntoFuture := time.Now().Add(7 * 24 * time.Hour) + if weekIntoFuture.Before(validUntil.Time()) { + validUntil = gomatrixserverlib.AsTimestamp(weekIntoFuture) + } + notaryID, err := d.NotaryServerKeysJSON.InsertJSONResponse(ctx, txn, serverKeys, serverName, validUntil) + if err != nil { + return err + } + // update the metadata for the keys + for keyID := range serverKeys.OldVerifyKeys { + _, err = d.NotaryServerKeysMetadata.UpsertKey(ctx, txn, serverName, keyID, notaryID, validUntil) + if err != nil { + return err + } + } + for keyID := range serverKeys.VerifyKeys { + _, err = d.NotaryServerKeysMetadata.UpsertKey(ctx, txn, serverName, keyID, notaryID, validUntil) + if err != nil { + return err + } + } + + // clean up old responses + return d.NotaryServerKeysMetadata.DeleteOldJSONResponses(ctx, txn) + }) +} + +func (d *Database) GetNotaryKeys( + ctx context.Context, serverName gomatrixserverlib.ServerName, optKeyIDs []gomatrixserverlib.KeyID, +) (sks []gomatrixserverlib.ServerKeys, err error) { + err = d.Writer.Do(d.DB, nil, func(txn *sql.Tx) error { + sks, err = d.NotaryServerKeysMetadata.SelectKeys(ctx, txn, serverName, optKeyIDs) + return err + }) + return sks, err +} diff --git a/federationsender/storage/sqlite3/notary_server_keys_json_table.go b/federationsender/storage/sqlite3/notary_server_keys_json_table.go new file mode 100644 index 00000000..6990036a --- /dev/null +++ b/federationsender/storage/sqlite3/notary_server_keys_json_table.go @@ -0,0 +1,63 @@ +// Copyright 2021 The Matrix.org Foundation C.I.C. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package sqlite3 + +import ( + "context" + "database/sql" + + "github.com/matrix-org/dendrite/federationsender/storage/tables" + "github.com/matrix-org/gomatrixserverlib" +) + +const notaryServerKeysJSONSchema = ` +CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_json ( + notary_id INTEGER PRIMARY KEY AUTOINCREMENT, + response_json TEXT NOT NULL, + server_name TEXT NOT NULL, + valid_until BIGINT NOT NULL +); +` + +const insertServerKeysJSONSQL = "" + + "INSERT INTO federationsender_notary_server_keys_json (response_json, server_name, valid_until) VALUES ($1, $2, $3)" + + " RETURNING notary_id" + +type notaryServerKeysStatements struct { + db *sql.DB + insertServerKeysJSONStmt *sql.Stmt +} + +func NewSQLiteNotaryServerKeysTable(db *sql.DB) (s *notaryServerKeysStatements, err error) { + s = ¬aryServerKeysStatements{ + db: db, + } + _, err = db.Exec(notaryServerKeysJSONSchema) + if err != nil { + return + } + + if s.insertServerKeysJSONStmt, err = db.Prepare(insertServerKeysJSONSQL); err != nil { + return + } + return +} + +func (s *notaryServerKeysStatements) InsertJSONResponse( + ctx context.Context, txn *sql.Tx, keyQueryResponseJSON gomatrixserverlib.ServerKeys, serverName gomatrixserverlib.ServerName, validUntil gomatrixserverlib.Timestamp, +) (tables.NotaryID, error) { + var notaryID tables.NotaryID + return notaryID, txn.Stmt(s.insertServerKeysJSONStmt).QueryRowContext(ctx, string(keyQueryResponseJSON.Raw), serverName, validUntil).Scan(¬aryID) +} diff --git a/federationsender/storage/sqlite3/notary_server_keys_metadata_table.go b/federationsender/storage/sqlite3/notary_server_keys_metadata_table.go new file mode 100644 index 00000000..a2959407 --- /dev/null +++ b/federationsender/storage/sqlite3/notary_server_keys_metadata_table.go @@ -0,0 +1,169 @@ +// Copyright 2021 The Matrix.org Foundation C.I.C. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package sqlite3 + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "strings" + + "github.com/matrix-org/dendrite/federationsender/storage/tables" + "github.com/matrix-org/dendrite/internal" + "github.com/matrix-org/dendrite/internal/sqlutil" + "github.com/matrix-org/gomatrixserverlib" +) + +const notaryServerKeysMetadataSchema = ` +CREATE TABLE IF NOT EXISTS federationsender_notary_server_keys_metadata ( + notary_id BIGINT NOT NULL, + server_name TEXT NOT NULL, + key_id TEXT NOT NULL, + UNIQUE (server_name, key_id) +); +` + +const upsertServerKeysSQL = "" + + "INSERT INTO federationsender_notary_server_keys_metadata (notary_id, server_name, key_id) VALUES ($1, $2, $3)" + + " ON CONFLICT (server_name, key_id) DO UPDATE SET notary_id = $1" + +// for a given (server_name, key_id), find the existing notary ID and valid until. Used to check if we will replace it +// JOINs with the json table +const selectNotaryKeyMetadataSQL = ` + SELECT federationsender_notary_server_keys_metadata.notary_id, valid_until FROM federationsender_notary_server_keys_json + JOIN federationsender_notary_server_keys_metadata ON + federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id + WHERE federationsender_notary_server_keys_metadata.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id = $2 +` + +// select the response which has the highest valid_until value +// JOINs with the json table +const selectNotaryKeyResponsesSQL = ` + SELECT response_json FROM federationsender_notary_server_keys_json + WHERE server_name = $1 AND valid_until = ( + SELECT MAX(valid_until) FROM federationsender_notary_server_keys_json WHERE server_name = $1 + ) +` + +// select the responses which have the given key IDs +// JOINs with the json table +const selectNotaryKeyResponsesWithKeyIDsSQL = ` + SELECT response_json FROM federationsender_notary_server_keys_json + JOIN federationsender_notary_server_keys_metadata ON + federationsender_notary_server_keys_metadata.notary_id = federationsender_notary_server_keys_json.notary_id + WHERE federationsender_notary_server_keys_json.server_name = $1 AND federationsender_notary_server_keys_metadata.key_id IN ($2) + GROUP BY federationsender_notary_server_keys_json.notary_id +` + +// JOINs with the metadata table +const deleteUnusedServerKeysJSONSQL = ` + DELETE FROM federationsender_notary_server_keys_json WHERE federationsender_notary_server_keys_json.notary_id NOT IN ( + SELECT DISTINCT notary_id FROM federationsender_notary_server_keys_metadata + ) +` + +type notaryServerKeysMetadataStatements struct { + db *sql.DB + upsertServerKeysStmt *sql.Stmt + selectNotaryKeyResponsesStmt *sql.Stmt + selectNotaryKeyMetadataStmt *sql.Stmt + deleteUnusedServerKeysJSONStmt *sql.Stmt +} + +func NewSQLiteNotaryServerKeysMetadataTable(db *sql.DB) (s *notaryServerKeysMetadataStatements, err error) { + s = ¬aryServerKeysMetadataStatements{ + db: db, + } + _, err = db.Exec(notaryServerKeysMetadataSchema) + if err != nil { + return + } + + if s.upsertServerKeysStmt, err = db.Prepare(upsertServerKeysSQL); err != nil { + return + } + if s.selectNotaryKeyResponsesStmt, err = db.Prepare(selectNotaryKeyResponsesSQL); err != nil { + return + } + if s.selectNotaryKeyMetadataStmt, err = db.Prepare(selectNotaryKeyMetadataSQL); err != nil { + return + } + if s.deleteUnusedServerKeysJSONStmt, err = db.Prepare(deleteUnusedServerKeysJSONSQL); err != nil { + return + } + return +} + +func (s *notaryServerKeysMetadataStatements) UpsertKey( + ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyID gomatrixserverlib.KeyID, newNotaryID tables.NotaryID, newValidUntil gomatrixserverlib.Timestamp, +) (tables.NotaryID, error) { + notaryID := newNotaryID + // see if the existing notary ID a) exists, b) has a longer valid_until + var existingNotaryID tables.NotaryID + var existingValidUntil gomatrixserverlib.Timestamp + if err := txn.Stmt(s.selectNotaryKeyMetadataStmt).QueryRowContext(ctx, serverName, keyID).Scan(&existingNotaryID, &existingValidUntil); err != nil { + if err != sql.ErrNoRows { + return 0, err + } + } + if existingValidUntil.Time().After(newValidUntil.Time()) { + // the existing valid_until is valid longer, so use that. + return existingNotaryID, nil + } + // overwrite the notary_id for this (server_name, key_id) tuple + _, err := txn.Stmt(s.upsertServerKeysStmt).ExecContext(ctx, notaryID, serverName, keyID) + return notaryID, err +} + +func (s *notaryServerKeysMetadataStatements) SelectKeys(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error) { + var rows *sql.Rows + var err error + if len(keyIDs) == 0 { + rows, err = txn.Stmt(s.selectNotaryKeyResponsesStmt).QueryContext(ctx, string(serverName)) + } else { + iKeyIDs := make([]interface{}, len(keyIDs)+1) + iKeyIDs[0] = serverName + for i := range keyIDs { + iKeyIDs[i+1] = string(keyIDs[i]) + } + sql := strings.Replace(selectNotaryKeyResponsesWithKeyIDsSQL, "($2)", sqlutil.QueryVariadicOffset(len(keyIDs), 1), 1) + fmt.Println(sql) + fmt.Println(iKeyIDs...) + rows, err = s.db.QueryContext(ctx, sql, iKeyIDs...) + } + if err != nil { + return nil, err + } + defer internal.CloseAndLogIfError(ctx, rows, "selectNotaryKeyResponsesStmt close failed") + var results []gomatrixserverlib.ServerKeys + for rows.Next() { + var sk gomatrixserverlib.ServerKeys + var raw string + if err = rows.Scan(&raw); err != nil { + return nil, err + } + if err = json.Unmarshal([]byte(raw), &sk); err != nil { + return nil, err + } + results = append(results, sk) + } + return results, nil +} + +func (s *notaryServerKeysMetadataStatements) DeleteOldJSONResponses(ctx context.Context, txn *sql.Tx) error { + _, err := txn.Stmt(s.deleteUnusedServerKeysJSONStmt).ExecContext(ctx) + return err +} diff --git a/federationsender/storage/sqlite3/storage.go b/federationsender/storage/sqlite3/storage.go index 84a9ff86..c5d1d1ac 100644 --- a/federationsender/storage/sqlite3/storage.go +++ b/federationsender/storage/sqlite3/storage.go @@ -71,6 +71,14 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS if err != nil { return nil, err } + notaryKeys, err := NewSQLiteNotaryServerKeysTable(d.db) + if err != nil { + return nil, err + } + notaryKeysMetadata, err := NewSQLiteNotaryServerKeysMetadataTable(d.db) + if err != nil { + return nil, err + } m := sqlutil.NewMigrations() deltas.LoadRemoveRoomsTable(m) if err = m.RunDeltas(d.db, dbProperties); err != nil { @@ -87,6 +95,8 @@ func NewDatabase(dbProperties *config.DatabaseOptions, cache caching.FederationS FederationSenderBlacklist: blacklist, FederationSenderOutboundPeeks: outboundPeeks, FederationSenderInboundPeeks: inboundPeeks, + NotaryServerKeysJSON: notaryKeys, + NotaryServerKeysMetadata: notaryKeysMetadata, } if err = d.PartitionOffsetStatements.Prepare(d.db, d.writer, "federationsender"); err != nil { return nil, err diff --git a/federationsender/storage/tables/interface.go b/federationsender/storage/tables/interface.go index 995b6f47..663a4cb2 100644 --- a/federationsender/storage/tables/interface.go +++ b/federationsender/storage/tables/interface.go @@ -22,6 +22,8 @@ import ( "github.com/matrix-org/gomatrixserverlib" ) +type NotaryID int64 + type FederationSenderQueuePDUs interface { InsertQueuePDU(ctx context.Context, txn *sql.Tx, transactionID gomatrixserverlib.TransactionID, serverName gomatrixserverlib.ServerName, nid int64) error DeleteQueuePDUs(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, jsonNIDs []int64) error @@ -80,3 +82,25 @@ type FederationSenderInboundPeeks interface { DeleteInboundPeek(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, roomID, peekID string) (err error) DeleteInboundPeeks(ctx context.Context, txn *sql.Tx, roomID string) (err error) } + +// FederationSenderNotaryServerKeysJSON contains the byte-for-byte responses from servers which contain their keys and is signed by them. +type FederationSenderNotaryServerKeysJSON interface { + // InsertJSONResponse inserts a new response JSON. Useless on its own, needs querying via FederationSenderNotaryServerKeysMetadata + // `validUntil` should be the value of `valid_until_ts` with the 7-day check applied from: + // "Servers MUST use the lesser of this field and 7 days into the future when determining if a key is valid. + // This is to avoid a situation where an attacker publishes a key which is valid for a significant amount of time + // without a way for the homeserver owner to revoke it."" + InsertJSONResponse(ctx context.Context, txn *sql.Tx, keyQueryResponseJSON gomatrixserverlib.ServerKeys, serverName gomatrixserverlib.ServerName, validUntil gomatrixserverlib.Timestamp) (NotaryID, error) +} + +// FederationSenderNotaryServerKeysMetadata persists the metadata for FederationSenderNotaryServerKeysJSON +type FederationSenderNotaryServerKeysMetadata interface { + // UpsertKey updates or inserts a (server_name, key_id) tuple, pointing it via NotaryID at the the response which has the longest valid_until_ts + // `newNotaryID` and `newValidUntil` should be the notary ID / valid_until which has this (server_name, key_id) tuple already, e.g one you just inserted. + UpsertKey(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyID gomatrixserverlib.KeyID, newNotaryID NotaryID, newValidUntil gomatrixserverlib.Timestamp) (NotaryID, error) + // SelectKeys returns the signed JSON objects which contain the given key IDs. This will be at most the length of `keyIDs` and at least 1 (assuming + // the keys exist in the first place). If `keyIDs` is empty, the signed JSON object with the longest valid_until_ts will be returned. + SelectKeys(ctx context.Context, txn *sql.Tx, serverName gomatrixserverlib.ServerName, keyIDs []gomatrixserverlib.KeyID) ([]gomatrixserverlib.ServerKeys, error) + // DeleteOldJSONResponses removes all responses which are not referenced in FederationSenderNotaryServerKeysMetadata + DeleteOldJSONResponses(ctx context.Context, txn *sql.Tx) error +} diff --git a/go.mod b/go.mod index 88a4a08c..a0816012 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,7 @@ module github.com/matrix-org/dendrite require ( + github.com/Arceliar/ironwood v0.0.0-20210619124114-6ad55cae5031 github.com/DATA-DOG/go-sqlmock v1.5.0 github.com/HdrHistogram/hdrhistogram-go v1.0.1 // indirect github.com/Masterminds/semver/v3 v3.1.1 @@ -31,31 +32,34 @@ require ( github.com/matrix-org/dugong v0.0.0-20180820122854-51a565b5666b github.com/matrix-org/go-http-js-libp2p v0.0.0-20200518170932-783164aeeda4 github.com/matrix-org/go-sqlite3-js v0.0.0-20210625141222-bd2b7124cee8 - github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd - github.com/matrix-org/gomatrixserverlib v0.0.0-20210702152949-0cac5159e7d6 + github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 + github.com/matrix-org/gomatrixserverlib v0.0.0-20210714141824-52d282133140 github.com/matrix-org/naffka v0.0.0-20210623111924-14ff508b58e0 github.com/matrix-org/pinecone v0.0.0-20210623102758-74f885644c1b github.com/matrix-org/util v0.0.0-20200807132607-55161520e1d4 github.com/mattn/go-sqlite3 v1.14.7-0.20210414154423-1157a4212dcb github.com/morikuni/aec v1.0.0 // indirect github.com/nats-io/nats.go v1.11.0 + github.com/neilalexander/utp v0.1.1-0.20210705212447-691f29ad692b github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 github.com/ngrok/sqlmw v0.0.0-20200129213757-d5c93a81bec6 github.com/opentracing/opentracing-go v1.2.0 + github.com/patrickmn/go-cache v2.1.0+incompatible github.com/pierrec/lz4 v2.6.1+incompatible // indirect github.com/pkg/errors v0.9.1 github.com/pressly/goose v2.7.0+incompatible github.com/prometheus/client_golang v1.9.0 - github.com/sirupsen/logrus v1.8.0 - github.com/tidwall/gjson v1.6.8 - github.com/tidwall/sjson v1.1.5 + github.com/sirupsen/logrus v1.8.1 + github.com/tidwall/gjson v1.8.1 + github.com/tidwall/pretty v1.2.0 // indirect + github.com/tidwall/sjson v1.1.7 github.com/uber/jaeger-client-go v2.25.0+incompatible github.com/uber/jaeger-lib v2.4.0+incompatible - github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20210218094457-e77ca8019daa + github.com/yggdrasil-network/yggdrasil-go v0.4.1-0.20210715083903-52309d094c00 go.uber.org/atomic v1.7.0 golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a golang.org/x/mobile v0.0.0-20210220033013-bdb1ca9a1e08 - golang.org/x/net v0.0.0-20210525063256-abc453219eb5 + golang.org/x/net v0.0.0-20210610132358-84b48f89b13b golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 gopkg.in/h2non/bimg.v1 v1.1.5 gopkg.in/yaml.v2 v2.4.0 diff --git a/go.sum b/go.sum index 959898ce..6cb58d93 100644 --- a/go.sum +++ b/go.sum @@ -31,8 +31,10 @@ dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg= github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= -github.com/Arceliar/phony v0.0.0-20191006174943-d0c68492aca0 h1:p3puK8Sl2xK+2FnnIvY/C0N1aqJo2kbEsdAzU+Tnv48= -github.com/Arceliar/phony v0.0.0-20191006174943-d0c68492aca0/go.mod h1:6Lkn+/zJilRMsKmbmG1RPoamiArC6HS73xbwRyp3UyI= +github.com/Arceliar/ironwood v0.0.0-20210619124114-6ad55cae5031 h1:DZVDfYhVdu+0wAiRHoY1olyNkKxIot9UjBnbQFzuUlM= +github.com/Arceliar/ironwood v0.0.0-20210619124114-6ad55cae5031/go.mod h1:RP72rucOFm5udrnEzTmIWLRVGQiV/fSUAQXJ0RST/nk= +github.com/Arceliar/phony v0.0.0-20210209235338-dde1a8dca979 h1:WndgpSW13S32VLQ3ugUxx2EnnWmgba1kCqPkd4Gk1yQ= +github.com/Arceliar/phony v0.0.0-20210209235338-dde1a8dca979/go.mod h1:6Lkn+/zJilRMsKmbmG1RPoamiArC6HS73xbwRyp3UyI= github.com/Azure/azure-sdk-for-go v16.2.1+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc= github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8= github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8= @@ -97,6 +99,7 @@ github.com/Shopify/sarama v1.29.0/go.mod h1:2QpgD79wpdAESqNQMxNc0KYMkycd4slxGdV3 github.com/Shopify/toxiproxy v2.1.4+incompatible h1:TKdv8HiTLgE5wdJuEML90aBgNWsokNbMijUGhmcoBJc= github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI= github.com/VividCortex/ewma v1.1.1/go.mod h1:2Tkkvm3sRDVXaiyucHiACn4cqf7DpdyLvmxzcbUokwA= +github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4= github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5/go.mod h1:SkGFH1ia65gfNATL8TAiHDNxPzPdmEL5uirI2Uyuz6c= @@ -117,7 +120,9 @@ github.com/anacrolix/log v0.3.0/go.mod h1:lWvLTqzAnCWPJA08T2HCstZi0L1y2Wyvm3FJgw github.com/anacrolix/missinggo v1.1.2-0.20190815015349-b888af804467/go.mod h1:MBJu3Sk/k3ZfGYcS7z18gwfu72Ey/xopPFJJbTi5yIo= github.com/anacrolix/missinggo v1.2.1 h1:0IE3TqX5y5D0IxeMwTyIgqdDew4QrzcXaaEnJQyjHvw= github.com/anacrolix/missinggo v1.2.1/go.mod h1:J5cMhif8jPmFoC3+Uvob3OXXNIhOUikzMt+uUjeM21Y= +github.com/anacrolix/missinggo/perf v1.0.0 h1:7ZOGYziGEBytW49+KmYGTaNfnwUqP1HBsy6BqESAJVw= github.com/anacrolix/missinggo/perf v1.0.0/go.mod h1:ljAFWkBuzkO12MQclXzZrosP5urunoLS0Cbvb4V0uMQ= +github.com/anacrolix/sync v0.2.0 h1:oRe22/ZB+v7v/5Mbc4d2zE0AXEZy0trKyKLjqYOt6tY= github.com/anacrolix/sync v0.2.0/go.mod h1:BbecHL6jDSExojhNtgTFSBcdGerzNc64tz3DCOj/I0g= github.com/anacrolix/tagflag v0.0.0-20180109131632-2146c8d41bf0/go.mod h1:1m2U/K6ZT+JZG0+bdMK6qauP49QT4wE5pmhJXOKKCHw= github.com/anacrolix/utp v0.1.0/go.mod h1:MDwc+vsGEq7RMw6lr2GKOEqjWny5hO5OZXRVNaBJ2Dk= @@ -183,7 +188,7 @@ github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XL github.com/checkpoint-restore/go-criu/v4 v4.1.0/go.mod h1:xUQBLp4RLc5zJtWY++yjOoMoB5lihDt7fai+75m+rGw= github.com/cheekybits/genny v1.0.0 h1:uGGa4nei+j20rOSeDeP5Of12XVm7TGUd4dJA9RDitfE= github.com/cheekybits/genny v1.0.0/go.mod h1:+tQajlRqAUrPI7DOSpB0XAqZYtQakVtB7wXkRAgjxjQ= -github.com/cheggaaa/pb/v3 v3.0.4/go.mod h1:7rgWxLrAUcFMkvJuv09+DYi7mMUYi8nO9iOWcvGJPfw= +github.com/cheggaaa/pb/v3 v3.0.8/go.mod h1:UICbiLec/XO6Hw6k+BHEtHeQFzzBH4i2/qk/ow1EJTA= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= @@ -367,6 +372,8 @@ github.com/etcd-io/bbolt v1.3.3/go.mod h1:ZF2nL25h33cCyBtcyWeZ2/I3HQOfTP+0PIEvHj github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fasthttp-contrib/websocket v0.0.0-20160511215533-1f3b11f56072/go.mod h1:duJ4Jxv5lDcvg4QuQr0oowTf7dz4/CR8NtyCooz9HL8= github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= +github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM= +github.com/fatih/color v1.12.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM= github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M= github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc= github.com/flynn/noise v0.0.0-20180327030543-2492fe189ae6 h1:u/UEqS66A5ckRmS4yNpjmVH56sVtS/RfclBAYocb4as= @@ -492,7 +499,6 @@ github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8l github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golang/snappy v0.0.3 h1:fHPg5GQYlCeLIPB9BZqMVR5nR9A+IM5zcgeTdjMYmLA= github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= -github.com/gologme/log v0.0.0-20181207131047-4e5d8ccb38e8/go.mod h1:gq31gQ8wEHkR+WekdWsqDuf8pXTUZA9BnnzTuPz1Y9U= github.com/gologme/log v1.2.0 h1:Ya5Ip/KD6FX7uH0S31QO87nCCSucKtF44TLbTtO7V4c= github.com/gologme/log v1.2.0/go.mod h1:gq31gQ8wEHkR+WekdWsqDuf8pXTUZA9BnnzTuPz1Y9U= github.com/gomodule/redigo v1.7.1-0.20190724094224-574c33c3df38/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4= @@ -594,7 +600,7 @@ github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ= github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I= github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc= -github.com/hjson/hjson-go v3.0.2-0.20200316202735-d5d0e8b0617d+incompatible/go.mod h1:qsetwF8NlsTsOTwZTApNlTCerV+b2GjYRRcIk4JMFio= +github.com/hjson/hjson-go v3.1.0+incompatible/go.mod h1:qsetwF8NlsTsOTwZTApNlTCerV+b2GjYRRcIk4JMFio= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/huandu/xstrings v1.0.0/go.mod h1:4qWG/gcEcfX4z/mBDHJ++3ReCw9ibxbsNJbcucJdbSo= github.com/huandu/xstrings v1.2.0 h1:yPeWdRnmynF7p+lLYz0H2tthW9lqhMJrQV/U7yy4wX0= @@ -737,7 +743,7 @@ github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7V github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/k0kubun/colorstring v0.0.0-20150214042306-9440f1994b88/go.mod h1:3w7q1U84EfirKl04SVQ/s7nPm1ZPhiXd34z40TNz36k= github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= -github.com/kardianos/minwinsvc v0.0.0-20151122163309-cad6b2b879b0/go.mod h1:rUi0/YffDo1oXBOGn1KRq7Fr07LX48XEBecQnmwjsAo= +github.com/kardianos/minwinsvc v1.0.0/go.mod h1:Bgd0oc+D0Qo3bBytmNtyRKVlp85dAloLKhfxanPFFRc= github.com/kataras/golog v0.0.10/go.mod h1:yJ8YKCmyL+nWjERB90Qwn+bdyBZsaQwU3bTVFgkFIp8= github.com/kataras/iris/v12 v12.1.8/go.mod h1:LMYy4VlP67TQ3Zgriz8RE2h2kMZV2SgMYbq3UhfoFmE= github.com/kataras/neffos v0.0.14/go.mod h1:8lqADm8PnbeFfL7CLXh1WHw53dG27MC3pgi2R1rmoTE= @@ -998,10 +1004,8 @@ github.com/lucas-clemente/quic-go v0.19.3/go.mod h1:ADXpNbTQjq1hIzCpB+y/k5iz4n4z github.com/lunixbochs/vtclean v0.0.0-20160125035106-4fbf7632a2c6/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI= github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI= github.com/lxn/walk v0.0.0-20210112085537-c389da54e794/go.mod h1:E23UucZGqpuUANJooIbHWCufXvOcT6E7Stq81gU+CSQ= -github.com/lxn/win v0.0.0-20201111105847-2a20daff6a55/go.mod h1:KxxjdtRkfNoYDCUP5ryK7XJJNTnpC8atvtmTheChOtk= +github.com/lxn/win v0.0.0-20210218163916-a377121e959e/go.mod h1:KxxjdtRkfNoYDCUP5ryK7XJJNTnpC8atvtmTheChOtk= github.com/lyft/protoc-gen-validate v0.0.13/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ= -github.com/magefile/mage v1.10.0 h1:3HiXzCUY12kh9bIuyXShaVe529fJfyqoVM42o/uom2g= -github.com/magefile/mage v1.10.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= @@ -1025,10 +1029,10 @@ github.com/matrix-org/go-http-js-libp2p v0.0.0-20200518170932-783164aeeda4/go.mo github.com/matrix-org/go-sqlite3-js v0.0.0-20210625141222-bd2b7124cee8 h1:/FKUeUlCATr1gXxYqlaJgH8FW/sw0Jz8t7s8BIlECfg= github.com/matrix-org/go-sqlite3-js v0.0.0-20210625141222-bd2b7124cee8/go.mod h1:e+cg2q7C7yE5QnAXgzo512tgFh1RbQLC0+jozuegKgo= github.com/matrix-org/gomatrix v0.0.0-20190528120928-7df988a63f26/go.mod h1:3fxX6gUjWyI/2Bt7J1OLhpCzOfO/bB3AiX0cJtEKud0= -github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd h1:xVrqJK3xHREMNjwjljkAUaadalWc0rRbmVuQatzmgwg= -github.com/matrix-org/gomatrix v0.0.0-20200827122206-7dd5e2a05bcd/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= -github.com/matrix-org/gomatrixserverlib v0.0.0-20210702152949-0cac5159e7d6 h1:Rm5saCQeFfS15T0uD6qanSpx2kj/PfPBogbj5kFfQ+4= -github.com/matrix-org/gomatrixserverlib v0.0.0-20210702152949-0cac5159e7d6/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU= +github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16 h1:ZtO5uywdd5dLDCud4r0r55eP4j9FuUNpl60Gmntcop4= +github.com/matrix-org/gomatrix v0.0.0-20210324163249-be2af5ef2e16/go.mod h1:/gBX06Kw0exX1HrwmoBibFA98yBk/jxKpGVeyQbff+s= +github.com/matrix-org/gomatrixserverlib v0.0.0-20210714141824-52d282133140 h1:EWOcw9M3zoXz45aLgaOCay31Xa2QzzMX6vRLh0xNbzY= +github.com/matrix-org/gomatrixserverlib v0.0.0-20210714141824-52d282133140/go.mod h1:JsAzE1Ll3+gDWS9JSUHPJiiyAksvOOnGWF2nXdg4ZzU= github.com/matrix-org/naffka v0.0.0-20210623111924-14ff508b58e0 h1:HZCzy4oVzz55e+cOMiX/JtSF2UOY1evBl2raaE7ACcU= github.com/matrix-org/naffka v0.0.0-20210623111924-14ff508b58e0/go.mod h1:sjyPyRxKM5uw1nD2cJ6O2OxI6GOqyVBfNXqKjBZTBZE= github.com/matrix-org/pinecone v0.0.0-20210623102758-74f885644c1b h1:5X5vdWQ13xrNkJVqaJHPsrt7rKkMJH5iac0EtfOuxSg= @@ -1040,6 +1044,7 @@ github.com/mattn/go-colorable v0.0.6/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaO github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= +github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= github.com/mattn/go-isatty v0.0.0-20160806122752-66b8e73f3f5c/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= @@ -1047,11 +1052,12 @@ github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hd github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ= -github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84= -github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= +github.com/mattn/go-isatty v0.0.13 h1:qdl+GuBjcsKKDco5BsxPJlId98mSWNKqYA+Co0SC1yA= +github.com/mattn/go-isatty v0.0.13/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= -github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI= +github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk= +github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w= github.com/mattn/go-shellwords v1.0.3/go.mod h1:3xCvwCdWdlDJUrvuMn7Wuy9eWs4pE8vqg+NOMyg4B2o= github.com/mattn/go-sqlite3 v1.14.2/go.mod h1:JIl7NbARA7phWnGvh0LKTyg7S9BA+6gx71ShQilpsus= github.com/mattn/go-sqlite3 v1.14.7-0.20210414154423-1157a4212dcb h1:ax2vG2unlxsjwS7PMRo4FECIfAdQLowd6ejWYwPQhBo= @@ -1090,6 +1096,7 @@ github.com/mitchellh/gox v0.4.0/go.mod h1:Sd9lOJ0+aimLBi73mGofS1ycjY8lL3uZM3JPS4 github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0QubkSMEySY= github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/osext v0.0.0-20151018003038-5e2d6d41470f/go.mod h1:OkQIRizQZAeMln+1tSwduZz7+Af5oFlKirV/MSYes2A= github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc= github.com/moby/sys/mountinfo v0.4.0/go.mod h1:rEr8tzG/lsIZHBtN/JjGG+LMYx9eXgW2JI+6q0qou+A= @@ -1194,8 +1201,9 @@ github.com/nbio/st v0.0.0-20140626010706-e9e8d9816f32/go.mod h1:9wM+0iRr9ahx58uY github.com/ncw/swift v1.0.47/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM= github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo= github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM= -github.com/neilalexander/utp v0.1.1-0.20210622132614-ee9a34a30488 h1:xZk82i6JK2d0SqRIXwaxj7J/NQB6ngq0PuMx3wXBaRQ= github.com/neilalexander/utp v0.1.1-0.20210622132614-ee9a34a30488/go.mod h1:NPHGhPc0/wudcaCqL/H5AOddkRf8GPRhzOujuUKGQu8= +github.com/neilalexander/utp v0.1.1-0.20210705212447-691f29ad692b h1:XNm+Ks3bVziRJxcMaIbzumWEw7l52z9Rek6cMHgln1g= +github.com/neilalexander/utp v0.1.1-0.20210705212447-691f29ad692b/go.mod h1:ylsx0342RjGHjOoVKhR/wz/7Lhiusonihfj4QLxEMcU= github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 h1:zYyBkD/k9seD2A7fsi6Oo2LfFZAehjjQMERAvZLEDnQ= github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646/go.mod h1:jpp1/29i3P1S/RLdc7JQKbRpFeM1dOBd8T9ki5s+AY8= github.com/ngrok/sqlmw v0.0.0-20200129213757-d5c93a81bec6 h1:evlcQnJY+v8XRRchV3hXzpHDl6GcEZeLXAhlH9Csdww= @@ -1266,6 +1274,8 @@ github.com/openzipkin/zipkin-go v0.2.1/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnh github.com/openzipkin/zipkin-go v0.2.2/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnhQw8ySjnjRyN4= github.com/pact-foundation/pact-go v1.0.4/go.mod h1:uExwJY4kCzNPcHRj+hCR/HBbOOIwwtUjcrb0b5/5kLM= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= +github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc= +github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ= github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc= @@ -1341,6 +1351,8 @@ github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqn github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4= github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 h1:N/ElC8H3+5XpJzTSTfLsJV/mx9Q9g7kxmchpfZyxgzM= github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4= +github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= +github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= @@ -1385,8 +1397,8 @@ github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMB github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= -github.com/sirupsen/logrus v1.8.0 h1:nfhvjKcUMhBMVqbKHJlk5RPrrfYr/NMo3692g0dwfWU= -github.com/sirupsen/logrus v1.8.0/go.mod h1:4GuYW9TZmE769R5STWrRakJc4UqQ3+QQ95fyz7ENv1A= +github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE= +github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s= github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= @@ -1443,18 +1455,20 @@ github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpP github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA= github.com/tchap/go-patricia v2.2.6+incompatible/go.mod h1:bmLyhP68RS6kStMGxByiQ23RP/odRBOTVjwp2cDyi6I= github.com/tidwall/gjson v1.6.0/go.mod h1:P256ACg0Mn+j1RXIDXoss50DeIABTYK1PULOJHhxOls= -github.com/tidwall/gjson v1.6.8 h1:CTmXMClGYPAmln7652e69B7OLXfTi5ABcPPwjIWUv7w= -github.com/tidwall/gjson v1.6.8/go.mod h1:zeFuBCIqD4sN/gmqBzZ4j7Jd6UcA2Fc56x7QFsv+8fI= +github.com/tidwall/gjson v1.8.0/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk= +github.com/tidwall/gjson v1.8.1 h1:8j5EE9Hrh3l9Od1OIEDAb7IpezNA20UdRngNAj5N0WU= +github.com/tidwall/gjson v1.8.1/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk= github.com/tidwall/match v1.0.1/go.mod h1:LujAq0jyVjBy028G1WhWfIzbpQfMO8bBZ6Tyb0+pL9E= github.com/tidwall/match v1.0.3 h1:FQUVvBImDutD8wJLN6c5eMzWtjgONK9MwIBCOrUJKeE= github.com/tidwall/match v1.0.3/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM= github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= github.com/tidwall/pretty v1.0.1/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= -github.com/tidwall/pretty v1.0.2 h1:Z7S3cePv9Jwm1KwS0513MRaoUe3S01WPbLNV40pwWZU= -github.com/tidwall/pretty v1.0.2/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= +github.com/tidwall/pretty v1.1.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= +github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs= +github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU= github.com/tidwall/sjson v1.0.3/go.mod h1:bURseu1nuBkFpIES5cz6zBtjmYeOQmEESshn7VpF15Y= -github.com/tidwall/sjson v1.1.5 h1:wsUceI/XDyZk3J1FUvuuYlK62zJv2HO2Pzb8A5EWdUE= -github.com/tidwall/sjson v1.1.5/go.mod h1:VuJzsZnTowhSxWdOgsAnb886i4AjEyTkk7tNtsL7EYE= +github.com/tidwall/sjson v1.1.7 h1:sgVPwu/yygHJ2m1pJDLgGM/h+1F5odx5Q9ljG3imRm8= +github.com/tidwall/sjson v1.1.7/go.mod h1:w/yG+ezBeTdUxiKs5NcPicO9diP38nk96QBAbIIGeFs= github.com/tinylib/msgp v1.0.2/go.mod h1:+d+yLhGm8mzTaHzB+wgMYrodPfmZrzkirds8fDWklFE= github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= @@ -1480,13 +1494,12 @@ github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU= github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM= github.com/vishvananda/netlink v0.0.0-20181108222139-023a6dafdcdf/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk= -github.com/vishvananda/netlink v1.0.0/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk= github.com/vishvananda/netlink v1.1.0/go.mod h1:cTgwzPIzzgDAYoQrMm0EdrjRUBkTqKYppBueQtXaqoE= github.com/vishvananda/netlink v1.1.1-0.20201029203352-d40f9887b852/go.mod h1:twkDnbuQxJYemMlGd4JFIcuhgX83tXhKS2B/PRMpOho= github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI= -github.com/vishvananda/netns v0.0.0-20190625233234-7109fa855b0f/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI= github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17pCcGlemwknint6hfoeCVQrEMVwxRLRjXpq+BU= github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0= +github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 h1:EKhdznlJHPMoKr0XTrX+IlJs1LH3lyx2nfr1dOlZ79k= github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1/go.mod h1:8UvriyWtv5Q5EOgjHaSseUEdkQfvwFv1I/In/O2M9gc= github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1:bopw91TMyo8J3tvftk8xmU2kPmlrt4nScJQZU2hE5EM= @@ -1513,8 +1526,8 @@ github.com/xeipuuv/gojsonschema v1.2.0/go.mod h1:anYRn/JVcOK2ZgGU+IjEV4nwlhoK5sQ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0/go.mod h1:/LWChgwKmvncFJFHJ7Gvn9wZArjbV5/FppcK2fKk/tI= -github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20210218094457-e77ca8019daa h1:YHeZ1KN4KmuAjqmBSan1JtwyoPQoklzMjMqIbaS5Ywo= -github.com/yggdrasil-network/yggdrasil-go v0.3.15-0.20210218094457-e77ca8019daa/go.mod h1:G716RAw9WTLbLFI7lVj1GKTU16wb9MYl6iE9j4JlWeI= +github.com/yggdrasil-network/yggdrasil-go v0.4.1-0.20210715083903-52309d094c00 h1:bv6+5Dv7XHbThfXirMV9hh45hUH26LtuZKHrVlM/3rQ= +github.com/yggdrasil-network/yggdrasil-go v0.4.1-0.20210715083903-52309d094c00/go.mod h1:/iMJjOrXRsjlFgqhWOPhecOKi7xHmHiY4/En3A42Fog= github.com/yudai/gojsondiff v1.0.0/go.mod h1:AY32+k2cwILAkW1fbgxQ5mUmMiZFgLIV+FBNExI05xg= github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82/go.mod h1:lgjkn3NuSvDfVJdfcVVdX+jpBxNmX4rDAzaS45IcYoM= github.com/yudai/pp v2.0.1+incompatible/go.mod h1:PuxR/8QJ7cyCkFp/aUDS+JY727OFEZkTdatxwunjIkc= @@ -1592,11 +1605,11 @@ golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= golang.org/x/crypto v0.0.0-20210314154223-e6e6c4f2bb5b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= +golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a h1:kr2P4QFmQr29mSLA43kwrOcgcReGTfbE9N577tCTuBc= golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= @@ -1683,11 +1696,12 @@ golang.org/x/net v0.0.0-20201006153459-a7d1128ccaa0/go.mod h1:sp8m0HH+o8qH0wwXwY golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210427231257-85d9c07bbe3a/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= -golang.org/x/net v0.0.0-20210525063256-abc453219eb5 h1:wjuX4b5yYQnEQHzd+CBcrcC6OVR2J1CN6mUy0oSxIPo= +golang.org/x/net v0.0.0-20210510120150-4163338589ed/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20210610132358-84b48f89b13b h1:k+E048sYJHyVnsr1GDrRZWQ32D2C7lWs9JRc0bel53A= +golang.org/x/net v0.0.0-20210610132358-84b48f89b13b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= @@ -1748,12 +1762,10 @@ golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191115151921-52ab43148777/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191128015809-6d18c012aee9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191210023423-ac6580df4449/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191220142924-d4481acd189f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -1771,6 +1783,7 @@ golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200625212154-ddb9806d33ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -1787,12 +1800,14 @@ golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201202213521-69691e467435/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201214210602-f9fddec55a1e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210105210732-16f7687f5001/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210309040221-94ec62e08169/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210324051608-47abb6519492/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210423082822-04245dca01da h1:b3NXsE2LusjYGGjL5bxEVZZORm/YEFFrWFjR8eFrw/c= +golang.org/x/sys v0.0.0-20210403161142-5e06dd20ab57/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210611083646-a4fc73990273 h1:faDu4veV+8pcThn4fewv6TVlNCezafGoC1gM/mxQLbQ= +golang.org/x/sys v0.0.0-20210611083646-a4fc73990273/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= @@ -1802,9 +1817,9 @@ golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.7-0.20210503195748-5c7c50ebbd4f h1:yQJrRE0hDxDFmZLlRaw+3vusO4fwNHgHIjUOMO7bHYI= +golang.org/x/text v0.3.7-0.20210503195748-5c7c50ebbd4f/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -1866,10 +1881,9 @@ golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.zx2c4.com/wireguard v0.0.0-20210203165646-9c7bd73be2cc/go.mod h1:r0ExowOoGFfDoLDxx+M9SYbNVsoZ0xviLL+K4f2mt+A= -golang.zx2c4.com/wireguard v0.0.0-20210212170059-7a0fb5bbb172/go.mod h1:r0ExowOoGFfDoLDxx+M9SYbNVsoZ0xviLL+K4f2mt+A= golang.zx2c4.com/wireguard v0.0.0-20210510202332-9844c74f67ec/go.mod h1:a057zjmoc00UN7gVkaJt2sXVK523kMJcogDTEvPIasg= -golang.zx2c4.com/wireguard/windows v0.3.5/go.mod h1:ATrIFNoq3rsK735WJiQzfWYyNFc9xLBhMMjW9DWIvnU= +golang.zx2c4.com/wireguard v0.0.0-20210604143328-f9b48a961cd2/go.mod h1:laHzsbfMhGSobUmruXWAyMKKHSqvIcrqZJMyHD+/3O8= +golang.zx2c4.com/wireguard/windows v0.3.14/go.mod h1:3P4IEAsb+BjlKZmpUXgy74c0iX9AVwwr3WcVJ8nPgME= google.golang.org/api v0.0.0-20160322025152-9bf6e6e569ff/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0= diff --git a/internal/version.go b/internal/version.go index 0d348779..37f0c30d 100644 --- a/internal/version.go +++ b/internal/version.go @@ -16,8 +16,8 @@ var build string const ( VersionMajor = 0 - VersionMinor = 3 - VersionPatch = 11 + VersionMinor = 4 + VersionPatch = 0 VersionTag = "" // example: "rc1" ) diff --git a/roomserver/api/query.go b/roomserver/api/query.go index af35f7e7..c70db65c 100644 --- a/roomserver/api/query.go +++ b/roomserver/api/query.go @@ -170,7 +170,8 @@ type QueryMembershipsForRoomResponse struct { // QueryServerJoinedToRoomRequest is a request to QueryServerJoinedToRoom type QueryServerJoinedToRoomRequest struct { - // Server name of the server to find + // Server name of the server to find. If not specified, we will + // default to checking if the local server is joined. ServerName gomatrixserverlib.ServerName `json:"server_name"` // ID of the room to see if we are still joined to RoomID string `json:"room_id"` @@ -182,7 +183,8 @@ type QueryServerJoinedToRoomResponse struct { RoomExists bool `json:"room_exists"` // True if we still believe that we are participating in the room IsInRoom bool `json:"is_in_room"` - // List of servers that are also in the room + // List of servers that are also in the room. This will not be populated + // if the queried ServerName is the local server name. ServerNames []gomatrixserverlib.ServerName `json:"server_names"` } diff --git a/roomserver/internal/api.go b/roomserver/internal/api.go index c9f92f9f..f39b26ea 100644 --- a/roomserver/internal/api.go +++ b/roomserver/internal/api.go @@ -59,6 +59,7 @@ func NewRoomserverAPI( Queryer: &query.Queryer{ DB: roomserverDB, Cache: caches, + ServerName: cfg.Matrix.ServerName, ServerACLs: serverACLs, }, Inputer: &input.Inputer{ @@ -92,6 +93,7 @@ func (r *RoomserverInternalAPI) SetFederationSenderAPI(fsAPI fsAPI.FederationSen FSAPI: r.fsAPI, RSAPI: r, Inputer: r.Inputer, + Queryer: r.Queryer, } r.Peeker = &perform.Peeker{ ServerName: r.Cfg.Matrix.ServerName, diff --git a/roomserver/internal/helpers/helpers.go b/roomserver/internal/helpers/helpers.go index a829bffc..a389cc89 100644 --- a/roomserver/internal/helpers/helpers.go +++ b/roomserver/internal/helpers/helpers.go @@ -50,6 +50,10 @@ func UpdateToInviteMembership( return updates, nil } +// IsServerCurrentlyInRoom checks if a server is in a given room, based on the room +// memberships. If the servername is not supplied then the local server will be +// checked instead using a faster code path. +// TODO: This should probably be replaced by an API call. func IsServerCurrentlyInRoom(ctx context.Context, db storage.Database, serverName gomatrixserverlib.ServerName, roomID string) (bool, error) { info, err := db.RoomInfo(ctx, roomID) if err != nil { @@ -59,6 +63,10 @@ func IsServerCurrentlyInRoom(ctx context.Context, db storage.Database, serverNam return false, fmt.Errorf("unknown room %s", roomID) } + if serverName == "" { + return db.GetLocalServerInRoom(ctx, info.RoomNID) + } + eventNIDs, err := db.GetMembershipEventNIDsForRoom(ctx, info.RoomNID, true, false) if err != nil { return false, err diff --git a/roomserver/internal/input/input.go b/roomserver/internal/input/input.go index 6bc43c9c..de40e133 100644 --- a/roomserver/internal/input/input.go +++ b/roomserver/internal/input/input.go @@ -30,9 +30,16 @@ import ( "github.com/matrix-org/gomatrixserverlib" "github.com/prometheus/client_golang/prometheus" log "github.com/sirupsen/logrus" + "github.com/tidwall/gjson" "go.uber.org/atomic" ) +var keyContentFields = map[string]string{ + "m.room.join_rules": "join_rule", + "m.room.history_visibility": "history_visibility", + "m.room.member": "membership", +} + type Inputer struct { DB storage.Database Producer sarama.SyncProducer @@ -95,15 +102,27 @@ func (r *Inputer) WriteOutputEvents(roomID string, updates []api.OutputEvent) er "type": updates[i].Type, }) if updates[i].NewRoomEvent != nil { + eventType := updates[i].NewRoomEvent.Event.Type() logger = logger.WithFields(log.Fields{ - "event_type": updates[i].NewRoomEvent.Event.Type(), + "event_type": eventType, "event_id": updates[i].NewRoomEvent.Event.EventID(), "adds_state": len(updates[i].NewRoomEvent.AddsStateEventIDs), "removes_state": len(updates[i].NewRoomEvent.RemovesStateEventIDs), "send_as_server": updates[i].NewRoomEvent.SendAsServer, "sender": updates[i].NewRoomEvent.Event.Sender(), }) - if updates[i].NewRoomEvent.Event.Type() == "m.room.server_acl" && updates[i].NewRoomEvent.Event.StateKeyEquals("") { + if updates[i].NewRoomEvent.Event.StateKey() != nil { + logger = logger.WithField("state_key", *updates[i].NewRoomEvent.Event.StateKey()) + } + contentKey := keyContentFields[eventType] + if contentKey != "" { + value := gjson.GetBytes(updates[i].NewRoomEvent.Event.Content(), contentKey) + if value.Exists() { + logger = logger.WithField("content_value", value.String()) + } + } + + if eventType == "m.room.server_acl" && updates[i].NewRoomEvent.Event.StateKeyEquals("") { ev := updates[i].NewRoomEvent.Event.Unwrap() defer r.ACLs.OnServerACLUpdate(ev) } diff --git a/roomserver/internal/perform/perform_join.go b/roomserver/internal/perform/perform_join.go index 048496d4..876888e2 100644 --- a/roomserver/internal/perform/perform_join.go +++ b/roomserver/internal/perform/perform_join.go @@ -28,6 +28,7 @@ import ( rsAPI "github.com/matrix-org/dendrite/roomserver/api" "github.com/matrix-org/dendrite/roomserver/internal/helpers" "github.com/matrix-org/dendrite/roomserver/internal/input" + "github.com/matrix-org/dendrite/roomserver/internal/query" "github.com/matrix-org/dendrite/roomserver/storage" "github.com/matrix-org/dendrite/setup/config" "github.com/matrix-org/gomatrixserverlib" @@ -42,6 +43,7 @@ type Joiner struct { DB storage.Database Inputer *input.Inputer + Queryer *query.Queryer } // PerformJoin handles joining matrix rooms, including over federation by talking to the federationsender. @@ -205,7 +207,14 @@ func (r *Joiner) performJoinRoomByID( // Force a federated join if we aren't in the room and we've been // given some server names to try joining by. - serverInRoom, _ := helpers.IsServerCurrentlyInRoom(ctx, r.DB, r.ServerName, req.RoomIDOrAlias) + inRoomReq := &api.QueryServerJoinedToRoomRequest{ + RoomID: req.RoomIDOrAlias, + } + inRoomRes := &api.QueryServerJoinedToRoomResponse{} + if err = r.Queryer.QueryServerJoinedToRoom(ctx, inRoomReq, inRoomRes); err != nil { + return "", "", fmt.Errorf("r.Queryer.QueryServerJoinedToRoom: %w", err) + } + serverInRoom := inRoomRes.IsInRoom forceFederatedJoin := len(req.ServerNames) > 0 && !serverInRoom // Force a federated join if we're dealing with a pending invite diff --git a/roomserver/internal/perform/perform_leave.go b/roomserver/internal/perform/perform_leave.go index 4d10dea6..88eb7e1e 100644 --- a/roomserver/internal/perform/perform_leave.go +++ b/roomserver/internal/perform/perform_leave.go @@ -26,6 +26,7 @@ import ( "github.com/matrix-org/dendrite/roomserver/storage" "github.com/matrix-org/dendrite/setup/config" "github.com/matrix-org/gomatrixserverlib" + "github.com/matrix-org/util" ) type Leaver struct { @@ -171,7 +172,9 @@ func (r *Leaver) performFederatedRejectInvite( } leaveRes := fsAPI.PerformLeaveResponse{} if err := r.FSAPI.PerformLeave(ctx, &leaveReq, &leaveRes); err != nil { - return nil, err + // failures in PerformLeave should NEVER stop us from telling other components like the + // sync API that the invite was withdrawn. Otherwise we can end up with stuck invites. + util.GetLogger(ctx).WithError(err).Errorf("failed to PerformLeave, still retiring invite event") } // Withdraw the invite, so that the sync API etc are diff --git a/roomserver/internal/query/query.go b/roomserver/internal/query/query.go index 408f9766..4af0e639 100644 --- a/roomserver/internal/query/query.go +++ b/roomserver/internal/query/query.go @@ -36,6 +36,7 @@ import ( type Queryer struct { DB storage.Database Cache caching.RoomServerCaches + ServerName gomatrixserverlib.ServerName ServerACLs *acls.ServerACLs } @@ -328,6 +329,16 @@ func (r *Queryer) QueryServerJoinedToRoom( } response.RoomExists = true + if request.ServerName == r.ServerName || request.ServerName == "" { + var joined bool + joined, err = r.DB.GetLocalServerInRoom(ctx, info.RoomNID) + if err != nil { + return fmt.Errorf("r.DB.GetLocalServerInRoom: %w", err) + } + response.IsInRoom = joined + return nil + } + eventNIDs, err := r.DB.GetMembershipEventNIDsForRoom(ctx, info.RoomNID, true, false) if err != nil { return fmt.Errorf("r.DB.GetMembershipEventNIDsForRoom: %w", err) @@ -377,10 +388,16 @@ func (r *Queryer) QueryServerAllowedToSeeEvent( return } roomID := events[0].RoomID() - isServerInRoom, err := helpers.IsServerCurrentlyInRoom(ctx, r.DB, request.ServerName, roomID) - if err != nil { - return + + inRoomReq := &api.QueryServerJoinedToRoomRequest{ + RoomID: roomID, + ServerName: request.ServerName, } + inRoomRes := &api.QueryServerJoinedToRoomResponse{} + if err = r.QueryServerJoinedToRoom(ctx, inRoomReq, inRoomRes); err != nil { + return fmt.Errorf("r.Queryer.QueryServerJoinedToRoom: %w", err) + } + info, err := r.DB.RoomInfo(ctx, roomID) if err != nil { return err @@ -389,7 +406,7 @@ func (r *Queryer) QueryServerAllowedToSeeEvent( return fmt.Errorf("QueryServerAllowedToSeeEvent: no room info for room %s", roomID) } response.AllowedToSeeEvent, err = helpers.CheckServerAllowedToSeeEvent( - ctx, r.DB, *info, request.EventID, request.ServerName, isServerInRoom, + ctx, r.DB, *info, request.EventID, request.ServerName, inRoomRes.IsInRoom, ) return } diff --git a/roomserver/storage/interface.go b/roomserver/storage/interface.go index d2b0e75c..c25820aa 100644 --- a/roomserver/storage/interface.go +++ b/roomserver/storage/interface.go @@ -154,6 +154,8 @@ type Database interface { GetBulkStateContent(ctx context.Context, roomIDs []string, tuples []gomatrixserverlib.StateKeyTuple, allowWildcards bool) ([]tables.StrippedEvent, error) // JoinedUsersSetInRooms returns all joined users in the rooms given, along with the count of how many times they appear. JoinedUsersSetInRooms(ctx context.Context, roomIDs []string) (map[string]int, error) + // GetLocalServerInRoom returns true if we think we're in a given room or false otherwise. + GetLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) // GetKnownUsers searches all users that userID knows about. GetKnownUsers(ctx context.Context, userID, searchString string, limit int) ([]string, error) // GetKnownRooms returns a list of all rooms we know about. diff --git a/roomserver/storage/postgres/membership_table.go b/roomserver/storage/postgres/membership_table.go index 3466da6d..9102f26a 100644 --- a/roomserver/storage/postgres/membership_table.go +++ b/roomserver/storage/postgres/membership_table.go @@ -124,6 +124,14 @@ var selectKnownUsersSQL = "" + " SELECT DISTINCT room_nid FROM roomserver_membership WHERE target_nid=$1 AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) + ") AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) + " AND event_state_key LIKE $2 LIMIT $3" +// selectLocalServerInRoomSQL is an optimised case for checking if we, the local server, +// are in the room by using the target_local column of the membership table. Normally when +// we want to know if a server is in a room, we have to unmarshal the entire room state which +// is expensive. The presence of a single row from this query suggests we're still in the +// room, no rows returned suggests we aren't. +const selectLocalServerInRoomSQL = "" + + "SELECT room_nid FROM roomserver_membership WHERE target_local = true AND membership_nid = $1 AND room_nid = $2 LIMIT 1" + type membershipStatements struct { insertMembershipStmt *sql.Stmt selectMembershipForUpdateStmt *sql.Stmt @@ -137,6 +145,7 @@ type membershipStatements struct { selectJoinedUsersSetForRoomsStmt *sql.Stmt selectKnownUsersStmt *sql.Stmt updateMembershipForgetRoomStmt *sql.Stmt + selectLocalServerInRoomStmt *sql.Stmt } func createMembershipTable(db *sql.DB) error { @@ -160,6 +169,7 @@ func prepareMembershipTable(db *sql.DB) (tables.Membership, error) { {&s.selectJoinedUsersSetForRoomsStmt, selectJoinedUsersSetForRoomsSQL}, {&s.selectKnownUsersStmt, selectKnownUsersSQL}, {&s.updateMembershipForgetRoomStmt, updateMembershipForgetRoom}, + {&s.selectLocalServerInRoomStmt, selectLocalServerInRoomSQL}, }.Prepare(db) } @@ -324,3 +334,16 @@ func (s *membershipStatements) UpdateForgetMembership( ) return err } + +func (s *membershipStatements) SelectLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) { + var nid types.RoomNID + err := s.selectLocalServerInRoomStmt.QueryRowContext(ctx, tables.MembershipStateJoin, roomNID).Scan(&nid) + if err != nil { + if err == sql.ErrNoRows { + return false, nil + } + return false, err + } + found := nid > 0 + return found, nil +} diff --git a/roomserver/storage/shared/storage.go b/roomserver/storage/shared/storage.go index e77d62e0..9d9434cb 100644 --- a/roomserver/storage/shared/storage.go +++ b/roomserver/storage/shared/storage.go @@ -1059,6 +1059,11 @@ func (d *Database) JoinedUsersSetInRooms(ctx context.Context, roomIDs []string) return result, nil } +// GetLocalServerInRoom returns true if we think we're in a given room or false otherwise. +func (d *Database) GetLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) { + return d.MembershipTable.SelectLocalServerInRoom(ctx, roomNID) +} + // GetKnownUsers searches all users that userID knows about. func (d *Database) GetKnownUsers(ctx context.Context, userID, searchString string, limit int) ([]string, error) { stateKeyNID, err := d.EventStateKeysTable.SelectEventStateKeyNID(ctx, nil, userID) diff --git a/roomserver/storage/sqlite3/membership_table.go b/roomserver/storage/sqlite3/membership_table.go index d9fe32cf..82babe0d 100644 --- a/roomserver/storage/sqlite3/membership_table.go +++ b/roomserver/storage/sqlite3/membership_table.go @@ -100,6 +100,14 @@ var selectKnownUsersSQL = "" + " SELECT DISTINCT room_nid FROM roomserver_membership WHERE target_nid=$1 AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) + ") AND membership_nid = " + fmt.Sprintf("%d", tables.MembershipStateJoin) + " AND event_state_key LIKE $2 LIMIT $3" +// selectLocalServerInRoomSQL is an optimised case for checking if we, the local server, +// are in the room by using the target_local column of the membership table. Normally when +// we want to know if a server is in a room, we have to unmarshal the entire room state which +// is expensive. The presence of a single row from this query suggests we're still in the +// room, no rows returned suggests we aren't. +const selectLocalServerInRoomSQL = "" + + "SELECT room_nid FROM roomserver_membership WHERE target_local = 1 AND membership_nid = $1 AND room_nid = $2 LIMIT 1" + type membershipStatements struct { db *sql.DB insertMembershipStmt *sql.Stmt @@ -113,6 +121,7 @@ type membershipStatements struct { updateMembershipStmt *sql.Stmt selectKnownUsersStmt *sql.Stmt updateMembershipForgetRoomStmt *sql.Stmt + selectLocalServerInRoomStmt *sql.Stmt } func createMembershipTable(db *sql.DB) error { @@ -137,6 +146,7 @@ func prepareMembershipTable(db *sql.DB) (tables.Membership, error) { {&s.selectRoomsWithMembershipStmt, selectRoomsWithMembershipSQL}, {&s.selectKnownUsersStmt, selectKnownUsersSQL}, {&s.updateMembershipForgetRoomStmt, updateMembershipForgetRoom}, + {&s.selectLocalServerInRoomStmt, selectLocalServerInRoomSQL}, }.Prepare(db) } @@ -304,3 +314,16 @@ func (s *membershipStatements) UpdateForgetMembership( ) return err } + +func (s *membershipStatements) SelectLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) { + var nid types.RoomNID + err := s.selectLocalServerInRoomStmt.QueryRowContext(ctx, tables.MembershipStateJoin, roomNID).Scan(&nid) + if err != nil { + if err == sql.ErrNoRows { + return false, nil + } + return false, err + } + found := nid > 0 + return found, nil +} diff --git a/roomserver/storage/tables/interface.go b/roomserver/storage/tables/interface.go index dd486873..4a893663 100644 --- a/roomserver/storage/tables/interface.go +++ b/roomserver/storage/tables/interface.go @@ -135,6 +135,7 @@ type Membership interface { SelectJoinedUsersSetForRooms(ctx context.Context, roomNIDs []types.RoomNID) (map[types.EventStateKeyNID]int, error) SelectKnownUsers(ctx context.Context, userID types.EventStateKeyNID, searchString string, limit int) ([]string, error) UpdateForgetMembership(ctx context.Context, txn *sql.Tx, roomNID types.RoomNID, targetUserNID types.EventStateKeyNID, forget bool) error + SelectLocalServerInRoom(ctx context.Context, roomNID types.RoomNID) (bool, error) } type Published interface { diff --git a/setup/base.go b/setup/base.go index 6bdeb80f..7b691608 100644 --- a/setup/base.go +++ b/setup/base.go @@ -77,6 +77,7 @@ type BaseDendrite struct { PublicKeyAPIMux *mux.Router PublicMediaAPIMux *mux.Router InternalAPIMux *mux.Router + SynapseAdminMux *mux.Router UseHTTPAPIs bool apiHttpClient *http.Client httpClient *http.Client @@ -199,6 +200,7 @@ func NewBaseDendrite(cfg *config.Dendrite, componentName string, useHTTPAPIs boo PublicKeyAPIMux: mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicKeyPathPrefix).Subrouter().UseEncodedPath(), PublicMediaAPIMux: mux.NewRouter().SkipClean(true).PathPrefix(httputil.PublicMediaPathPrefix).Subrouter().UseEncodedPath(), InternalAPIMux: mux.NewRouter().SkipClean(true).PathPrefix(httputil.InternalPathPrefix).Subrouter().UseEncodedPath(), + SynapseAdminMux: mux.NewRouter().SkipClean(true).PathPrefix("/_synapse/").Subrouter().UseEncodedPath(), apiHttpClient: &apiClient, httpClient: &client, } @@ -391,6 +393,7 @@ func (b *BaseDendrite) SetupAndServeHTTP( externalRouter.PathPrefix(httputil.PublicKeyPathPrefix).Handler(b.PublicKeyAPIMux) externalRouter.PathPrefix(httputil.PublicFederationPathPrefix).Handler(federationHandler) } + externalRouter.PathPrefix("/_synapse/").Handler(b.SynapseAdminMux) externalRouter.PathPrefix(httputil.PublicMediaPathPrefix).Handler(b.PublicMediaAPIMux) if internalAddr != NoListener && internalAddr != externalAddr { diff --git a/setup/monolith.go b/setup/monolith.go index 235be447..5ceb4ed3 100644 --- a/setup/monolith.go +++ b/setup/monolith.go @@ -57,9 +57,9 @@ type Monolith struct { } // AddAllPublicRoutes attaches all public paths to the given router -func (m *Monolith) AddAllPublicRoutes(process *process.ProcessContext, csMux, ssMux, keyMux, mediaMux *mux.Router) { +func (m *Monolith) AddAllPublicRoutes(process *process.ProcessContext, csMux, ssMux, keyMux, mediaMux, synapseMux *mux.Router) { clientapi.AddPublicRoutes( - csMux, &m.Config.ClientAPI, m.AccountDB, + csMux, synapseMux, &m.Config.ClientAPI, m.AccountDB, m.FedClient, m.RoomserverAPI, m.EDUInternalAPI, m.AppserviceAPI, transactions.New(), m.FederationSenderAPI, m.UserAPI, m.KeyAPI, m.ExtPublicRoomsProvider, diff --git a/syncapi/streams/stream_invite.go b/syncapi/streams/stream_invite.go index 10a0dda8..70374c6a 100644 --- a/syncapi/streams/stream_invite.go +++ b/syncapi/streams/stream_invite.go @@ -2,8 +2,13 @@ package streams import ( "context" + "crypto/sha256" + "encoding/base64" + "strconv" + "time" "github.com/matrix-org/dendrite/syncapi/types" + "github.com/matrix-org/gomatrixserverlib" ) type InviteStreamProvider struct { @@ -56,6 +61,17 @@ func (p *InviteStreamProvider) IncrementalSync( for roomID := range retiredInvites { if _, ok := req.Response.Rooms.Join[roomID]; !ok { lr := types.NewLeaveResponse() + h := sha256.Sum256(append([]byte(roomID), []byte(strconv.FormatInt(int64(to), 10))...)) + lr.Timeline.Events = append(lr.Timeline.Events, gomatrixserverlib.ClientEvent{ + // fake event ID which muxes in the to position + EventID: "$" + base64.RawURLEncoding.EncodeToString(h[:]), + OriginServerTS: gomatrixserverlib.AsTimestamp(time.Now()), + RoomID: roomID, + Sender: req.Device.UserID, + StateKey: &req.Device.UserID, + Type: "m.room.member", + Content: gomatrixserverlib.RawJSON(`{"membership":"leave"}`), + }) req.Response.Rooms.Leave[roomID] = *lr } } diff --git a/sytest-blacklist b/sytest-blacklist index 4d9587d0..a0aba69c 100644 --- a/sytest-blacklist +++ b/sytest-blacklist @@ -72,4 +72,8 @@ Can re-join room if re-invited Local device key changes get to remote servers with correct prev_id # Flakey -Local device key changes appear in /keys/changes \ No newline at end of file +Local device key changes appear in /keys/changes + +# we don't support groups +Remove group category +Remove group role diff --git a/sytest-whitelist b/sytest-whitelist index 8c458571..f6a051bd 100644 --- a/sytest-whitelist +++ b/sytest-whitelist @@ -520,3 +520,18 @@ Inviting an AS-hosted user asks the AS server Can generate a openid access_token that can be exchanged for information about a user Invalid openid access tokens are rejected Requests to userinfo without access tokens are rejected +POST /_synapse/admin/v1/register with shared secret +POST /_synapse/admin/v1/register admin with shared secret +POST /_synapse/admin/v1/register with shared secret downcases capitals +POST /_synapse/admin/v1/register with shared secret disallows symbols +Membership event with an invalid displayname in the send_join response should not cause room join to fail +Inbound federation rejects incorrectly-signed invite rejections +Inbound federation can receive invite rejections +Inbound federation can receive invite and reject when remote replies with a 403 +Inbound federation can receive invite and reject when remote replies with a 500 +Inbound federation can receive invite and reject when remote is unreachable +Remote servers cannot set power levels in rooms without existing powerlevels +Remote servers should reject attempts by non-creators to set the power levels +Federation handles empty auth_events in state_ids sanely +Key notary server should return an expired key if it can't find any others +Key notary server must not overwrite a valid key with a spurious result from the origin server