Send Application Service Events (#477)

* Prevent sql scanning into nil value in accounts_table

Signed-off-by: Andrew Morgan <andrewm@matrix.org>

* Remove uneccessary logging, null checking

* Don't forget to set the localpart

* Simplify error checking

* Store And Send Application Service Events

* Modify INSTALL.md and dendrite-config.yaml for the new appservice database
* Correct all instances of casing on 'application service' to align with
spec
* Store incoming events that an app service is interested in in the
database to be later read by transaction workers.
* Retrieve these events from transaction workers, one per AS.
* Minimal transaction ID data is stored as well to recover after
server failure.
* Send events to AS and exponentially backoff on failure.

Signed-off-by: Andrew Morgan <andrewm@matrix.org>

* Finish my own sentences.

* Fix up database interaction

* Change to event-based AS sending

* Reduce cyclomatic complexity

* Appease the errcheck gods

* Delete by int ID instead of string.

This was causing some events to not be deleted, as < an eventID doesn't
really make much sense.

* Check if there are more events to send before sleeping

* Send same transaction if last send attempt failed

* Don't backoff on non-200s, tight send loop, 1 event query

* Remove tight send loop. Fix events not being deleted

* Additionally order by event id, track main.go

* Return the last txnID, which our events are using

* Remove old main.go file

* Prevent duplicate events from being sent...

* Strip event content if it doesn't contain anything

Signed-off-by: Andrew Morgan <andrewm@matrix.org>

* Update gomatrixserverlib and use Unsigned AS event prop

* Fixes

* Fix sync server comment
* Remove unnecessary printlns
* Use logrus Fields
* Worker state methods
* Remove sillyness

* Fix up event filtering

* Handle transaction event limit in loop

* Switch to using a sequence for transaction IDs

* Don't verify self-signed AS certificates

* Fix logging

* Use gmsl.Event instead of AS-only event in transactions

Also clear up the logic on lookupStateEvents a little bit.

* Change invalid_txn_id to global (for efficiency)

* Use a bool for EventsReady instead of an int

vendor/manifest

@@ -135,7 +135,7 @@
 		{
 			"importpath": "github.com/matrix-org/gomatrixserverlib",
 			"repository": "https://github.com/matrix-org/gomatrixserverlib",
-			"revision": "38a4f0f648bf357adc4bdb601cdc0535cee14e21",
+			"revision": "929828872b51e6733166553d6b1a20155b6ab829",
 			"branch": "master"
 		},
 		{

vendor/src/github.com/matrix-org/gomatrixserverlib/appservice.go

@@ -15,21 +15,8 @@
 
 package gomatrixserverlib
 
-// ApplicationServiceEvent is an event format that is sent off to an
-// application service as part of a transaction.
-type ApplicationServiceEvent struct {
-	Age                   int64   `json:"age,omitempty"`
-	Content               RawJSON `json:"content,omitempty"`
-	EventID               string  `json:"event_id,omitempty"`
-	OriginServerTimestamp int64   `json:"origin_server_ts,omitempty"`
-	RoomID                string  `json:"room_id,omitempty"`
-	Sender                string  `json:"sender,omitempty"`
-	Type                  string  `json:"type,omitempty"`
-	UserID                string  `json:"user_id,omitempty"`
-}
-
 // ApplicationServiceTransaction is the transaction that is sent off to an
 // application service.
 type ApplicationServiceTransaction struct {
-	Events []ApplicationServiceEvent `json:"events"`
+	Events []Event `json:"events"`
 }

vendor/src/github.com/matrix-org/gomatrixserverlib/eventcontent.go

@@ -261,6 +261,9 @@ func newPowerLevelContentFromAuthEvents(authEvents AuthEventProvider, creatorUse
 	// If there is no power level event then the creator gets level 100
 	// https://github.com/matrix-org/synapse/blob/v0.18.5/synapse/api/auth.py#L569
 	c.userLevels = map[string]int64{creatorUserID: 100}
+	// If there is no power level event then the state_default is level 0
+	// https://github.com/matrix-org/synapse/blob/v0.18.5/synapse/api/auth.py#L997
+	c.stateDefaultLevel = 0
 	return
 }
 

vendor/src/github.com/matrix-org/gomatrixserverlib/federationtypes.go

@@ -37,6 +37,12 @@ type RespState struct {
 	AuthEvents []Event `json:"auth_chain"`
 }
 
+// A RespEventAuth is the content of a response to GET /_matrix/federation/v1/event_auth/{roomID}/{eventID}
+type RespEventAuth struct {
+	// A list of events needed to authenticate the state events.
+	AuthEvents []Event `json:"auth_chain"`
+}
+
 // Events combines the auth events and the state events and returns
 // them in an order where every event comes after its auth events.
 // Each event will only appear once in the output list.