Commit graph

1372 commits

Author SHA1 Message Date
Mark Haines
e6d77d6bde Use HTTP API for roomserver input. (#161)
* Use HTTP API for roomserver input.

* Use synchronous HTTP API for writing events to the roomserver

* Remove unused config for kafka topic

* Tweak comments
2017-07-18 13:40:03 +01:00
Brendan Abolivier
d9b8e5de45 Keep track of membership in Client API (#159)
* Saving memberships

* Removed unused index

* Removed useless log

* Fixed membership not being saved on the right conditions + added membership removal

* Updated outdated comment

* Use server lib method + check server name + use new roomserver API

* Better handling of events from the room server

* Fixed membership removal

* Corrected indentation

* Fix tests (hopefully)

* Replace broken kafka mirror

* Apply requested changes on database management

* Remove useless check and function

* Moved memberships update to the database package

* Use new common function

* Remove useless function
2017-07-17 18:10:56 +01:00
Mark Haines
b06d1124f7 Factor out runTransaction to common code (#162) 2017-07-17 17:20:57 +01:00
Mark Haines
d3a29b7816 Add a HTTP API for roomserver input (#158)
* Add a HTTP API for roomserver input

* Fix integration tests
2017-07-13 11:41:30 +01:00
Brendan Abolivier
f40b776aa3 Updated config file example with federation sender database (#157) 2017-07-12 14:48:12 +01:00
Mark Haines
4e7862e3b9 Make the room server RPC API json formats more consistent with our other APIs (#156) 2017-07-12 14:13:10 +01:00
Mark Haines
cea9e31723 Make the roomserver output format more flexible (#155)
* Make the roomserver output format more flexible

* Fix the sync server integration testdata

* Fix roomserver testdata

* Actually fix the sync server tests

* Fix typo
2017-07-12 10:46:29 +01:00
Brendan Abolivier
7d36ca03af Implemented logout (#154) 2017-07-11 16:04:34 +01:00
Brendan Abolivier
355ab5eedf Notify profile update (#153)
* Profile retrieval

* Saving avatar (without propagating it)

* Saving display name (without propagating it)

* Getters for display name and avatar URL

* Doc'd

* Introduced new Kafka topic and producer

* Updated config with new kafka topic

* Switched to samara producer and now sending messages

* Doc'd

* Put kafka update after the database insert

* Doc'd profileUpdate structure

* Removed unused parameter

* Moved user updates producer to clientapi/producers
2017-07-11 14:14:06 +01:00
Brendan Abolivier
1efbad8119 Profile API (#151)
* Profile retrieval

* Saving avatar (without propagating it)

* Saving display name (without propagating it)

* Getters for display name and avatar URL

* Doc'd

* Remove unused import

* Applied requested changes

* Added auth on PUT /profile/{userID}/...

* Improved error handling/reporting

* Using utils log reporting

* Removed useless checks
2017-07-10 14:52:41 +01:00
Mark Haines
69c29172c3 Use utility methods from gomatrixserverlib. (#152)
* Use utility methods from gomatrixserverlib, rather than reimplementing them

* Return string rather than pointer to string

* Update gomatrixserverlib
2017-07-07 14:11:32 +01:00
Brendan Abolivier
b13cbb18fb Response to createRoom is now the room's ID (#149) 2017-07-06 11:44:15 +01:00
Mark Haines
b2f9b8c641 Update the example configs with the new unified config (#146)
* Update the example configs with the new unified config

* Review comments

* Use an int for the config version
2017-07-05 13:10:28 +01:00
Mark Haines
8f1dca4e74 Add a component for sending event to remote matrix servers using fede… (#148)
* Add a component for sending event to remote matrix servers using federation

* Use named constants

* Comment on why we combine the deltas

* Comments

* Copyright headers!

* Fix the copy+Pasted comment

* Add tests for combineDeltas

* Review comments

* More review comments

* More comments

* Even more comments

* Name the mutexes

* Document what the mutexes are protecting

* Simplify the SQL, and more comments
2017-06-28 16:10:17 +01:00
Mark Haines
a6bb92520c Ignore hidden files and kafka directories 2017-06-28 10:28:13 +01:00
Mark Haines
54e7e3041b Set depth of events and whether they need to be federated. (#145)
* Set depth of events and whether they need to be federated.

Set the depth of each new event to one greater than the maximum depth
of it's direct ancestors.

Add a flag to each event passing through the roomserver that tells us
whether the event needs to be sent over federation.

We do this by passing the name of the server to send the event as.
We will need this capability if we want to support vhosting as it is
not possible to tell from the event alone which server to send it as.

(The reason for this is that sometimes a event needs to be sent on
behalf of a different remote matrix server)

* Update roomserver integration tests

* More comments

* Constants and comments

* Fix comments

* Document the int64 returned by LatestEventIDs
2017-06-27 15:28:44 +01:00
Mark Haines
572f6c399d Include the state before the event in roomserver output (#143)
* Include the state before the event in roomserver output

* Fix roomserver integration test

* Remove VisibilityEventIDs from the JSON

* More comments

* Remove spurious TODO
2017-06-27 13:20:04 +01:00
Mark Haines
524475f8a3 Stub APIs needed to let riot join a room (#147)
* Stub APIs needed to let riot join a room

* Fix comments
2017-06-27 12:37:25 +01:00
Matthew Hodgson
eb029116b0 trivial fix to actually parse fed-api-server's CLI args 2017-06-25 01:20:04 +01:00
Matthew Hodgson
cdd1fdcd3d trivial typo in usage output 2017-06-25 01:06:07 +01:00
Mark Haines
e67f9401be Replace the cmd specific config with common config. (#144)
* Move all the dendrite config in to a single place

* Add tests for config parsing

* replace syncserver config with common config

* Replace client API config with common config

* Replace federation API config with common config

* Replace media api config with common config

* Replace room server config with common config

* Remove unused readKey function

* Fix the integration tests

* Comment on hardcoding roomserver to HTTP

* Add a method for getting RoomServerURL

This moves the hardcoding of HTTPs into one place.
2017-06-19 15:21:04 +01:00
Mark Haines
3fbe728d94 Add utility functions for generating dendrite configs for integration tests (#142) 2017-06-19 12:56:55 +01:00
Mark Haines
c6e59ca967 Add a single config file format for dendrite. (#141)
* Add a single config file format for dendrite.

This config file combines all the configuration needed by the
dendrite components into a single file.

* Resolve relative paths against the current working directory

* Review comments

* more review comments!

* Document what the purpose of the version field is
2017-06-16 16:52:03 +01:00
Mark Haines
5e490e773f Add ineffassign to the pre-commit hooks (#139)
* Add ineffassign to the pre-comiit hooks

* Fix travis.yml

* Only apply ineffassign to the /src directory
2017-06-12 19:52:03 +01:00
Mark Haines
472155837b Add misspell and gofmt simplify to the pre-commit hooks (#138) 2017-06-12 18:30:47 +01:00
Mark Haines
7cbdab30f4 Add a database for storing the server keys (#137)
* Add a database for storing the server keys

* Tweak wording, and comment on the resolution of the timestamp

* Update gomatrixserverlib
2017-06-09 18:07:34 +01:00
Robert Swain
6eae6f7598 Foundation for media API testing (#136)
* cmd/mediaapi-integration-tests: Add foundation for testing

* common/test: Add some server init and client request utilities

* common/test/client: Handle timed out requests for tests that passed

* cmd/syncserver-integration-tests: Port to new common/test infra

* common/test/client: Remove stray debug print

* cmd/mediaapi-integration-tests: Simplify slice initialisation

* cmd/mediaapi-integration-tests: Simplify getMediaURL argument

* cmd/mediaapi-integration-tests: Make startMediaAPI return listen address

* common/test/client: Fix uninitialised LastRequestErr

* common/test/server: Remove redundant argument

* common/test/server: Add StartProxy to create a reverse proxy

* cmd/mediaapi-integration-tests: Add proxies in front of servers

This is needed so that origins can be correctly configured and used for
remote media.

* travis: Enable media API integration tests

* travis: Build the client-api-proxy for media tests

* common/test/client: Don't panic on EOF in CanonicalJSONInput

* cmd/mediaapi-integration-tests: Add upload/download/thumbnail tests

* mediaapi/thumbnailer: Store thumbnail according to requested size

* cmd/mediaapi-integration-tests: Add totem.jpg test file

* cmd/client-api-proxy: Optionally listen for HTTPS

* common/test/client: Do not verify TLS certs for testing

We will commonly use self-signed certs.

* cmd/mediaapi-integration-tests: Make HTTPS requests

* cmd/mediaapi-integration-tests: Log size and method for thumbnails

* mediaapi/thumbnailer: Factor out isThumbnailExists

Appease gocyclo^w^w simplify

* mediaapi/thumbnailer: Check if request is larger than original

* travis: Install openssl and generate server.{crt,key}

* cmd/mediaapi-integration-tests: Add valid dynamic thumbnail test

* cmd/mediaapi-integration-tests: Document state of tests

* cmd/mediaapi-integration-tests: Test remote thumbnail before download

This ordering also exercises the cold cache immediate generation of a
size configured for pregeneration.

* travis: Explain openssl key+cert generation

* common/test/server: Clarify postgresContainerName
2017-06-08 15:40:51 +02:00
Mark Haines
b184a48897 Fetching missing state from the roomserver. (#135)
* Fetching missing state from the roomserver.

Whenever the syncserver receives an event from the room server that adds
state that isn't in the syncserver's local database it should fetch
those state events from the roomserver.

* Fix append

* Put comment back

* Comments

* s/addsStateEvents/lookupStateEvents/

* Fix spelling

* Include the stream position that a state event was added at in the current state tables

* Fix comment

* Review comments
2017-06-07 16:35:41 +01:00
Mark Haines
515cce1a45 Add support for receiving room events over federation. (#130)
* Add API for querying events by ID.

* Fix tense

* Start implementing federation ingress

* More stuff

* Hook up federation event receiving

* Handle the case where we are missing state

* Fix docstring and comments

* Fix infinite loop when printing unknownRoomError
2017-06-07 14:32:53 +01:00
Robert Swain
2d202cec07 mediaapi: Add thumbnail support (#132)
* vendor: Add bimg image processing library

bimg is MIT licensed. It depends on the C library libvips which is LGPL
v2.1+ licensed. libvips must be installed separately.

* mediaapi: Add YAML config file support

* mediaapi: Add thumbnail support

* mediaapi: Add missing thumbnail files

* travis: Add ppa and install libvips-dev

* travis: Another ppa and install libvips-dev attempt

* travis: Add sudo: required for sudo apt* usage

* mediaapi/thumbnailer: Make comparison code more readable

* mediaapi: Simplify logging of thumbnail properties

* mediaapi/thumbnailer: Rename metrics to fitness

Metrics is used in the context of monitoring with Prometheus so renaming
to avoid confusion.

* mediaapi/thumbnailer: Use math.Inf() for max aspect and size

* mediaapi/thumbnailer: Limit number of parallel generators

Fall back to selecting from already-/pre-generated thumbnails or serving
the original.

* mediaapi/thumbnailer: Split bimg code into separate file

* vendor: Add github.com/nfnt/resize pure go image scaler

* mediaapi/thumbnailer: Add nfnt/resize thumbnailer

* travis: Don't install libvips-dev via ppa

* mediaapi: Add notes to README about resizers

* mediaapi: Elaborate on scaling libs in README
2017-06-07 01:12:49 +02:00
Robert Swain
def49400bc cmd/client-api-proxy: Add proxy for media API (#133) 2017-06-05 15:55:04 +02:00
Mark Haines
721c2df484 Clean up syncapi storage to match the coding style set by roomserver storage (#131) 2017-06-05 10:37:04 +01:00
Mark Haines
ef7b934d51 Add Query API for querying events by ID (#129) 2017-06-02 14:32:36 +01:00
Mark Haines
1b6e06aa59 Revert "Add API for querying events by ID. (#127)" (#128)
This reverts commit ce82158abb.
2017-06-02 14:24:13 +01:00
Mark Haines
ce82158abb Add API for querying events by ID. (#127)
* Add API for querying events by ID.

* Fix tense

* Start implementing federation ingress

* More stuff

* Hook up federation event receiving

* Fix comments

* Comment on the order of the arrays
2017-06-02 11:19:34 +01:00
Robert Swain
3b498c8074 Merge pull request #126 from matrix-org/rob/media-download-remote
Add support for downloading of remote files
2017-06-01 18:01:42 +02:00
Robert Swain
7244fc15b3 media/writers/download: Make use of CreateMediaDownloadRequest from lib 2017-06-01 17:57:05 +02:00
Robert Swain
0fff33928e vendor: Bump gomatrixserverlib to pull in CreateMediaDownloadRequest 2017-06-01 17:56:30 +02:00
Robert Swain
a398cdd193 mediaapi/writers/download: Use mime.ParseMediaType to parse Content-Disposition 2017-06-01 16:04:41 +02:00
Robert Swain
2d822c57f9 mediaapi/writers/download: Use DNS address as not all HSes have SRV 2017-06-01 15:30:36 +02:00
Robert Swain
cae309fa26 mediaapi/writers/download: Handle panic to ensure waking of goroutines
If the active request were to panic, we need to ensure all the waiting
goroutines get woken up.
2017-06-01 14:54:59 +02:00
Robert Swain
ce21b32d38 mediaapi/writers/download: Wrap broadcast call in closure to re-evaluate args 2017-06-01 14:44:00 +02:00
Robert Swain
4457ebddca mediaapi/writers/download: Rework remote file download synchronisation
Avoid locking around db requests by only locking around active requests
and always creating an active request if there is none. A nice side
effect of this is that if many parallel requests for remote media come
in (a common case) then only one database query is made for the duration
of the query.
2017-06-01 12:32:15 +02:00
Robert Swain
a3b1c7535a mediaapi/writers/download: Remove unnecesary Unlock, Lock after Cond.Wait 2017-06-01 08:39:35 +02:00
Robert Swain
923013aa55 mediaapi: Add support for downloading of remote files 2017-05-31 17:56:11 +02:00
Robert Swain
07c821daec mediaapi/writers: Fix InternalServerError responses 2017-05-31 17:41:42 +02:00
Robert Swain
57ba466da1 Merge pull request #125 from matrix-org/rob/media-download-local
mediaapi/writers/download: Add local download support
2017-05-31 15:47:43 +02:00
Robert Swain
2e013e3408 mediaapi/writers/download: Remove unnecessary closeConnection() 2017-05-31 15:39:19 +02:00
Robert Swain
bd96d99a3a mediaapi/writers/download: 500 ISE if disk and db file size differ 2017-05-31 15:08:00 +02:00
Robert Swain
e33438a37e mediaapi/writers/upload: Add standard fields to logger
So that they apply to all log messages thereafter
2017-05-31 14:54:10 +02:00