diff --git a/CHANGELOG.md b/CHANGELOG.md index 8452d2d7..329b2ea8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,196 @@ +# 2021-02-19 + +## GroupMe bridging support via mx-puppet-groupme + +Thanks to [Cody Neiman](https://github.com/xangelix), the playbook can now install the [mx-puppet-groupme](https://gitlab.com/robintown/mx-puppet-groupme) bridge for bridging to [GroupMe](https://groupme.com). + +This brings the total number of bridges supported by the playbook up to 18. See all supported bridges [here](docs/configuring-playbook.md#bridging-other-networks). + +To get started, follow our [Setting up MX Puppet GroupMe](docs/configuring-playbook-bridge-mx-puppet-groupme.md) docs. + +## Mautrix Instagram bridging support + +The playbook now supports bridging with [Instagram](https://www.instagram.com/) by installing the [mautrix-instagram](https://github.com/tulir/mautrix-instagram) bridge. This playbook functionality is available thanks to [@MarcProe](https://github.com/MarcProe). + +Additional details are available in [Setting up Mautrix Instagram bridging](docs/configuring-playbook-bridge-mautrix-instagram.md). + +## Synapse workers support + +After [lots and lots of work](https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456) (done over many months by [Marcel Partap](https://github.com/eMPee584), [Max Klenk](https://github.com/maxklenk), a few others from the [Technical University of Dresden, Germany](https://tu-dresden.de/) and various other contributors), support for Synapse workers has finally landed. + +Having support for workers makes the playbook suitable for larger homeserver deployments. + +Our setup is not yet perfect (we don't support all types of workers; scaling some of them (like `pusher`, `federation_sender`) beyond a single instance is not yet supported). Still, it's a great start and can already power homeservers with thousands of users, like the [Matrix deployment at TU Dresden](https://doc.matrix.tu-dresden.de/en/) discussed in [Matrix Live S06E09 - TU Dresden on their Matrix deployment](https://www.youtube.com/watch?v=UHJX2pmT2gk). + +By default, workers are disabled and Synapse runs as a single process (homeservers don't necessarily need the complexity and increased memory requirements of running a worker-based setup). + +To enable Synapse workers, follow our [Load balancing with workers](docs/configuring-playbook-synapse.md#load-balancing-with-workers) documentation. + + +# 2021-02-12 + +## (Potential Breaking Change) Monitoring/metrics support using Prometheus and Grafana + +Thanks to [@Peetz0r](https://github.com/Peetz0r), the playbook can now install a bunch of tools for monitoring your Matrix server: the [Prometheus](https://prometheus.io) time-series database server, the Prometheus [node-exporter](https://prometheus.io/docs/guides/node-exporter/) host metrics exporter, and the [Grafana](https://grafana.com/) web UI. + +To get get these installed, follow our [Enabling metrics and graphs (Prometheus, Grafana) for your Matrix server](docs/configuring-playbook-prometheus-grafana.md) docs page. + +This update comes with a **potential breaking change** for people who were already exposing Synapse metrics (for consumption via another Prometheus installation). From now on, `matrix_synapse_metrics_enabled: true` no longer exposes metrics publicly via matrix-nginx-proxy (at `https://matrix.DOMAIN/_synapse/metrics`). To do so, you'd need to explicitly set `matrix_nginx_proxy_proxy_synapse_metrics: true`. + + +# 2021-01-31 + +## Etherpad support + +Thanks to [@pushytoxin](https://github.com/pushytoxin), the playbook can now install the [Etherpad](https://etherpad.org) realtime collaborative text editor. It can be used in a [Jitsi](https://jitsi.org/) audio/video call or integrated as a widget into Matrix chat rooms via the [Dimension](https://dimension.t2bot.io) integration manager. + +To get it installed, follow [our Etherpad docs page](docs/configuring-playbook-etherpad.md). + + +# 2021-01-22 + +## (Breaking Change) Postgres changes that require manual intervention + +We've made a lot of changes to our Postgres setup and some manual action is required (described below). Sorry about the hassle. + +**TLDR**: people running an [external Postgres server](docs/configuring-playbook-external-postgres.md) don't need to change anything for now. Everyone else (the common/default case) is affected and manual intervention is required. + +### Why? + +- we had a default Postgres password (`matrix_postgres_connection_password: synapse-password`), which we think is **not ideal for security anymore**. We now ask you to generate/provide a strong password yourself. Postgres is normally not exposed outside the container network, making it relatively secure, but still: + - by tweaking the configuration, you may end up intentionally or unintentionally exposing your Postgres server to the local network (or even publicly), while still using the default default credentials (`synapse` + `synapse-password`) + - we can't be sure we trust all these services (bridges, etc). Some of them may try to talk to or attack `matrix-postgres` using the default credentials (`synapse` + `synapse-password`) + - you may have other containers running on the same Docker network, which may try to talk to or attack `matrix-postgres` using the default credentials (`synapse` + `synapse-password`) +- our Postgres usage **was overly-focused on Synapse** (default username of `synapse` and default/main database of `homeserver`). Additional homeserver options are likely coming in the future ([Dendrite](https://matrix.org/docs/projects/server/dendrite), [Conduit](https://matrix.org/docs/projects/server/conduit), [The Construct](https://matrix.org/docs/projects/server/construct)), so being too focused on `matrix-synapse` is not great. From now on, Synapse is just another component of this playbook, which happens to have an *additional database* (called `synapse`) on the Postgres server. +- we try to reorganize things a bit, to make the playbook even friendlier to people running an [external Postgres server](docs/configuring-playbook-external-postgres.md). Work on this will proceed in the future. + +So, this is some **effort to improve security** and to **prepare for a brighter future of having more homeserver options** than just Synapse. + +### What has really changed? + +- the default superuser Postgres username is now `matrix` (used to be `synapse`) +- the default Postgres database is now `matrix` (used to be `homeserver`) +- Synapse's database is now `synapse` (used to be `homeserver`). This is now just another "additional database" that the playbook manages for you +- Synapse's user called `synapse` is just a regular user that can only use the `synapse` database (not a superuser anymore) + +### What do I do if I'm using the integrated Postgres server (default)? + +By default, the playbook runs an integrated Postgres server for you in a container (`matrix-postgres`). Unless you've explicitly configured an [external Postgres server](docs/configuring-playbook-external-postgres.md), these steps are meant for you. + +To migrate to the new setup, expect a few minutes of downtime, while you follow these steps: + +1. We believe the steps below are safe and you won't encounter any data loss, but consider [making a Postgres backup](docs/maintenance-postgres.md#backing-up-postgresql) anyway. If you've never backed up Postgres, now would be a good time to try it. + +2. Generate a strong password to be used for your superuser Postgres user (called `matrix`). You can use `pwgen -s 64 1` to generate it, or some other tool. The **maximum length** for a Postgres password is 100 bytes (characters). Don't go crazy! + +3. Update your playbook's `inventory/host_vars/matrix.DOMAIN/vars.yml` file, adding a line like this: +```yaml +matrix_postgres_connection_password: 'YOUR_POSTGRES_PASSWORD_HERE' +``` + +.. where `YOUR_POSTGRES_PASSWORD_HERE` is to be replaced with the password you generated during step #2. + +4. Stop all services: `ansible-playbook -i inventory/hosts setup.yml --tags=stop` +5. Log in to the server via SSH. The next commands will be performed there. +6. Start the Postgres database server: `systemctl start matrix-postgres` +7. Open a Postgres shell: `/usr/local/bin/matrix-postgres-cli` +8. Execute the following query, while making sure to **change the password inside** (**don't forget the ending `;`**): + +```sql +CREATE ROLE matrix LOGIN SUPERUSER PASSWORD 'YOUR_POSTGRES_PASSWORD_HERE'; +``` + +.. where `YOUR_POSTGRES_PASSWORD_HERE` is to be replaced with the password you generated during step #2. + +9. Execute the following queries as you see them (no modifications necessary, so you can just **paste them all at once**): + +```sql +CREATE DATABASE matrix OWNER matrix; + +ALTER DATABASE postgres OWNER TO matrix; +ALTER DATABASE template0 OWNER TO matrix; +ALTER DATABASE template1 OWNER TO matrix; + +\c matrix; + +ALTER DATABASE homeserver RENAME TO synapse; + +ALTER ROLE synapse NOSUPERUSER NOCREATEDB NOCREATEROLE; + +\quit +``` + +You may need to press *Enter* after pasting the lines above. + +10. Re-run the playbook normally: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +### What do I do if I'm using an external Postgres server? + +If you've explicitly configured an [external Postgres server](docs/configuring-playbook-external-postgres.md), there are **no changes** that you need to do at this time. + +The fact that we've renamed Synapse's database from `homeserver` to `synapse` (in our defaults) should not affect you, as you're already explicitly defining `matrix_synapse_database_database` (if you've followed our guide, that is). If you're not explicitly defining this variable, you may wish to do so (`matrix_synapse_database_database: homeserver`), to avoid the new `synapse` default and keep things as they were. + + +# 2021-01-20 + +## (Breaking Change) The mautrix-facebook bridge now requires a Postgres database + +A new version of the [mautrix-facebook](https://github.com/tulir/mautrix-facebook) bridge has been released. It's a full rewrite of its backend and the bridge now requires Postgres. New versions of the bridge can no longer run on SQLite. + +**TLDR**: if you're NOT using an [external Postgres server](docs/configuring-playbook-external-postgres.md) and have NOT forcefully kept the bridge on SQLite during [The big move to all-on-Postgres (potentially dangerous)](#the-big-move-to-all-on-postgres-potentially-dangerous), you will be automatically upgraded without manual intervention. All you need to do is send a `login` message to the Facebook bridge bot again. + +Whether this change requires your intervention depends mostly on: +- whether you're using an [external Postgres server](docs/configuring-playbook-external-postgres.md). If yes, then [you need to do something](#upgrade-path-for-people-running-an-external-postgres-server). +- or whether you've force-changed the bridge's database engine to SQLite (`matrix_mautrix_facebook_database_engine: 'sqlite'` in your `vars.yml`) some time in the past (likely during [The big move to all-on-Postgres (potentially dangerous)](#the-big-move-to-all-on-postgres-potentially-dangerous)). + +As already mentioned above, you most likely don't need to do anything. If you rerun the playbook and don't get an error, you've been automatically upgraded. Just send a `login` message to the Facebook bridge bot again. Otherwise, read below for a solution. + +### Upgrade path for people NOT running an external Postgres server (default for the playbook) + +If you're **not running an external Postgres server**, then this bridge either already works on Postgres for you, or you've intentionally kept it back on SQLite with custom configuration (`matrix_mautrix_facebook_database_engine: 'sqlite'` in your `vars.yml`) . + +Simply remove that custom configuration from your `vars.yml` file (if it's there) and re-run the playbook. It should upgrade you automatically. +You'll need to send a `login` message to the Facebook bridge bot again. + +Alternatively, [you can stay on SQLite for a little longer](#staying-on-sqlite-for-a-little-longer-temporary-solution). + +### Upgrade path for people running an external Postgres server + +For people using the internal Postgres server (the default for the playbook): +- we automatically create an additional `matrix_mautrix_facebook` Postgres database and credentials to access it +- we automatically adjust the bridge's `matrix_mautrix_facebook_database_*` variables to point the bridge to that Postgres database +- we use [pgloader](https://pgloader.io/) to automatically import the existing SQLite data for the bridge into the `matrix_mautrix_facebook` Postgres database + +If you are using an [external Postgres server](docs/configuring-playbook-external-postgres.md), unfortunately we currently can't do any of that for you. + +You have 3 ways to proceed: + +- contribute to the playbook to make this possible (difficult) +- or, do the migration "steps" manually: + - stop the bridge (`systemctl stop matrix-mautrix-facebook`) + - create a new `matrix_mautrix_facebook` Postgres database for it + - run [pgloader](https://pgloader.io/) manually (we import this bridge's data using default settings and it works well) + - define `matrix_mautrix_facebook_database_*` variables in your `vars.yml` file (credentials, etc.) - you can find their defaults in `roles/matrix-mautrix-facebook/defaults/main.yml` + - switch the bridge to Postgres (`matrix_mautrix_facebook_database_engine: 'postgres'` in your `vars.yml` file) + - re-run the playbook (`--tags=setup-all,start`) and ensure the bridge works (`systemctl status matrix-mautrix-facebook` and `journalctl -fu matrix-mautrix-facebook`) + - send a `login` message to the Facebook bridge bot again +- or, [stay on SQLite for a little longer (temporary solution)](#staying-on-sqlite-for-a-little-longer-temporary-solution) + +### Staying on SQLite for a little longer (temporary solution) + +To keep using this bridge with SQLite for a little longer (**not recommended**), use the following configuration in your `vars.yml` file: + +```yaml +# Force-change the database engine to SQLite. +matrix_mautrix_facebook_database_engine: 'sqlite' + +# Force-downgrade to the last bridge version which supported SQLite. +matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:da1b4ec596e334325a1589e70829dea46e73064b" +``` + +If you do this, keep in mind that **you can't run this forever**. This SQLite-supporting bridge version is not getting any updates and will break sooner or later. The playbook will also drop support for SQLite at some point in the future. + + # 2021-01-17 ## matrix-corporal goes 2.0 diff --git a/README.md b/README.md index b195a7f8..463a1504 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,16 @@ ## Purpose -This Ansible playbook is meant to easily let you run your own [Matrix](http://matrix.org/) homeserver. +This [Ansible](https://www.ansible.com/) playbook is meant to help you run your own [Matrix](http://matrix.org/) homeserver, along with the [various services](#supported-services) related to that. -That is, it lets you join the Matrix network with your own `@:` identifier, all hosted on your own server. +That is, it lets you join the Matrix network using your own `@:` identifier, all hosted on your own server (see [prerequisites](docs/prerequisites.md)). + +We run all services in [Docker](https://www.docker.com/) containers (see [the container images we use](docs/container-images.md)), which lets us have a predictable and up-to-date setup, across multiple supported distros (see [prerequisites](docs/prerequisites.md)) and [architectures](docs/alternative-architectures.md) (x86/amd64 being recommended). + +[Installation](docs/README.md) (upgrades) and some maintenance tasks are automated using [Ansible](https://www.ansible.com/) (see [our Ansible guide](docs/ansible.md)). + + +## Supported services Using this playbook, you can get the following services configured on your server: @@ -46,6 +53,8 @@ Using this playbook, you can get the following services configured on your serve - (optional) the [mautrix-hangouts](https://github.com/tulir/mautrix-hangouts) bridge for bridging your Matrix server to [Google Hangouts](https://en.wikipedia.org/wiki/Google_Hangouts) +- (optional) the [mautrix-instagram](https://github.com/tulir/mautrix-instagram) bridge for bridging your Matrix server to [Instagram](https://instagram.com/) + - (optional) the [mautrix-signal](https://github.com/tulir/mautrix-signal) bridge for bridging your Matrix server to [Signal](https://www.signal.org/) - (optional) the [matrix-appservice-irc](https://github.com/matrix-org/matrix-appservice-irc) bridge for bridging your Matrix server to [IRC](https://wikipedia.org/wiki/Internet_Relay_Chat) @@ -56,20 +65,28 @@ Using this playbook, you can get the following services configured on your serve - (optional) the [matrix-appservice-webhooks](https://github.com/turt2live/matrix-appservice-webhooks) bridge for slack compatible webhooks ([ConcourseCI](https://concourse-ci.org/), [Slack](https://slack.com/) etc. pp.) +- (optional) the [matrix-sms-bridge](https://github.com/benkuly/matrix-sms-bridge) for bridging your Matrix server to SMS - see [docs/configuring-playbook-bridge-matrix-bridge-sms.md](docs/configuring-playbook-bridge-matrix-bridge-sms.md) for setup documentation + +- (optional) the [mx-puppet-skype](https://hub.docker.com/r/sorunome/mx-puppet-skype) for bridging your Matrix server to [Skype](https://www.skype.com) - see [docs/configuring-playbook-bridge-mx-puppet-skype.md](docs/configuring-playbook-bridge-mx-puppet-skype.md) for setup documentation + +- (optional) the [mx-puppet-slack](https://hub.docker.com/r/sorunome/mx-puppet-slack) for bridging your Matrix server to [Slack](https://slack.com) - see [docs/configuring-playbook-bridge-mx-puppet-slack.md](docs/configuring-playbook-bridge-mx-puppet-slack.md) for setup documentation + - (optional) the [mx-puppet-instagram](https://github.com/Sorunome/mx-puppet-instagram) bridge for Instagram-DMs ([Instagram](https://www.instagram.com/)) - see [docs/configuring-playbook-bridge-mx-puppet-instagram.md](docs/configuring-playbook-bridge-mx-puppet-instagram.md) for setup documentation -- (optional) the [mx-puppet-twitter](https://github.com/Sorunome/mx-puppet-twitter) bridge for Twitter-DMs ([Twitter](https://twitter.com/) - see [docs/configuring-playbook-bridge-mx-puppet-twitter.md](docs/configuring-playbook-bridge-mx-puppet-twitter.md) for setup documentation +- (optional) the [mx-puppet-twitter](https://github.com/Sorunome/mx-puppet-twitter) bridge for Twitter-DMs ([Twitter](https://twitter.com/)) - see [docs/configuring-playbook-bridge-mx-puppet-twitter.md](docs/configuring-playbook-bridge-mx-puppet-twitter.md) for setup documentation -- (optional) the [mx-puppet-discord](https://github.com/matrix-discord/mx-puppet-discord) bridge for [Discord](https://discordapp.com/)) - see [docs/configuring-playbook-bridge-mx-puppet-discord.md](docs/configuring-playbook-bridge-mx-puppet-discord.md) for setup documentation +- (optional) the [mx-puppet-discord](https://github.com/matrix-discord/mx-puppet-discord) bridge for [Discord](https://discordapp.com/) - see [docs/configuring-playbook-bridge-mx-puppet-discord.md](docs/configuring-playbook-bridge-mx-puppet-discord.md) for setup documentation -- (optional) the [mx-puppet-steam](https://github.com/icewind1991/mx-puppet-steam) bridge for [Steam](https://steamapp.com/)) - see [docs/configuring-playbook-bridge-mx-puppet-steam.md](docs/configuring-playbook-bridge-mx-puppet-steam.md) for setup documentation +- (optional) the [mx-puppet-groupme](https://gitlab.com/robintown/mx-puppet-groupme) bridge for [GroupMe](https://groupme.com/) - see [docs/configuring-playbook-bridge-mx-puppet-groupme.md](docs/configuring-playbook-bridge-mx-puppet-groupme.md) for setup documentation -- (optional) the [matrix-sms-bridge](https://github.com/benkuly/matrix-sms-bridge) for bridging your Matrix server to SMS - see [docs/configuring-playbook-bridge-matrix-bridge-sms.md](docs/configuring-playbook-bridge-matrix-bridge-sms.md) for setup documentation +- (optional) the [mx-puppet-steam](https://github.com/icewind1991/mx-puppet-steam) bridge for [Steam](https://steamapp.com/) - see [docs/configuring-playbook-bridge-mx-puppet-steam.md](docs/configuring-playbook-bridge-mx-puppet-steam.md) for setup documentation - (optional) [Email2Matrix](https://github.com/devture/email2matrix) for relaying email messages to Matrix rooms - see [docs/configuring-playbook-email2matrix.md](docs/configuring-playbook-email2matrix.md) for setup documentation - (optional) [Dimension](https://github.com/turt2live/matrix-dimension), an open source integrations manager for matrix clients - see [docs/configuring-playbook-dimension.md](docs/configuring-playbook-dimension.md) for setup documentation +- (optional) [Etherpad](https://etherpad.org), an open source collaborative text editor - see [docs/configuring-playbook-etherpad.md](docs/configuring-playbook-etherpad.md) for setup documentation + - (optional) [Jitsi](https://jitsi.org/), an open source video-conferencing platform - see [docs/configuring-playbook-jitsi.md](docs/configuring-playbook-jitsi.md) for setup documentation - (optional) [matrix-reminder-bot](https://github.com/anoadragon453/matrix-reminder-bot) for scheduling one-off & recurring reminders and alarms - see [docs/configuring-playbook-bot-matrix-reminder-bot.md](docs/configuring-playbook-bot-matrix-reminder-bot.md) for setup documentation @@ -78,6 +95,8 @@ Using this playbook, you can get the following services configured on your serve - (optional) [matrix-registration](https://github.com/ZerataX/matrix-registration), a simple python application to have a token based matrix registration - see [docs/configuring-playbook-matrix-registration.md](docs/configuring-playbook-matrix-registration.md) for setup documentation +- (optional) the [Prometheus](https://prometheus.io) time-series database server, the Prometheus [node-exporter](https://prometheus.io/docs/guides/node-exporter/) host metrics exporter, and the [Grafana](https://grafana.com/) web UI - see [Enabling metrics and graphs (Prometheus, Grafana) for your Matrix server](docs/configuring-playbook-prometheus-grafana.md) for setup documentation + Basically, this playbook aims to get you up-and-running with all the basic necessities around Matrix, without you having to do anything else. **Note**: the list above is exhaustive. It includes optional or even some advanced components that you will most likely not need. @@ -85,33 +104,6 @@ Sticking with the defaults (which install a subset of the above components) is t You can always re-run the playbook later to add or remove components. -## What's different about this Ansible playbook? - -This is similar to the [EMnify/matrix-synapse-auto-deploy](https://github.com/EMnify/matrix-synapse-auto-deploy) Ansible deployment, but: - -- this one is a complete Ansible playbook (instead of just a role), so it's **easier to run** - especially for folks not familiar with Ansible - -- this one installs and hooks together **a lot more Matrix-related services** for you (see above) - -- this one **can be executed more than once** without causing trouble - -- works on various distros: **CentOS** (7.0+), Debian-based distributions (**Debian** 9/Stretch+, **Ubuntu** 16.04+), **Archlinux** - -- this one installs everything in a single directory (`/matrix` by default) and **doesn't "contaminate" your server** with files all over the place - -- this one **doesn't necessarily take over** ports 80 and 443. By default, it sets up nginx for you there, but you can also [use your own webserver](docs/configuring-playbook-own-webserver.md) - -- this one **runs everything in Docker containers**, so it's likely more predictable and less fragile (see [Docker images used by this playbook](#docker-images-used-by-this-playbook)) - -- this one retrieves and automatically renews free [Let's Encrypt](https://letsencrypt.org/) **SSL certificates** for you - -- this one optionally can store the `media_store` content repository files on [Amazon S3](https://aws.amazon.com/s3/) (but defaults to storing files on the server's filesystem) - -- this one optionally **allows you to use an external PostgreSQL server** for Synapse's database (but defaults to running one in a container) - -- helps you **import data from a previous installation** (so you can migrate your manual virtualenv/Docker setup to a more managed one) - - ## Installation To configure and install Matrix on your own server, follow the [README in the docs/ directory](docs/README.md). @@ -124,88 +116,6 @@ This playbook evolves over time, sometimes with backward-incompatible changes. When updating the playbook, refer to [the changelog](CHANGELOG.md) to catch up with what's new. -## Docker images used by this playbook - -This playbook sets up your server using the following Docker images: - -- [matrixdotorg/synapse](https://hub.docker.com/r/matrixdotorg/synapse/) - the official [Synapse](https://github.com/matrix-org/synapse) Matrix homeserver (optional) - -- [instrumentisto/coturn](https://hub.docker.com/r/instrumentisto/coturn/) - the [Coturn](https://github.com/coturn/coturn) STUN/TURN server (optional) - -- [vectorim/element-web](https://hub.docker.com/r/vectorim/element-web/) - the [Element](https://element.io/) web client (optional) - -- [ma1uta/ma1sd](https://hub.docker.com/r/ma1uta/ma1sd/) - the [ma1sd](https://github.com/ma1uta/ma1sd) Matrix Identity server (optional) - -- [postgres](https://hub.docker.com/_/postgres/) - the [Postgres](https://www.postgresql.org/) database server (optional) - -- [ewoutp/goofys](https://hub.docker.com/r/ewoutp/goofys/) - the [Goofys](https://github.com/kahing/goofys) Amazon [S3](https://aws.amazon.com/s3/) file-system-mounting program (optional) - -- [devture/exim-relay](https://hub.docker.com/r/devture/exim-relay/) - the [Exim](https://www.exim.org/) email server (optional) - -- [devture/email2matrix](https://hub.docker.com/r/devture/email2matrix/) - the [Email2Matrix](https://github.com/devture/email2matrix) email server, which can relay email messages to Matrix rooms (optional) - -- [devture/matrix-corporal](https://hub.docker.com/r/devture/matrix-corporal/) - [Matrix Corporal](https://github.com/devture/matrix-corporal): reconciliator and gateway for a managed Matrix server (optional) - -- [zeratax/matrix-registration](https://hub.docker.com/r/devture/zeratax-matrix-registration/) - [matrix-registration](https://github.com/ZerataX/matrix-registration): a simple python application to have a token based matrix registration (optional) - -- [nginx](https://hub.docker.com/_/nginx/) - the [nginx](http://nginx.org/) web server (optional) - -- [certbot/certbot](https://hub.docker.com/r/certbot/certbot/) - the [certbot](https://certbot.eff.org/) tool for obtaining SSL certificates from [Let's Encrypt](https://letsencrypt.org/) (optional) - -- [tulir/mautrix-telegram](https://mau.dev/tulir/mautrix-telegram/container_registry) - the [mautrix-telegram](https://github.com/tulir/mautrix-telegram) bridge to [Telegram](https://telegram.org/) (optional) - -- [tulir/mautrix-whatsapp](https://mau.dev/tulir/mautrix-whatsapp/container_registry) - the [mautrix-whatsapp](https://github.com/tulir/mautrix-whatsapp) bridge to [Whatsapp](https://www.whatsapp.com/) (optional) - -- [tulir/mautrix-facebook](https://mau.dev/tulir/mautrix-facebook/container_registry) - the [mautrix-facebook](https://github.com/tulir/mautrix-facebook) bridge to [Facebook](https://facebook.com/) (optional) - -- [tulir/mautrix-hangouts](https://mau.dev/tulir/mautrix-hangouts/container_registry) - the [mautrix-hangouts](https://github.com/tulir/mautrix-hangouts) bridge to [Google Hangouts](https://en.wikipedia.org/wiki/Google_Hangouts) (optional) - -- [tulir/mautrix-signal](https://mau.dev/tulir/mautrix-signal/container_registry) - the [mautrix-signal](https://github.com/tulir/mautrix-signal) bridge to [Signal](https://www.signal.org/) (optional) - -- [matrixdotorg/matrix-appservice-irc](https://hub.docker.com/r/matrixdotorg/matrix-appservice-irc) - the [matrix-appservice-irc](https://github.com/matrix-org/matrix-appservice-irc) bridge to [IRC](https://wikipedia.org/wiki/Internet_Relay_Chat) (optional) - -- [halfshot/matrix-appservice-discord](https://hub.docker.com/r/halfshot/matrix-appservice-discord) - the [matrix-appservice-discord](https://github.com/Half-Shot/matrix-appservice-discord) bridge to [Discord](https://discordapp.com/) (optional) - -- [cadair/matrix-appservice-slack](https://hub.docker.com/r/cadair/matrix-appservice-slack) - the [matrix-appservice-slack](https://github.com/matrix-org/matrix-appservice-slack) bridge to [Slack](https://slack.com/) (optional) - -- [turt2live/matrix-appservice-webhooks](https://hub.docker.com/r/turt2live/matrix-appservice-webhooks) - the [Appservice Webhooks](https://github.com/turt2live/matrix-appservice-webhooks) bridge (optional) - -- [folivonet/matrix-sms-bridge](https://hub.docker.com/repository/docker/folivonet/matrix-sms-bridge) - the [matrix-sms-brdige](https://github.com/benkuly/matrix-sms-bridge) (optional) - -- [sorunome/mx-puppet-skype](https://hub.docker.com/r/sorunome/mx-puppet-skype) - the [mx-puppet-skype](https://github.com/Sorunome/mx-puppet-skype) bridge to [Skype](https://www.skype.com) (optional) - -- [sorunome/mx-puppet-slack](https://hub.docker.com/r/sorunome/mx-puppet-slack) - the [mx-puppet-slack](https://github.com/Sorunome/mx-puppet-slack) bridge to [Slack](https://slack.com) (optional) - -- [sorunome/mx-puppet-instagram](https://hub.docker.com/r/sorunome/mx-puppet-instagram) - the [mx-puppet-instagram](https://github.com/Sorunome/mx-puppet-instagram) bridge to [Instagram](https://www.instagram.com) (optional) - -- [sorunome/mx-puppet-twitter](https://hub.docker.com/r/sorunome/mx-puppet-twitter) - the [mx-puppet-twitter](https://github.com/Sorunome/mx-puppet-twitter) bridge to [Twitter](https://twitter.com) (optional) - -- [sorunome/mx-puppet-discord](https://hub.docker.com/r/sorunome/mx-puppet-discord) - the [mx-puppet-discord](https://github.com/matrix-discord/mx-puppet-discord) bridge to [Discord](https://discordapp.com) (optional) - -- [icewind1991/mx-puppet-steam](https://hub.docker.com/r/icewind1991/mx-puppet-steam) - the [mx-puppet-steam](https://github.com/icewind1991/mx-puppet-steam) bridge to [Steam](https://steampowered.com) (optional) - -- [turt2live/matrix-dimension](https://hub.docker.com/r/turt2live/matrix-dimension) - the [Dimension](https://dimension.t2bot.io/) integrations manager (optional) - -- [jitsi/web](https://hub.docker.com/r/jitsi/web) - the [Jitsi](https://jitsi.org/) web UI (optional) - -- [jitsi/jicofo](https://hub.docker.com/r/jitsi/jicofo) - the [Jitsi](https://jitsi.org/) Focus component (optional) - -- [jitsi/prosody](https://hub.docker.com/r/jitsi/prosody) - the [Jitsi](https://jitsi.org/) Prosody XMPP server component (optional) - -- [jitsi/jvb](https://hub.docker.com/r/jitsi/jvb) - the [Jitsi](https://jitsi.org/) Video Bridge component (optional) - -- [anoa/matrix-reminder-bot](https://hub.docker.com/r/anoa/matrix-reminder-bot) - the [matrix-reminder-bot](https://github.com/anoadragon453/matrix-reminder-bot) bot for one-off & recurring reminders and alarms (optional) - -- [awesometechnologies/synapse-admin](https://hub.docker.com/r/awesometechnologies/synapse-admin) - the [synapse-admin](https://github.com/Awesome-Technologies/synapse-admin) web UI tool for administrating users and rooms on your Matrix server (optional) - - -## Deficiencies - -This Ansible playbook can be improved in the following ways: - -- setting up automatic backups to one or more storage providers - - ## Support - Matrix room: [#matrix-docker-ansible-deploy:devture.com](https://matrix.to/#/#matrix-docker-ansible-deploy:devture.com) diff --git a/ansible.cfg b/ansible.cfg index 0ee3babf..224e3c44 100644 --- a/ansible.cfg +++ b/ansible.cfg @@ -2,6 +2,10 @@ pipelining = True retry_files_enabled = False stdout_callback = yaml +# trying allow world readable to bypass a weird issue with ansible +# see: https://github.com/georchestra/ansible/issues/55 +allow_world_readable_tmpfiles=true + vault_password_file=open_the_vault.sh ansible_python_interpreter=/usr/bin/python3 interpreter_python=/usr/bin/python3 diff --git a/docs/README.md b/docs/README.md index 8c95eff3..77c5099d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,8 @@ # Table of Contents -- [Prerequisites](prerequisites.md) +- [FAQ](faq.md) - lots of questions and answers. Jump to [Prerequisites](prerequisites.md) to avoid reading too much and to just start a guided installation. + +- [Prerequisites](prerequisites.md) - go here to a guided installation using this Ansible playbook - [Configuring your DNS server](configuring-dns.md) diff --git a/docs/alternative-architectures.md b/docs/alternative-architectures.md index 7a3e35eb..80749adf 100644 --- a/docs/alternative-architectures.md +++ b/docs/alternative-architectures.md @@ -21,6 +21,6 @@ matrix_architecture: "arm32" ## Implementation details -For `amd64`, prebuilt images are used everywhere (because all images are available for this architecture). +For `amd64`, prebuilt container images (see the [container images we use](container-images.md)) are used everywhere, because all images are available for this architecture. For other architectures, components which have a prebuilt image make use of it. If the component is not available for the specific architecture, [self-building](self-building.md) will be used. Not all components support self-building though, so your mileage may vary. diff --git a/docs/configuring-dns.md b/docs/configuring-dns.md index 9d738477..c25b079e 100644 --- a/docs/configuring-dns.md +++ b/docs/configuring-dns.md @@ -15,20 +15,25 @@ As we discuss in [Server Delegation](howto-server-delegation.md), there are 2 di This playbook mostly discusses the well-known file method, because it's easier to manage with regard to certificates. If you decide to go with the alternative method ([Server Delegation via a DNS SRV record (advanced)](howto-server-delegation.md#server-delegation-via-a-dns-srv-record-advanced)), please be aware that the general flow that this playbook guides you through may not match what you need to do. - -## General outline of DNS settings you need to do +## Required DNS settings for services enabled by default | Type | Host | Priority | Weight | Port | Target | | ----- | ---------------------------- | -------- | ------ | ---- | ---------------------- | | A | `matrix` | - | - | - | `matrix-server-IP` | | CNAME | `element` | - | - | - | `matrix.` | -| CNAME | `dimension` (*) | - | - | - | `matrix.` | -| CNAME | `jitsi` (*) | - | - | - | `matrix.` | | SRV | `_matrix-identity._tcp` | 10 | 0 | 443 | `matrix.` | +Be mindful as to how long it will take for the DNS records to propagate. -DNS records marked with `(*)` above are optional. They refer to services that will not be installed by default (see the section below). If you won't be installing these services, feel free to skip creating these DNS records. Also be mindful as to how long it will take for the DNS records to propagate. +If you are using Cloudflare DNS, make sure to disable the proxy and set all records to `DNS only`. Otherwise, fetching certificates will fail. +## Required DNS settings for optional services + +| Type | Host | Priority | Weight | Port | Target | +| ----- | ---------------------------- | -------- | ------ | ---- | ---------------------- | +| CNAME | `dimension` (*) | - | - | - | `matrix.` | +| CNAME | `jitsi` (*) | - | - | - | `matrix.` | +| CNAME | `stats` (*) | - | - | - | `matrix.` | ## Subdomains setup @@ -41,6 +46,8 @@ The `dimension.` subdomain may be necessary, because this playbook The `jitsi.` subdomain may be necessary, because this playbook could install the [Jitsi video-conferencing platform](https://jitsi.org/) for you. Jitsi installation is disabled by default, because it may be heavy and is not a core required component. To learn how to install it, see our [Jitsi](configuring-playbook-jitsi.md) guide. If you do not wish to set up Jitsi, feel free to skip the `jitsi.` DNS record. +The `stats.` subdomain may be necessary, because this playbook could install [Grafana](https://grafana.com/) and setup performance metrics for you. Grafana installation is disabled by default, it is not a core required component. To learn how to install it, see our [metrics and graphs guide](configuring-playbook-prometheus-grafana.md). If you do not wish to set up Grafana, feel free to skip the `stats.` DNS record. It is possible to install Prometheus without installing Grafana, this would also not require the `stats.` subdomain. + ## `_matrix-identity._tcp` SRV record setup diff --git a/docs/configuring-playbook-bridge-appservice-discord.md b/docs/configuring-playbook-bridge-appservice-discord.md index f3efc555..82a2edc2 100644 --- a/docs/configuring-playbook-bridge-appservice-discord.md +++ b/docs/configuring-playbook-bridge-appservice-discord.md @@ -38,8 +38,9 @@ To [adjust room access privileges](#adjusting-room-access-privileges) or do vari There's the Discord bridge's guide for [setting privileges on bridge managed rooms](https://github.com/Half-Shot/matrix-appservice-discord/blob/master/docs/howto.md#set-privileges-on-bridge-managed-rooms). To do the same with our container setup, run the following command on the server: -``` -docker exec -it matrix-appservice-discord /bin/sh -c 'cp /build/tools/adminme.js /tmp/adminme.js && cp /cfg/registration.yaml /tmp/discord-registration.yaml && cd /tmp && node /tmp/adminme.js -c /cfg/config.yaml -r "!ROOM_ID:SERVER" -u "@USER:SERVER" -p 100' +```sh +docker exec -it matrix-appservice-discord \ +/bin/sh -c 'cp /cfg/registration.yaml /tmp/discord-registration.yaml && cd /tmp && node /build/tools/adminme.js -c /cfg/config.yaml -m "!ROOM_ID:SERVER" -u "@USER:SERVER" -p 100' ``` diff --git a/docs/configuring-playbook-bridge-appservice-slack.md b/docs/configuring-playbook-bridge-appservice-slack.md index 371195b6..a409a050 100644 --- a/docs/configuring-playbook-bridge-appservice-slack.md +++ b/docs/configuring-playbook-bridge-appservice-slack.md @@ -27,7 +27,7 @@ matrix_appservice_slack_control_room_id: "Your matrix admin room id" Note that the bot's domain is your server's domain **without the `matrix.` prefix.** -5. Create a new Slack App [here](https://api.slack.com/apps). +5. Create a Classic Slack App [here](https://api.slack.com/apps?new_classic_app=1). Name the app "matrixbot" (or anything else you'll remember). diff --git a/docs/configuring-playbook-bridge-appservice-webhooks.md b/docs/configuring-playbook-bridge-appservice-webhooks.md index a4851146..3654bfa4 100644 --- a/docs/configuring-playbook-bridge-appservice-webhooks.md +++ b/docs/configuring-playbook-bridge-appservice-webhooks.md @@ -45,7 +45,7 @@ matrix_appservice_webhooks_log_level: '' "text": "Hello world!", "format": "plain", "displayName": "My Cool Webhook", - "avatarUrl": "http://i.imgur.com/IDOBtEJ.png" + "avatar_url": "http://i.imgur.com/IDOBtEJ.png" } ``` @@ -57,7 +57,7 @@ curl --header "Content-Type: application/json" \ "text": "Hello world!", "format": "plain", "displayName": "My Cool Webhook", -"avatarUrl": "http://i.imgur.com/IDOBtEJ.png" +"avatar_url": "http://i.imgur.com/IDOBtEJ.png" }' \ ``` diff --git a/docs/configuring-playbook-bridge-mautrix-instagram.md b/docs/configuring-playbook-bridge-mautrix-instagram.md new file mode 100644 index 00000000..7cdbc7a8 --- /dev/null +++ b/docs/configuring-playbook-bridge-mautrix-instagram.md @@ -0,0 +1,17 @@ +# Setting up Mautrix Instagram (optional) + +The playbook can install and configure [mautrix-instagram](https://github.com/tulir/mautrix-instagram) for you. + +See the project's [documentation](https://docs.mau.fi/bridges/python/instagram/index.html) to learn what it does and why it might be useful to you. + +```yaml +matrix_mautrix_instagram_enabled: true +``` + +## Usage + +You then need to start a chat with `@instagrambot:YOUR_DOMAIN` (where `YOUR_DOMAIN` is your base domain, not the `matrix.` domain). + +Send `login YOUR_INSTAGRAM_EMAIL_ADDRESS YOUR_INSTAGRAM_PASSWORD` to the bridge bot to enable bridging for your instagram/Messenger account. + +You can learn more here about authentication from the bridge's [official documentation on Authentication](https://docs.mau.fi/bridges/python/instagram/authentication.html). diff --git a/docs/configuring-playbook-bridge-mautrix-signal.md b/docs/configuring-playbook-bridge-mautrix-signal.md index 164b06de..6d3c4dfb 100644 --- a/docs/configuring-playbook-bridge-mautrix-signal.md +++ b/docs/configuring-playbook-bridge-mautrix-signal.md @@ -14,7 +14,7 @@ matrix_mautrix_signal_enabled: true ## Set up Double Puppeting -If you'd like to use [Double Puppeting](https://github.com/tulir/mautrix-whatsapp/wiki/Authentication#replacing-whatsapp-accounts-matrix-puppet-with-matrix-account) (hint: you most likely do), you have 2 ways of going about it. +If you'd like to use [Double Puppeting](https://github.com/tulir/mautrix-signal/wiki/Authentication#double-puppeting) (hint: you most likely do), you have 2 ways of going about it. ### Method 1: automatically, by enabling Shared Secret Auth diff --git a/docs/configuring-playbook-bridge-mx-puppet-groupme.md b/docs/configuring-playbook-bridge-mx-puppet-groupme.md new file mode 100644 index 00000000..c3b9663f --- /dev/null +++ b/docs/configuring-playbook-bridge-mx-puppet-groupme.md @@ -0,0 +1,38 @@ +# Setting up MX Puppet GroupMe (optional) + +The playbook can install and configure +[mx-puppet-groupme](https://gitlab.com/robintown/mx-puppet-groupme) for you. + +See the project page to learn what it does and why it might be useful to you. + +To enable the [GroupMe](https://groupme.com/) bridge just use the following +playbook configuration: + + +```yaml +matrix_mx_puppet_groupme_enabled: true +matrix_mx_puppet_groupme_client_id: "" +matrix_mx_puppet_groupme_client_secret: "" +``` + + +## Usage + +Once the bot is enabled you need to start a chat with `GroupMe Puppet Bridge` with +the handle `@_groupmepuppet_bot:YOUR_DOMAIN` (where `YOUR_DOMAIN` is your base +domain, not the `matrix.` domain). + +One authentication method is available. + +To link your GroupMe account, go to [dev.groupme.com](https://dev.groupme.com/), sign in, and select "Access Token" from the top menu. Copy the token and message the bridge with: + +``` +link +``` + +Once logged in, send `listrooms` to the bot user to list the available rooms. + +Clicking rooms in the list will result in you receiving an invitation to the +bridged room. + +Also send `help` to the bot to see the commands available. diff --git a/docs/configuring-playbook-etherpad.md b/docs/configuring-playbook-etherpad.md new file mode 100644 index 00000000..e5533e71 --- /dev/null +++ b/docs/configuring-playbook-etherpad.md @@ -0,0 +1,31 @@ +# Setting up Etherpad (optional) + +[Etherpad](https://etherpad.org) is is an open source collaborative text editor that can be embedded in a Matrix chat room using the [Dimension integrations manager](https://dimension.t2bot.io) + +When enabled together with the Jitsi audio/video conferencing system (see [our docs on Jitsi](configuring-playbook-jitsi.md)), it will be made available as an option during the conferences. + +## Prerequisites + +For the self-hosted Etherpad instance to be available to your users, you must first enable and configure the **Dimension integrations manager** as described in [the playbook documentation](configuring-playbook-dimension.md) + +## Installing + +[Etherpad](https://etherpad.org) installation is disabled by default. You can enable it in your configuration file (`inventory/host_vars/matrix./vars.yml`): + +```yaml +matrix_etherpad_enabled: true +``` + +## Set Dimension default to the self-hosted Etherpad + +The Dimension administrator users can configure the default URL template. The Dimension configuration menu can be accessed with the sprocket icon as you begin to add a widget to a room in Element. There you will find the Etherpad Widget Configuration action beneath the _Widgets_ tab. Replace `scalar.vector.im` with your own Dimension domain. + +### Removing the integrated Etherpad chat + +If you wish to disable the Etherpad chat button, you can do it by appending `?showChat=false` to the end of the pad URL, or the template. +Example: `https://dimension./etherpad/p/$roomId_$padName?showChat=false` + +## Known issues + +If your Etherpad widget fails to load, this might be due to Dimension generating a Pad name so long, the Etherpad app rejects it. +`$roomId_$padName` can end up being longer than 50 characters. You can avoid having this problem by altering the template so it only contains the three word random identifier `$padName`. diff --git a/docs/configuring-playbook-external-postgres.md b/docs/configuring-playbook-external-postgres.md index f3671a64..0becc8ff 100644 --- a/docs/configuring-playbook-external-postgres.md +++ b/docs/configuring-playbook-external-postgres.md @@ -4,6 +4,7 @@ By default, this playbook would set up a PostgreSQL database server on your mach If that's alright, you can skip this. If you'd like to use an external PostgreSQL server that you manage, you can edit your configuration file (`inventory/host_vars/matrix./vars.yml`). + It should be something like this: ```yaml diff --git a/docs/configuring-playbook-matrix-corporal.md b/docs/configuring-playbook-matrix-corporal.md index 15de634e..fb12e94a 100644 --- a/docs/configuring-playbook-matrix-corporal.md +++ b/docs/configuring-playbook-matrix-corporal.md @@ -89,4 +89,4 @@ The following local filesystem paths are mounted in the `matrix-corporal` contai - `/matrix/corporal/cache` is mounted at `/var/cache/matrix-corporal` (read and write) -As an example: you can create your own configuration files in `/matrix/corporal/config` and they will appear in `/etc/matrix-corporal` in the Docker container. Your configuration (stuff in `matrix_corporal_policy_provider_config`) needs to refer to these files via the local container path `/etc/matrix-corporal` +As an example: you can create your own configuration files in `/matrix/corporal/config` and they will appear in `/etc/matrix-corporal` in the Docker container. Your configuration (stuff in `matrix_corporal_policy_provider_config`) needs to refer to these files via the local container paths - `/etc/matrix-corporal` (read-only), `/var/matrix-corporal` (read and write), `/var/cache/matrix-corporal` (read and write). diff --git a/docs/configuring-playbook-nginx.md b/docs/configuring-playbook-nginx.md index ba6c5c12..c8500b37 100644 --- a/docs/configuring-playbook-nginx.md +++ b/docs/configuring-playbook-nginx.md @@ -34,8 +34,7 @@ Possible values are: - `"intermediate"` (**default**) - Recommended configuration for a general-purpose server - `"old"` - Services accessed by very old clients or libraries, such as Internet Explorer 8 (Windows XP), Java 6, or OpenSSL 0.9.8 -**Be really carefull when setting it to `"modern"`**. This could break comunication with other Matrix servers, limiting your federation posibilities. The -[Federarion tester](https://federationtester.matrix.org/) also won't work. +**Be really carefull when setting it to `"modern"`**. This could break comunication with other Matrix servers, limiting your federation posibilities. Besides changing the preset (`matrix_nginx_proxy_ssl_preset`), you can also directly override these 3 variables: @@ -60,3 +59,26 @@ This will disable the access logging for nginx. ```yaml matrix_nginx_proxy_access_log_enabled: false ``` + +## Additional configuration + +This playbook also allows for additional configuration to be applied to the nginx server. + +If you want this playbook to obtain and renew certificates for other domains, then you can set the `matrix_ssl_additional_domains_to_obtain_certificates_for` variable (as mentioned in the [Obtaining SSL certificates for additional domains](configuring-playbook-ssl-certificates.md#obtaining-ssl-certificates-for-additional-domains) documentation as well). Make sure that you have set the DNS configuration for the domains you want to include to point at your server. + +```yaml +matrix_ssl_additional_domains_to_obtain_certificates_for: + - domain.one.example + - domain.two.example +``` + +You can include additional nginx configuration by setting the `matrix_nginx_proxy_proxy_http_additional_server_configuration_blocks` variable. + +```yaml +matrix_nginx_proxy_proxy_http_additional_server_configuration_blocks: + - | + # These lines will be included in the nginx configuration. + # This is at the top level of the file, so you will need to define all of the `server { ... }` blocks. + - | + # For advanced use, have a look at the template files in `roles/matrix-nginx-proxy/templates/nginx/conf.d` +``` diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md new file mode 100644 index 00000000..a10497cc --- /dev/null +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -0,0 +1,66 @@ +# Enabling metrics and graphs for your Matrix server (optional) + +It can be useful to have some (visual) insight into the performance of your homeserver. + +You can enable this with the following settings in your configuration file (`inventory/host_vars/matrix./vars.yml`): + +```yaml +matrix_prometheus_enabled: true + +matrix_prometheus_node_exporter_enabled: true + +matrix_grafana_enabled: true + +matrix_grafana_anonymous_access: false + +# This has no relation to your Matrix user id. It can be any username you'd like. +# Changing the username subsequently won't work. +matrix_grafana_default_admin_user: some_username_chosen_by_you + +# Passwords containing special characters may be troublesome. +# Changing the password subsequently won't work. +matrix_grafana_default_admin_password: some_strong_password_chosen_by_you +``` + +By default, a [Grafana](https://grafana.com/) web user-interface will be available at `https://stats.`. + + +## What does it do? + +Name | Description +-----|---------- +`matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. +`matrix_prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures +`matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in +`matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option. +`matrix_grafana_default_admin_user`
`matrix_grafana_default_admin_password`|By default Grafana creates a user with `admin` as the username and password. If you feel this is insecure and you want to change it beforehand, you can do that here + + +## Security and privacy + +Metrics and resulting graphs can contain a lot of information. This includes system specs but also usage patterns. This applies especially to small personal/family scale homeservers. Someone might be able to figure out when you wake up and go to sleep by looking at the graphs over time. Think about this before enabling anonymous access. And you should really not forget to change your Grafana password. + +Most of our docker containers run with limited system access, but the `prometheus-node-exporter` has access to the host network stack and (readonly) root filesystem. This is required to report on them. If you don't like that, you can set `matrix_prometheus_node_exporter_enabled: false` (which is actually the default). You will still get Synapse metrics with this container disabled. Both of the dashboards will always be enabled, so you can still look at historical data after disabling either source. + + +## Collecting metrics to an external Prometheus server + +If you wish, you could expose homeserver metrics without enabling (installing) Prometheus and Grafana via the playbook. This may be useful for hooking Matrix services to an external Prometheus/Grafana installation. + +To do this, you may be interested in the following variables: + +Name | Description +-----|---------- +`matrix_synapse_metrics_enabled`|Set this to `true` to make Synapse expose metrics (locally, on the container network) +`matrix_nginx_proxy_proxy_synapse_metrics`|Set this to `true` to make matrix-nginx-proxy expose the Synapse metrics at `https://matrix.DOMAIN/_synapse/metrics` +`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled`|Set this to `true` to password-protect (using HTTP Basic Auth) `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus`, the password is defined in `matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`) +`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`|Set this to a password to use for HTTP Basic Auth for protecting `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus` - it's not configurable) + + +## More inforation + +- [Understanding Synapse Performance Issues Through Grafana Graphs](https://github.com/matrix-org/synapse/wiki/Understanding-Synapse-Performance-Issues-Through-Grafana-Graphs) at the Synapse Github Wiki +- [The Prometheus scraping rules](https://github.com/matrix-org/synapse/tree/master/contrib/prometheus) (we use v2) +- [The Synapse Grafana dashboard](https://github.com/matrix-org/synapse/tree/master/contrib/grafana) +- [The Node Exporter dashboard](https://github.com/rfrail3/grafana-dashboards) (for generic non-synapse performance graphs) + diff --git a/docs/configuring-playbook-s3.md b/docs/configuring-playbook-s3.md index 643edb5b..9132ff71 100644 --- a/docs/configuring-playbook-s3.md +++ b/docs/configuring-playbook-s3.md @@ -6,6 +6,11 @@ If that's alright, you can skip this. If you'd like to store Synapse's content repository (`media_store`) files on Amazon S3 (or other S3-compatible service), you can let this playbook configure [Goofys](https://github.com/kahing/goofys) for you. +Using a Goofys-backed media store works, but performance may not be ideal. If possible, try to use a region which is close to your Matrix server. + +If you'd like to move your locally-stored media store data to Amazon S3 (or another S3-compatible object store), we also provide some migration instructions below. + + ## Amazon S3 You'll need an Amazon S3 bucket and some IAM user credentials (access key + secret key) with full write access to the bucket. Example security policy: @@ -50,3 +55,133 @@ matrix_s3_media_store_custom_endpoint_enabled: true # Example: "https://storage.googleapis.com" matrix_s3_media_store_custom_endpoint: "your-custom-endpoint" ``` + +### Backblaze B2 + +To use [Backblaze B2](https://www.backblaze.com/b2/cloud-storage.html): + +- create a new **private** bucket through its user interface (you can call it something like `matrix-DOMAIN-media-store`) +- note the **Endpoint** for your bucket (something like `s3.us-west-002.backblazeb2.com`) +- adjust its lifecycle rules to use the following **custom** rules: + - File Path: *empty value* + - Days Till Hide: *empty value* + - Days Till Delete: `1` +- go to [App Keys](https://secure.backblaze.com/app_keys.htm) and use the **Add a New Application Key** to create a new one + - restrict it to the previously created bucket (e.g. `matrix-DOMAIN-media-store`) + - give it *Read & Write* access + +Copy the `keyID` and `applicationKey`. + +You need the following *additional* playbook configuration (on top of what you see above): + +```yaml +matrix_s3_media_store_bucket_name: "YOUR_BUCKET_NAME_GOES_HERE" +matrix_s3_media_store_aws_access_key: "YOUR_keyID_GOES_HERE" +matrix_s3_media_store_aws_secret_key: "YOUR_applicationKey_GOES_HERE" +matrix_s3_media_store_custom_endpoint_enabled: true +matrix_s3_media_store_custom_endpoint: "https://s3.us-west-002.backblazeb2.com" # this may be different for your bucket +``` + +If you have local media store files and wish to migrate to Backblaze B2 subsequently, follow our [migration guide to Backblaze B2](#migrating-to-backblaze-b2) below instead of applying this configuration as-is. + + +## Migrating from local filesystem storage to S3 + +It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before migrating your local media store to an S3-backed one. + +Follow one of the guides below for a migration path from a locally-stored media store to one stored on S3-compatible storage: + +- [Migrating to any S3-compatible storage (universal, but likely slow)](#migrating-to-any-s3-compatible-storage-universal-but-likely-slow) +- [Migrating to Backblaze B2](#migrating-to-backblaze-b2) + +### Migrating to any S3-compatible storage (universal, but likely slow) + +It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before doing this. + +1. Proceed with the steps below without stopping Matrix services + +2. Start by adding the base S3 configuration in your `vars.yml` file (seen above, may be different depending on the S3 provider of your choice) + +3. In addition to the base configuration you see above, add this to your `vars.yml` file: + +```yaml +matrix_s3_media_store_path: /matrix/s3-media-store +``` + +This enables S3 support, but mounts the S3 storage bucket to `/matrix/s3-media-store` without hooking it to your homeserver yet. Your homeserver will still continue using your local filesystem for its media store. + +5. Run the playbook to apply the changes: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +6. Do an **initial sync of your files** by running this **on the server** (it may take a very long time): + +```sh +sudo -u matrix -- rsync --size-only --ignore-existing -avr /matrix/synapse/storage/media-store/. /matrix/s3-media-store/. +``` + +You may need to install `rsync` manually. + +7. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`) + +8. Start the S3 service by running this **on the server**: `systemctl start matrix-goofys` + +9. Sync the files again by re-running the `rsync` command you see in step #6 + +10. Stop the S3 service by running this **on the server**: `systemctl stop matrix-goofys` + +11. Get the old media store out of the way by running this command on the server: + +```sh +mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup +``` + +12. Remove the `matrix_s3_media_store_path` configuration from your `vars.yml` file (undoing step #3 above) + +13. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +14. You're done! Verify that loading existing (old) media files works and that you can upload new ones. + +15. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup` + + +### Migrating to Backblaze B2 + +It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before doing this. + +1. While all Matrix services are running, run the following command on the server: + +(you need to adjust the 3 `--env` line below with your own data) + +```sh +docker run -it --rm -w /work \ +--env='B2_KEY_ID=YOUR_KEY_GOES_HERE' \ +--env='B2_KEY_SECRET=YOUR_SECRET_GOES_HERE' \ +--env='B2_BUCKET_NAME=YOUR_BUCKET_NAME_GOES_HERE' \ +-v /matrix/synapse/storage/media-store/:/work \ +--entrypoint=/bin/sh \ +docker.io/tianon/backblaze-b2:2.1.0 \ +-c 'b2 authorize-account $B2_KEY_ID $B2_KEY_SECRET > /dev/null && b2 sync /work/ b2://$B2_BUCKET_NAME' +``` + +This is some initial file sync, which may take a very long time. + +2. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`) + +3. Run the command from step #1 again. + +Doing this will sync any new files that may have been created locally in the meantime. + +Now that Matrix services aren't running, we're sure to get Backblaze B2 and your local media store fully in sync. + +4. Get the old media store out of the way by running this command on the server: + +```sh +mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup +``` + +5. Put the [Backblaze B2 settings seen above](#backblaze-b2) in your `vars.yml` file + +6. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +7. You're done! Verify that loading existing (old) media files works and that you can upload new ones. + +8. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup` diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index 7f05a5b2..1b5ea234 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -74,15 +74,12 @@ If you are hosting other domains on the Matrix machine, you can make the playboo To do that, simply define your own custom configuration like this: ```yaml -# Note: we need to explicitly list the aforementioned Matrix domains that you use (Matrix, Element, Dimension). -# In this example, we retrieve an extra certificate - one for the base domain (in the `matrix_domain` variable). +# In this example, we retrieve 2 extra certificates, +# one for the base domain (in the `matrix_domain` variable) and one for a hardcoded domain. # Adding any other additional domains (hosted on the same machine) is possible. -matrix_ssl_domains_to_obtain_certificates_for: - - '{{ matrix_server_fqn_matrix }}' - - '{{ matrix_server_fqn_element }}' - - '{{ matrix_server_fqn_dimension }}' - - '{{ matrix_server_fqn_jitsi }}' +matrix_ssl_additional_domains_to_obtain_certificates_for: - '{{ matrix_domain }}' + - 'another.domain.example.com' ``` After redefining `matrix_ssl_domains_to_obtain_certificates_for`, to actually obtain certificates you should: @@ -91,9 +88,9 @@ After redefining `matrix_ssl_domains_to_obtain_certificates_for`, to actually ob - re-run the SSL part of the playbook and restart all services: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-ssl,start` -The certificate files would be available in `/matrix/ssl/config/live//...`. +The certificate files would be made available in `/matrix/ssl/config/live//...`. For automated certificate renewal to work, each port `80` vhost for each domain you are obtaining certificates for needs to forward requests for `/.well-known/acme-challenge` to the certbot container we use for renewal. See how this is configured for the `matrix.` subdomain in `/matrix/nginx-proxy/conf.d/matrix-synapse.conf` -Don't be alarmed if the above configuraiton file says port `8080`, instead of port `80`. It's due to port mapping due to our use of containers. +Don't be alarmed if the above configuration file says port `8080`, instead of port `80`. It's due to port mapping due to our use of containers. diff --git a/docs/configuring-playbook-synapse.md b/docs/configuring-playbook-synapse.md index 9ae1e903..bdfdfa63 100644 --- a/docs/configuring-playbook-synapse.md +++ b/docs/configuring-playbook-synapse.md @@ -18,6 +18,35 @@ Alternatively, **if there is no pre-defined variable** for a Synapse setting you - or, if extending the configuration is still not powerful enough for your needs, you can **override the configuration completely** using `matrix_synapse_configuration` (or `matrix_synapse_configuration_yaml`). You can find information about this in [`roles/matrix-synapse/defaults/main.yml`](../roles/matrix-synapse/defaults/main.yml). +## Load balancing with workers + +To have Synapse gracefully handle thousands of users, worker support should be enabled. It factors out some homeserver tasks and spreads the load of incoming client and server-to-server traffic between multiple processes. More information can be found in the [official Synapse workers documentation](https://github.com/matrix-org/synapse/blob/master/docs/workers.md). + +To enable Synapse worker support, update your `inventory/host_vars/matrix.DOMAIN/vars.yml` file: + +```yaml +matrix_synapse_workers_enabled: true +``` + +We support a few configuration presets (`matrix_synapse_workers_preset: one-of-each` being the default configuration): +- `little-federation-helper` - a very minimal worker configuration to improve federation performance +- `one-of-each` - one worker of each supported type + +If you'd like more customization power, you can start with one of the presets and tweak various `matrix_synapse_workers_*_count` variables manually. + +If you increase worker counts too much, you may need to increase the maximum number of Postgres connections too (example): + +```yaml +matrix_postgres_process_extra_arguments: [ + "-c 'max_connections=200'" +] +``` + +If you're using the default setup (the `matrix-nginx-proxy` webserver being enabled) or you're using your own `nginx` server (which imports the configuration files generated by the playbook), you're good to go. If you use some other webserver, you may need to tweak your reverse-proxy setup manually to forward traffic to the various workers. + +In case any problems occur, make sure to have a look at the [list of synapse issues about workers](https://github.com/matrix-org/synapse/issues?q=workers+in%3Atitle) and your `journalctl --unit 'matrix-*'`. + + ## Synapse Admin Certain Synapse administration tasks (managing users and rooms, etc.) can be performed via a web user-interace, if you install [Synapse Admin](configuring-playbook-synapse-admin.md). diff --git a/docs/configuring-playbook.md b/docs/configuring-playbook.md index 3bb28c3f..34c52efc 100644 --- a/docs/configuring-playbook.md +++ b/docs/configuring-playbook.md @@ -10,7 +10,7 @@ You can then follow these steps inside the playbook directory: - create a directory to hold your configuration (`mkdir inventory/host_vars/matrix.`) -- copy the sample configuration file (`cp examples/host-vars.yml inventory/host_vars/matrix./vars.yml`) +- copy the sample configuration file (`cp examples/vars.yml inventory/host_vars/matrix./vars.yml`) - edit the configuration file (`inventory/host_vars/matrix./vars.yml`) to your liking. You may also take a look at the various `roles/ROLE_NAME_HERE/defaults/main.yml` files and see if there's something you'd like to copy over and override in your `vars.yml` configuration file. @@ -33,7 +33,9 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up the Jitsi video-conferencing platform](configuring-playbook-jitsi.md) (optional) -- [Setting Dynamic DNS](configuring-playbook-dynamic-dns.md) (optional) +- [Setting up Dynamic DNS](configuring-playbook-dynamic-dns.md) (optional) + +- [Enabling metrics and graphs (Prometheus, Grafana) for your Matrix server](configuring-playbook-prometheus-grafana.md) (optional) ### Core service adjustments @@ -94,6 +96,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up Mautrix Hangouts bridging](configuring-playbook-bridge-mautrix-hangouts.md) (optional) +- [Setting up Mautrix Instagram bridging](configuring-playbook-bridge-mautrix-instagram.md) (optional) + - [Setting up Mautrix Signal bridging](configuring-playbook-bridge-mautrix-signal.md) (optional) - [Setting up Appservice IRC bridging](configuring-playbook-bridge-appservice-irc.md) (optional) @@ -114,6 +118,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins - [Setting up MX Puppet Discord bridging](configuring-playbook-bridge-mx-puppet-discord.md) (optional) +- [Setting up MX Puppet GroupMe bridging](configuring-playbook-bridge-mx-puppet-groupme.md) (optional) + - [Setting up MX Puppet Steam bridging](configuring-playbook-bridge-mx-puppet-steam.md) (optional) - [Setting up Email2Matrix](configuring-playbook-email2matrix.md) (optional) diff --git a/docs/configuring-well-known.md b/docs/configuring-well-known.md index 5e910c3b..2bedaeed 100644 --- a/docs/configuring-well-known.md +++ b/docs/configuring-well-known.md @@ -148,6 +148,13 @@ backend matrix-backend rsprep ^Location:\ (http|https)://matrix.example.com\/(.*) Location:\ \1://matrix.example.com/.well-known/matrix/\2 if response-is-redirect ``` +**For Netlify**, it would be something like this: + +``` +# In the _redirects file in the website's root +/.well-known/matrix/* https://matrix.DOMAIN/.well-known/matrix/:splat 200! +``` + Make sure to: - **replace `DOMAIN`** in the server configuration with your actual domain name diff --git a/docs/container-images.md b/docs/container-images.md new file mode 100644 index 00000000..a5e304f4 --- /dev/null +++ b/docs/container-images.md @@ -0,0 +1,97 @@ +# Container Images used by the playbook + +This page summarizes the container ([Docker](https://www.docker.com/)) images used by the playbook when setting up your server. + +We try to stick to official images (provided by their respective projects) as much as possible. + + +## Container images used by default + +These services are enabled and used by default, but you can turn them off, if you wish. + +- [matrixdotorg/synapse](https://hub.docker.com/r/matrixdotorg/synapse/) - the official [Synapse](https://github.com/matrix-org/synapse) Matrix homeserver (optional) + +- [instrumentisto/coturn](https://hub.docker.com/r/instrumentisto/coturn/) - the [Coturn](https://github.com/coturn/coturn) STUN/TURN server (optional) + +- [vectorim/element-web](https://hub.docker.com/r/vectorim/element-web/) - the [Element](https://element.io/) web client (optional) + +- [ma1uta/ma1sd](https://hub.docker.com/r/ma1uta/ma1sd/) - the [ma1sd](https://github.com/ma1uta/ma1sd) Matrix Identity server (optional) + +- [postgres](https://hub.docker.com/_/postgres/) - the [Postgres](https://www.postgresql.org/) database server (optional) + +- [devture/exim-relay](https://hub.docker.com/r/devture/exim-relay/) - the [Exim](https://www.exim.org/) email server (optional) + +- [nginx](https://hub.docker.com/_/nginx/) - the [nginx](http://nginx.org/) web server (optional) + +- [certbot/certbot](https://hub.docker.com/r/certbot/certbot/) - the [certbot](https://certbot.eff.org/) tool for obtaining SSL certificates from [Let's Encrypt](https://letsencrypt.org/) (optional) + + +## Optional other container images we may use + +These services are not part of our default installation, but can be enabled by [configuring the playbook](configuring-playbook.md) (either before the initial installation or any time later): + +- [ewoutp/goofys](https://hub.docker.com/r/ewoutp/goofys/) - the [Goofys](https://github.com/kahing/goofys) Amazon [S3](https://aws.amazon.com/s3/) file-system-mounting program (optional) + +- [etherpad/etherpad](https://hub.docker.com/r/etherpad/etherpad/) - the [Etherpad](https://etherpad.org) realtime collaborative text editor that can be used in a Jitsi audio/video call or integrated as a widget into Matrix chat rooms via the Dimension integration manager (optional) + +- [devture/email2matrix](https://hub.docker.com/r/devture/email2matrix/) - the [Email2Matrix](https://github.com/devture/email2matrix) email server, which can relay email messages to Matrix rooms (optional) + +- [devture/matrix-corporal](https://hub.docker.com/r/devture/matrix-corporal/) - [Matrix Corporal](https://github.com/devture/matrix-corporal): reconciliator and gateway for a managed Matrix server (optional) + +- [zeratax/matrix-registration](https://hub.docker.com/r/devture/zeratax-matrix-registration/) - [matrix-registration](https://github.com/ZerataX/matrix-registration): a simple python application to have a token based matrix registration (optional) + +- [tulir/mautrix-telegram](https://mau.dev/tulir/mautrix-telegram/container_registry) - the [mautrix-telegram](https://github.com/tulir/mautrix-telegram) bridge to [Telegram](https://telegram.org/) (optional) + +- [tulir/mautrix-whatsapp](https://mau.dev/tulir/mautrix-whatsapp/container_registry) - the [mautrix-whatsapp](https://github.com/tulir/mautrix-whatsapp) bridge to [Whatsapp](https://www.whatsapp.com/) (optional) + +- [tulir/mautrix-facebook](https://mau.dev/tulir/mautrix-facebook/container_registry) - the [mautrix-facebook](https://github.com/tulir/mautrix-facebook) bridge to [Facebook](https://facebook.com/) (optional) + +- [tulir/mautrix-hangouts](https://mau.dev/tulir/mautrix-hangouts/container_registry) - the [mautrix-hangouts](https://github.com/tulir/mautrix-hangouts) bridge to [Google Hangouts](https://en.wikipedia.org/wiki/Google_Hangouts) (optional) + +- [tulir/mautrix-instagram](https://mau.dev/tulir/mautrix-instagram/container_registry) - the [mautrix-instagram](https://github.com/tulir/mautrix-instagram) bridge to [Instagram](https://instagram.com/) (optional) + +- [tulir/mautrix-signal](https://mau.dev/tulir/mautrix-signal/container_registry) - the [mautrix-signal](https://github.com/tulir/mautrix-signal) bridge to [Signal](https://www.signal.org/) (optional) + +- [matrixdotorg/matrix-appservice-irc](https://hub.docker.com/r/matrixdotorg/matrix-appservice-irc) - the [matrix-appservice-irc](https://github.com/matrix-org/matrix-appservice-irc) bridge to [IRC](https://wikipedia.org/wiki/Internet_Relay_Chat) (optional) + +- [halfshot/matrix-appservice-discord](https://hub.docker.com/r/halfshot/matrix-appservice-discord) - the [matrix-appservice-discord](https://github.com/Half-Shot/matrix-appservice-discord) bridge to [Discord](https://discordapp.com/) (optional) + +- [cadair/matrix-appservice-slack](https://hub.docker.com/r/cadair/matrix-appservice-slack) - the [matrix-appservice-slack](https://github.com/matrix-org/matrix-appservice-slack) bridge to [Slack](https://slack.com/) (optional) + +- [turt2live/matrix-appservice-webhooks](https://hub.docker.com/r/turt2live/matrix-appservice-webhooks) - the [Appservice Webhooks](https://github.com/turt2live/matrix-appservice-webhooks) bridge (optional) + +- [folivonet/matrix-sms-bridge](https://hub.docker.com/repository/docker/folivonet/matrix-sms-bridge) - the [matrix-sms-bridge](https://github.com/benkuly/matrix-sms-bridge) (optional) + +- [sorunome/mx-puppet-skype](https://hub.docker.com/r/sorunome/mx-puppet-skype) - the [mx-puppet-skype](https://github.com/Sorunome/mx-puppet-skype) bridge to [Skype](https://www.skype.com) (optional) + +- [sorunome/mx-puppet-slack](https://hub.docker.com/r/sorunome/mx-puppet-slack) - the [mx-puppet-slack](https://github.com/Sorunome/mx-puppet-slack) bridge to [Slack](https://slack.com) (optional) + +- [sorunome/mx-puppet-instagram](https://hub.docker.com/r/sorunome/mx-puppet-instagram) - the [mx-puppet-instagram](https://github.com/Sorunome/mx-puppet-instagram) bridge to [Instagram](https://www.instagram.com) (optional) + +- [sorunome/mx-puppet-twitter](https://hub.docker.com/r/sorunome/mx-puppet-twitter) - the [mx-puppet-twitter](https://github.com/Sorunome/mx-puppet-twitter) bridge to [Twitter](https://twitter.com) (optional) + +- [sorunome/mx-puppet-discord](https://hub.docker.com/r/sorunome/mx-puppet-discord) - the [mx-puppet-discord](https://github.com/matrix-discord/mx-puppet-discord) bridge to [Discord](https://discordapp.com) (optional) + +- [xangelix/mx-puppet-groupme](https://hub.docker.com/r/xangelix/mx-puppet-groupme) - the [mx-puppet-groupme](https://gitlab.com/robintown/mx-puppet-groupme) bridge to [GroupMe](https://groupme.com/) (optional) + +- [icewind1991/mx-puppet-steam](https://hub.docker.com/r/icewind1991/mx-puppet-steam) - the [mx-puppet-steam](https://github.com/icewind1991/mx-puppet-steam) bridge to [Steam](https://steampowered.com) (optional) + +- [turt2live/matrix-dimension](https://hub.docker.com/r/turt2live/matrix-dimension) - the [Dimension](https://dimension.t2bot.io/) integrations manager (optional) + +- [jitsi/web](https://hub.docker.com/r/jitsi/web) - the [Jitsi](https://jitsi.org/) web UI (optional) + +- [jitsi/jicofo](https://hub.docker.com/r/jitsi/jicofo) - the [Jitsi](https://jitsi.org/) Focus component (optional) + +- [jitsi/prosody](https://hub.docker.com/r/jitsi/prosody) - the [Jitsi](https://jitsi.org/) Prosody XMPP server component (optional) + +- [jitsi/jvb](https://hub.docker.com/r/jitsi/jvb) - the [Jitsi](https://jitsi.org/) Video Bridge component (optional) + +- [anoa/matrix-reminder-bot](https://hub.docker.com/r/anoa/matrix-reminder-bot) - the [matrix-reminder-bot](https://github.com/anoadragon453/matrix-reminder-bot) bot for one-off & recurring reminders and alarms (optional) + +- [awesometechnologies/synapse-admin](https://hub.docker.com/r/awesometechnologies/synapse-admin) - the [synapse-admin](https://github.com/Awesome-Technologies/synapse-admin) web UI tool for administrating users and rooms on your Matrix server (optional) + +- [prom/prometheus](https://hub.docker.com/r/prom/prometheus/) - [Prometheus](https://github.com/prometheus/prometheus/) is a systems and service monitoring system + +- [prom/node-exporter](https://hub.docker.com/r/prom/node-exporter/) - [Prometheus Node Exporter](https://github.com/prometheus/node_exporter/) is an addon for Prometheus that gathers standard system metrics + +- [grafana/grafana](https://hub.docker.com/r/grafana/grafana/) - [Grafana](https://github.com/grafana/grafana/) is a graphing tool that works well with the above two images. Our playbook also adds two dashboards for [Synapse](https://github.com/matrix-org/synapse/tree/master/contrib/grafana) and [Node Exporter](https://github.com/rfrail3/grafana-dashboards) diff --git a/docs/faq.md b/docs/faq.md new file mode 100644 index 00000000..a6782231 --- /dev/null +++ b/docs/faq.md @@ -0,0 +1,460 @@ +# Frequently Asked Questions + +This documentation page tries to answer various Frequently Asked Questions about all things [Matrix](https://matrix.org/), with a focus on this [Ansible](https://www.ansible.com/) playbook ([What is Ansible? How does it work?](#what-is-ansible-how-does-it-work)). + +This FAQ page does not intend to replace the [matrix.org FAQ](https://matrix.org/faq/) (please see that one too). + +We've only started this FAQ recently, so it's still somewhat empty. + +Also, we encourage you to not dig yourself into a hole by reading way too much. When you've heard enough, proceed to [Prerequisites](prerequisites.md) to get guided into installing Matrix. + + +## Introductory + +## Where do I find more questions and answers about Matrix? + +This is a Frequently Asked Questions page focused on this [Ansible](https://www.ansible.com/) playbook ([What is Ansible? How does it work?](#what-is-ansible-how-does-it-work)) for deploying a [Matrix](https://matrix.org/) server. + +For a lot more generic questions and answers, see the [matrix.org FAQ](https://matrix.org/faq/). + +## What is Matrix? What is Element? What is Synapse? Why are you confusing me with so many terms? + +[Matrix](https://matrix.org/) is a new type of realtime communication (chat) network, the closest analogy to which is probably "email". + +You don't just use the "email" protocols (SMTP, POP3, IMAP) directly though. There's a *server* somewhere which stores your data (`@gmail.com`, `@yahoo.com`, `@hotmail.com`, `@your-company.com`) and you access it by using these "email" protocols via some *client* program (Outlook, Thunderbird, some website, etc). + +In the world of the Matrix chat protocol, there are various client programs. The first and currently most full-featured one is called [Element](https://element.io/) (used to be called Riot.im and Vector.im in the past). There are [many other clients](https://matrix.org/clients/). You can switch clients as much as you want until you find the one that is right for you on a given platform (you may use Element on your desktop, but Fluffychat on your phone, etc). + +Matrix is also like email due to the fact that there are many servers around the world which can all talk to each other (you can send email from `@gmail.com` addresses to `@yahoo.com` and `@hotmail.com` addresses). It's the same with Matrix (`@bob:his-domain.com` can talk to `@alice:her-domain.org`). + +If someone else is hosting your Matrix server (you being `@user:matrix.org` or some other public server like this), all you need is a Matrix client program, like Element. + +If you'd like to host your own server (you being `@user:your-own-domain.com`), you'd need to set up a Matrix server program, like Synapse. + +In short: + +- Matrix is the protocol - a set of rules about how the chat network operates +- Element is a client program you can use to participate on the Matrix chat network via some server (yours or someone else's). There are also [many other client programs](https://matrix.org/clients/). +- Synapse is a server program you can use to host your very own Matrix server. + +This FAQ here mostly focuses on installing various Matrix services using the Ansible automation tool. You can learn much more about Matrix in the [matrix.org FAQ](https://matrix.org/faq/). + +## People I wish to talk to are not on Matrix. Can I talk to them? + +You most likely can. Besides Matrix-native chats, Matrix also supports the concept of "bridging", which allows you to plug other networks into it. + +This Ansible playbook can help you install [tens of bridges for various networks](configuring-playbook.md#bridging-other-networks). + +Besides setting up your own bridges (preferable), you can also use some [public bridges hosted by others](https://publiclist.anchel.nl/#bridges). + +## How do I get started with Matrix? + +One of [Matrix](https://matrix.org/)'s distinguishing strengths (compared to other chat networks) is its decentralized nature. There's not just one entity (company, organization) controlling the servers. Rather there's thousands of servers operated by different people - one server being insecure, slow or disrespective toward its users does not affect the rest of the network. To participate in that decentralization in its fullest, consider hosting your own server or using some public server other than the largest/default one (`matrix.org`). + +There are 3 ways to get into Martix, depending on your technical ability and needs: + +- **using the existing default server** - the easiest way is to use an existing server. The largest public Matrix server is `matrix.org` and it's configured as a default server in clients such as [Element](https://element.io) and many others. Just use Element on the browser via that link (or download the Element app on a smartphone), create an account and start chatting. + +- **using some other server** - instead of using the largest public server (`matrix.org`), you can use another public one. Here's a [list of public Matrix servers](https://publiclist.anchel.nl/) to choose from. Again, you download [Element](https://element.io) or [some other client](https://matrix.org/clients/) of your choosing and adjust the homeserver URL during login. + +- **using your own server** - running your own server puts you in ultimate control of your data. It also lets you have your own user identifiers (e.g. `@bob:your-domain.com`). See [How do I set up my own Matrix server](#how-do-i-set-up-my-own-matrix-server). + +### How do I set up my own Matrix server? + +Normally, you'd first choose the [Matrix](https://matrix.org/) server software you'd like to run. At the time of this writing (January/2021), there's only one fully-featured server program, so there's only one reasonable choice. That's [Synapse](https://github.com/matrix-org/synapse). + +There are [many guides about installing Synapse](https://matrix.org/docs/guides/#installing-synapse). Using this Ansible playbook is just one way of doing it. + +Naturally, we're biased, so our usual recommendation is to go with this [Ansible](https://www.ansible.com/) playbook, instead of installing Synapse (and many many other things around it) manually. +To get started with the playbook, start at the [Prerequisites](prerequisites.md) page. + +### What is Ansible? How does it work? + +[Ansible](https://www.ansible.com/) is an automation program. This "playbook" is a collection of tasks/scripts that will set up a [Matrix](https://matrix.org/) server for you, so you don't have to perform these tasks manually. + +We have written these automated tasks for you and all you need to do is execute them using the Ansible program. + +You can install Ansible and this playbook code repository on your own computer and tell it to install Matrix services at the server living at `matrix.DOMAIN`. We recommend installing Ansible on your own computer. + +Alternatively, you can download Ansible and the playbook itself directly on the `matrix.DOMAIN` server. + +To learn more, see our [dedicated Ansible documentation page](ansible.md). + +### Why use this playbook and not install Synapse and other things manually? + +There are various guides telling you how easy it is to install [Synapse](https://github.com/matrix-org/synapse). + +Reading the documentation of this Ansible playbook, you may also be thinking: + +> I don't know what [Ansible](https://www.ansible.com/) is. I don't know what [Docker](https://www.docker.com/) is. This looks more complicated. + +.. so you may be leaning toward [installing Synapse manually](https://github.com/matrix-org/synapse/blob/master/INSTALL.md). + +The problem with a manual installation is: + +- Synapse is written in Python. If not packaged for your distribution, you'd need to install various Python modules, etc., and keep them updated. +- Synapse requires a [Postgres](https://www.postgresql.org/) database (it can run on SQLite, but that's very much discouraged). So you'd need to install Postgres as well. +- you may also need a reverse-proxy server in front of it (nginx, Apache), so you'd need to be familiar with that +- SSL is required, so you'd need to obtain Let's Encrypt (or other free or non-free) certificates for one or more domain names. You'd need to be familiar with [certbot](https://certbot.eff.org/) (when using Let's Encrypt) or similar software. +- for each additional component you'd like to add (client like [Element](https://element.io), bridge to some other chat network, Integration Manager (sitckers, other services), Identity Manager, etc.), you'll need to spend extra time installing and wiring it with the rest of the system in a way that works. +- you'll likely get slower updates for all of these components, depending on your distro packaging or your own time and ability + +The playbook, on the other hand, installs a bunch of components for you by default, obtains SSL certificates for you, etc. If you'd like, you can enable various bridges and other services with very little effort. All the components are wired to work together. + +All services run in Docker containers (most being officially provided by each component's developers), so we're not at the mercy of distro packaging. + +### Why use this playbook and not just use the Docker image directly? + +Reasons are similar to the reasons for not installing manually. + +Besides Synapse, you'd need other things - a Postgres database, likely the [Element](https://element.io) client, etc., etc. + +Using the playbook, you get all these components in a way that works well together out of the box. + +### What's different about this Ansible playbook compared to [EMnify/matrix-synapse-auto-deploy](https://github.com/EMnify/matrix-synapse-auto-deploy)? + +This is similar to the [EMnify/matrix-synapse-auto-deploy](https://github.com/EMnify/matrix-synapse-auto-deploy) Ansible deployment, but: + +- this one is a complete Ansible playbook (instead of just a role), so it's **easier to run** - especially for folks not familiar with Ansible + +- this one installs and hooks together **a lot more Matrix-related services** for you (see above) + +- this one **can be executed more than once** without causing trouble + +- works on various distros: **CentOS** (7.0+), Debian-based distributions (**Debian** 9/Stretch+, **Ubuntu** 16.04+), **Archlinux** + +- this one installs everything in a single directory (`/matrix` by default) and **doesn't "contaminate" your server** with files all over the place + +- this one **doesn't necessarily take over** ports 80 and 443. By default, it sets up nginx for you there, but you can also [use your own webserver](configuring-playbook-own-webserver.md) + +- this one **runs everything in Docker containers**, so it's likely more predictable and less fragile (see [Docker images used by this playbook](container-images.md)) + +- this one retrieves and automatically renews free [Let's Encrypt](https://letsencrypt.org/) **SSL certificates** for you + +- this one optionally can store the `media_store` content repository files on [Amazon S3](https://aws.amazon.com/s3/) (but defaults to storing files on the server's filesystem) + +- this one optionally **allows you to use an external PostgreSQL server** for Synapse's database (but defaults to running one in a container) + +- helps you **import data from a previous installation** (so you can migrate your manual virtualenv/Docker setup to a more managed one) + +- this one is actually **maintained** + +## Server-related + +### What kind of server do I need to install Matrix using this Ansible playbook? + +We list our server requirements in [Prerequisites](prerequisites.md). + +### Why not run Matrix on Kubernetes? + +There's no reason not to run Matrix on [Kubernetes](https://kubernetes.io/). + +However, that's overly complicated for thousands of us who just want to run a single small (and sometimes not so small) Matrix server, either using "cloud" servers or even a [Raspberry Pi](https://www.raspberrypi.org/) at home. + +For us, a Kubernetes-based setup which requires a cluster of multiple computers and is more technically-involved is a no-go. + +There are others working on automating a Matrix-on-Kubernetes setup, such as this [Helm](https://helm.sh/) chart: https://github.com/dacruz21/matrix-chart. + +### Why don't you use Podman instead of Docker? + +We like the philosophy of a daemonless container runtime, but [Podman](https://podman.io) is just not ready for our use case yet. + +Learn more about our past experiences/attempts to give Podman a chance, by reading [this issue](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/520). + +In short, `alias podman=docker` is a lie (for us). + +### Why use Docker? + +[Docker](https://www.docker.com/) is one of our 2 hard dependencies (the other one being [systemd](https://systemd.io/)). + +It lets us run services in an isolated manner and independently of the (usually old) packages available for distributions. + +It also lets us have a unified setup which runs the same across various supported distros (see them on [Prerequisites](prerequisites.md)). + +### Is Docker a hard requirement? + +Yes. See [Why don't you use Podman instead of Docker?](#why-dont-you-use-podman-instead-of-docker) for why we're not using another container runtime. + +All of our services run in containers. It's how we achieve predictability and also how we support tens of different services across lots of distros. + +The only thing we need on the distro is systemd and Python (we install Docker ourselves, unless you ask us not to). + +### Why don't you use docker-compose? + +Instead of using [docker-compose](https://docs.docker.com/compose/), we prefer installing systemd services and scheduling those independently. + +There are people who have worked on turning this setup into a docker-compose-based one. See these experiments [here](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/64#issuecomment-603164625). + +### Can I run this on a distro without systemd? + +No. [systemd](https://systemd.io/) is one of our 2 hard dependencies (the other one being [Docker](https://www.docker.com/)). + +### Can I install this on a Raspberry Pi? + +Yes, you can. See our [Alternative Architectures](alternative-architectures.md) documentation page. + +Whether a Raspberry Pi has enough power to give you a good experience is another question. It depends on your use case. + +Also see: [What kind of server specs do I need?](#what-kind-of-server-specs-do-i-need). + +### What kind of server specs do I need? + +This largely depends on your use case. It's not so much the number of users that you plan to host, but rather the number of large rooms they will join. + +Federated rooms with lots of history and containing hundreds of other servers are very heavy CPU-wise and memory-wise. + +You can probably use a 1 CPU + 1GB memory server to host hundreds of local users just fine, but as soon as one of them joins a federated room like `#matrix:matrix.org` (Matrix HQ) or some IRC-bridged room (say `##linux`), your server will get the need for a lot more power (at least 2GB RAM, etc). + +Running Matrix on a server with 1GB of memory is possible (especially if you disable some not-so-important services). See [How do I optimize this setup for a low-power server?](#how-do-i-optimize-this-setup-for-a-low-power-server). + +**We recommend starting with a server having at least 2GB of memory** and even then using it sparingly. If you know for sure you'll be joining various large rooms, etc., then going for 4GB of memory or more is a good idea. + +Besides the regular Matrix stuff, we also support things like video-conferencing using [Jitsi](configuring-playbook-jitsi.md) and other additional services which (when installed) may use up a lot of memory. Things do add up. Besides the Synapse Matrix server, Jitsi is especially notorious for consuming a lot of resources. If you plan on running Jitsi, we recommend a server with at least 2GB of memory (preferrably more). See our [Jitsi documentation page](configuring-playbook-jitsi.md) to learn how to optimize its memory/CPU usage. + +### Can I run this in an LXC container? + +If your distro runs within an [LXC container](https://linuxcontainers.org/), you may hit [this issue](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/703). It can be worked around, if absolutely necessary, but we suggest that you avoid running from within an LXC container. + + +## Configuration + +### Why install my server at matrix.DOMAIN and not at the base DOMAIN? + +It's the same with email servers. Your email address is likely `name@company.com`, not `name@mail.company.com`, even though it's `mail.company.com` that is really handling your data for `@company.com` email to work. + +Using a separate domain name is easier to manage (although it's a little hard to get right at first) and keeps your Matrix server isolated from your website (if you have one), from your email server (if you have one), etc. + +We allow `matrix.DOMAIN` to be the Matrix server handling Matrix stuff for `DOMAIN` by [Server Delegation](howto-server-delegation.md). During the installation procedure, we recommend that you set up server delegation using the [.well-known](configuring-well-known.md) method. + +If you'd really like to install Matrix services directly on the base domain, see [How do I install on matrix.DOMAIN without involving the base DOMAIN?](#how-do-i-install-on-matrixdomain-without-involving-the-base-domain). + +### I don't control anything on the base domain and can't set up delegation to matrix.DOMAIN. What do I do? + +If you're not in control of your base domain (or the server handling it) at all, you can take a look at [How do I install on matrix.DOMAIN without involving the base DOMAIN?](#how-do-i-install-on-matrixdomain-without-involving-the-base-domain) + +### I can't set up HTTPS on the base domain. How will I get Matrix federating? + +If you really can't obtain an HTTPS certificate for your base domain, you can take a look at [How do I install on matrix.DOMAIN without involving the base DOMAIN?](#how-do-i-install-on-matrixdomain-without-involving-the-base-domain) + +### How do I install on matrix.DOMAIN without involving the base DOMAIN? + +This Ansible playbook guides you into installing a server for `DOMAIN` (user identifiers are like this: `@user:DOMAIN`), while the server is at `matrix.DOMAIN`. + +We allow `matrix.DOMAIN` to be the Matrix server handling Matrix stuff for `DOMAIN` by [Server Delegation](howto-server-delegation.md). During the installation procedure, we recommend that you set up server delegation using the [.well-known](configuring-well-known.md) method. + +If you're fine with uglier identifiers (`@user:matrix.DOMAIN`, which is the equivalent of having an email address like `bob@mail.company.com`, instead of just `bob@company.com`), you can do that as well using the following configuration in your `vars.yml` file: + +```yaml +# This is what your identifiers are like (e.g. `@bob:matrix.YOUR_BASE_DOMAIN`). +matrix_domain: "matrix.YOUR_BASE_DOMAIN" + +# This is where Matrix services +matrix_server_fqn_matrix: "matrix.YOUR_BASE_DOMAIN" + +# This is where you access the Element web UI from (if enabled via `matrix_client_element_enabled: true`; enabled by default). +# This and the Matrix FQN (see above) are expected to be on the same server. +# +# Feel free to use `element.matrix.YOUR_BASE_DOMAIN`, if you'd prefer that. +matrix_server_fqn_element: "element.YOUR_BASE_DOMAIN" + +# This is where you access Dimension (if enabled via `matrix_dimension_enabled: true`; NOT enabled by default). +# +# Feel free to use `dimension.matrix.YOUR_BASE_DOMAIN`, if you'd prefer that. +matrix_server_fqn_dimension: "dimension.YOUR_BASE_DOMAIN" + +# This is where you access Jitsi (if enabled via `matrix_jitsi_enabled: true`; NOT enabled by default). +# +# Feel free to use `jitsi.matrix.YOUR_BASE_DOMAIN`, if you'd prefer that. +matrix_server_fqn_jitsi: "jitsi.YOUR_BASE_DOMAIN" +``` + +### I don't use the base domain for anything. How am I supposed to set up Server Delegation for Matrix services? + +If you don't use your base domain for anything, then it's hard for you to "serve files over HTTPS" on it -- something we ask you to do for the [.well-known](configuring-well-known.md) setup (needed for [Server Delegation](howto-server-delegation.md)). + +Luckily, the playbook can set up your Matrix server (at `matrix.DOMAIN`) to also handle traffic for the base domain (`DOMAIN`). + +See [Serving the base domain](configuring-playbook-base-domain-serving.md). + +### How do I optimize this setup for a low-power server? + +You can disable some not-so-important services to save on memory. + +```yaml +# An identity server is not a must. +matrix_ma1sd_enabled: false + +# Disabling this will prevent email-notifications and other such things from working. +matrix_mailer_enabled: false + +# You can also disable this to save more RAM, +# at the expense of audio/video calls being unreliable. +matrix_coturn_enabled: true + +# This makes Synapse not keep track of who is online/offline. +# +# Keeping track of this and announcing such online-status in federated rooms with +# hundreds of servers inside is insanely heavy (https://github.com/matrix-org/synapse/issues/3971). +# +# If your server does not federate with hundreds of others, enabling this doesn't hurt much. +matrix_synapse_use_presence: false +``` + +You can also consider implementing a restriction on room complexity, in order to prevent users from joining very heavy rooms: + +```yaml +matrix_synapse_configuration_extension_yaml: | + limit_remote_rooms: + enabled: true + complexity: 1.0 # this limits joining complex (~large) rooms, can be + # increased, but larger values can require more RAM +``` + +If you've installed [Jitsi](configuring-playbook-jitsi.md) (not installed by default), there are additional optimizations listed on its documentation page that you can perform. + +### I already have Docker on my server. Can you stop installing Docker via the playbook? + +Yes, we can stop installing Docker ourselves. Just use this in your `vars.yml` file: + +```yaml +matrix_docker_installation_enabled: true +``` + +### I run another webserver on the same server where I wish to install Matrix. What now? + +By default, we install a webserver for you (nginx), but you can also use [your own webserver](configuring-playbook-own-webserver.md). + +### How is the effective configuration determined? + +Configuration variables are defined in multiple places in this playbook and are considered in this order: + +- there are defaults coming from each role's defaults file (`role/matrix*/defaults/main.yml`). These variable values aim to be good defaults for when the role is used standalone (outside of this collection of roles, also called playbook). + +- then, there are overrides in `group_vars/matrix_servers`, which aim to adjust these "standalone role defaults" to something which better fits the playbook in its entirety. + +- finally, there's your `inventory/host_vars/matrix.DOMAIN/vars.yml` file, which is the ultimate override + +### What configuration variables are available? + +You can discover the variables you can override in each role (`role/matrix*/defaults/main.yml`). + +As described in [How is the effective configuration determined?](#how-is-the-effective-configuration-determined), these role-defaults may be overriden by values defined in `group_vars/matrix_servers`. + +Refer to both of these for inspiration. Still, as mentioned in [Configuring the playbook](configuring-playbook.md), you're only ever supposed to edit your own `inventory/host_vars/matrix.DOMAIN/vars.yml` file and nothing else inside the playbook (unless you're meaning to contribute new features). + +### I'd like to adjust some configuration which doesn't have a corresponding variable. How do I do it? + +The playbook doesn't aim to expose all configuration settings for all services using variables. +Doing so would amount to hundreds of variables that we have to create and maintain. + +Instead, we only try to make some important basics configurable using dedicated variables you can see in each role. +See [What configuration variables are available?](#what-configuration-variables-are-available). + +Besides that, each role (component) aims to provide a `matrix_SOME_COMPONENT_configuration_extension_yaml` (or `matrix_SOME_COMPONENT_configuration_extension_json`) variable, which can be used to override the configuration. + +Check each role's `role/matrix*/defaults/main.yml` for the corresponding variable and an example for how use it. + + +## Installation + +### How do I run the installation? + +See [Installing](installing.md) to learn how to use Ansible to install Matrix services. + +Of course, don't just jump straight to Installing. Rather, start at [Prerequisites](prerequisites.md) and get guided from there (into [setting up DNS](configuring-dns.md), [configuring the playbook](configuring-playbook.md), etc). + +### I installed Synapse some other way. Can I migrate such a setup to the playbook? + +Yes, you can. + +You generally need to do a playbook installation (start at the [Prerequisites](prerequisites.md) page), followed by importing your existing data into it. + +This Ansible playbook guides you into installing a server for `DOMAIN` (user identifiers are like this: `@user:DOMAIN`), while the server is at `matrix.DOMAIN`. If your existing setup has a server name (`server_name` configuration setting in Synapse's `homeserver.yaml` file) other than the base `DOMAIN`, you may need to tweak some additional variables. This FAQ entry may be of use if you're dealing with a more complicated setup - [How do I install on matrix.DOMAIN without involving the base DOMAIN?](#how-do-i-install-on-matrixdomain-without-involving-the-base-domain) + +After configuring the playbook and installing and **before starting** services (done with `ansible-playbook ... --tags=start`) you'd import [your SQLite](importing-synapse-sqlite.md) (or [Postgres](importing-postgres.md)) database and also [import your media store](importing-synapse-media-store.md). + +### I've downloaded Ansible and the playbook on the server. It can't connect using SSH. + +If you're using the playbook directly on the server, then Ansible doesn't need to connect using SSH. + +It can perform a local connection instead. Just set `ansible_connection=local` at the end of the server line in `inventory/hosts` and re-run the playbook. + +If you're running Ansible from within a container (one of the possibilities we list on our [dedicated Ansible documentation page](ansible.md)), then using `ansible_connection=local` is not possible. + + +## Troubleshooting + +### I get "Error response from daemon: configured logging driver does not support reading" when I do `docker logs matrix-synapse`. + +See [How can I see the logs?](#how-can-i-see-the-logs). + +### How can I see the logs? + +We utilize [systemd/journald](https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html#Description) for logging. + +To see logs for Synapse, run `journalctl -fu matrix-synapse.service`. You may wish to see the [manual page for journalctl](https://www.commandlinux.com/man-page/man1/journalctl.1.html). + +Available service names can be seen by doing `ls /etc/systemd/system/matrix*.service` on the server. + +Some services also log to files in `/matrix/*/data/..`, but we're slowly moving away from that. + +We also disable Docker logging, so you can't use `docker logs matrix-*` either. We do this to prevent useless double (or even triple) logging and to avoid having to rotate log files. + +We just simply delegate logging to journald and it takes care of persistence and expiring old data. + +Also see: [How long do systemd/journald logs persist for?](#how-long-do-systemdjournald-logs-persist-for) + +### How long do systemd/journald logs persist for? + +On some distros, the journald logs are just in-memory and not persisted to disk. + +Consult (and feel free to adjust) your distro's journald logging configuration in `/etc/systemd/journald.conf`. + +To enable persistence and put some limits on how large the journal log files can become, adjust your configuration like this: + +```ini +[Journal] +RuntimeMaxUse=200M +SystemMaxUse=1G +RateLimitInterval=0 +RateLimitBurst=0 +Storage=persistent +``` + + +## Maintenance + +### Do I need to do anything to keep my Matrix server updated? + +Yes. We don't update anything for you automatically. + +See our [documentation page about upgrading services](maintenance-upgrading-services.md). + +### How do I move my existing installation to another (VM) server? + +If you have an existing installation done using this Ansible playbook, you can easily migrate that to another server using [our dedicated server migration guide](maintenance-migrating.md). + +If your previous installation is done in some other way (not using this Ansible playbook), see [I installed Synapse some other way. Can I migrate such a setup to the playbook?](#i-installed-synapse-some-other-way-can-i-migrate-such-a-setup-to-the-playbook). + +### How do I back up the data on my server? + +We haven't documented this properly yet, but the general advice is to: + +- back up Postgres by making a database dump. See [Backing up PostgreSQL](maintenance-postgres.md#backing-up-postgresql) + +- back up all `/matrix` files, except for `/matrix/postgres/data` (you already have a dump) and `/matrix/postgres/data-auto-upgrade-backup` (this directory may exist and contain your old data if you've [performed a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql)). + +You can later restore these roughly like this: + +- restore the `/matrix` directory and files on the new server manually +- run the playbook again (see [Installing](installing.md)), but **don't** start services yet (**don't run** `... --tags=start`). This step will fix any file permission mismatches and will also set up additional software (Docker, etc.) and files on the server (systemd service, etc.). +- perform a Postgres database import (see [Importing Postgres](importing-postgres.md)) to restore your database backup +- start services (see [Starting the services](installing.md#starting-the-services)) + +If your server's IP address has changed, you may need to [set up DNS](configuring-dns.md) again. + +### What is this `/matrix/postgres/data-auto-upgrade-backup` directory that is taking up so much space? + +When you [perform a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql), we save the the old data files in `/matrix/postgres/data-auto-upgrade-backup`, just so you could easily restore them should something have gone wrong. + +After verifying that everything still works after the Postgres upgrade, you can safely delete `/matrix/postgres/data-auto-upgrade-backup` diff --git a/docs/importing-postgres.md b/docs/importing-postgres.md index 0dd75cb2..b905ba7b 100644 --- a/docs/importing-postgres.md +++ b/docs/importing-postgres.md @@ -7,8 +7,8 @@ Run this if you'd like to import your database from a previous installation. ## Prerequisites For this to work, **the database name in Postgres must match** what this playbook uses. -This playbook uses a Postgres database name of `homeserver` by default (controlled by the `matrix_postgres_db_name` variable). -If your database name differs, be sure to change `matrix_postgres_db_name` to your desired name and to re-run the playbook before proceeding. +This playbook uses a Postgres database name of `synapse` by default (controlled by the `matrix_synapse_database_database` variable). +If your database name differs, be sure to change `matrix_synapse_database_database` to your desired name and to re-run the playbook before proceeding. The playbook supports importing Postgres dump files in **text** (e.g. `pg_dump > dump.sql`) or **gzipped** formats (e.g. `pg_dump | gzip -c > dump.sql.gz`). @@ -21,10 +21,17 @@ Before doing the actual import, **you need to upload your Postgres dump file to To import, run this command (make sure to replace `` with a file path on your server): - ansible-playbook -i inventory/hosts setup.yml --extra-vars='server_path_postgres_dump=' --tags=import-postgres +```sh +ansible-playbook -i inventory/hosts setup.yml \ +--extra-vars='postgres_default_import_database=synapse server_path_postgres_dump=' \ +--tags=import-postgres +``` + +We specify the `synapse` database as the default import database. If your dump is a single-database dump (`pg_dump`), then we need to tell it where to go to. If you're redefining `matrix_synapse_database_database` to something other than `synapse`, please adjust it here too. For database dumps spanning multiple databases (`pg_dumpall`), you can remove the `postgres_default_import_database` definition (but it doesn't hurt to keep it too). **Note**: `` must be a file path to a Postgres dump file on the server (not on your local machine!). + ## Troubleshooting A table ownership issue can occur if you are importing from a Synapse installation which was both: diff --git a/docs/installing.md b/docs/installing.md index a2ce1371..0e9dadd1 100644 --- a/docs/installing.md +++ b/docs/installing.md @@ -36,11 +36,19 @@ When you're ready to start the Matrix services (and set them up to auto-start in ansible-playbook -i inventory/hosts setup.yml --tags=start ``` -Now that the services are running, you might want to: - -- **finalize the installation process** (required for federation to work!) by [Configuring Service Discovery via .well-known](configuring-well-known.md) -- or [create your first user account](registering-users.md) -- or [set up the Dimension Integrations Manager](configuring-playbook-dimension.md) -- or [check if services work](maintenance-checking-services.md) -- or learn how to [upgrade your services when new versions are released](maintenance-upgrading-services.md) -- or learn how to [migrate to another server](maintenance-migrating.md) +Now that services are running, you need to **finalize the installation process** (required for federation to work!) by [Configuring Service Discovery via .well-known](configuring-well-known.md) + + +## Things to do next + +If you have started services and **finalized the installation process** (required for federation to work!) by [Configuring Service Discovery via .well-known](configuring-well-known.md), you can: + +- [check if services work](maintenance-checking-services.md) +- or [create your first Matrix user account](registering-users.md) +- or [set up additional services](configuring-playbook.md#other-configuration-options) (bridges to other chat networks, bots, etc.) +- or learn how to [upgrade services when new versions are released](maintenance-upgrading-services.md) +- or learn how to [maintain your server](faq.md#maintenance) +- or join some Matrix rooms: + * via the *Explore rooms* feature in Element or some other client, or by discovering them using this [matrix-static list](https://view.matrix.org). Note: joining large rooms may overload small servers. + * or come say Hi in our support room - [#matrix-docker-ansible-deploy:devture.com](https://matrix.to/#/#matrix-docker-ansible-deploy:devture.com). You might learn something or get to help someone else new to Matrix hosting. +- or help make this playbook better by contributing (code, documentation, or [coffee/beer](https://liberapay.com/s.pantaleev/donate)) diff --git a/docs/maintenance-postgres.md b/docs/maintenance-postgres.md index 7c936479..50f5a55d 100644 --- a/docs/maintenance-postgres.md +++ b/docs/maintenance-postgres.md @@ -19,6 +19,17 @@ You can use the `/usr/local/bin/matrix-postgres-cli` tool to get interactive ter If you are using an [external Postgres server](configuring-playbook-external-postgres.md), the above tool will not be available. +By default, this tool puts you in the `matrix` database, which contains nothing. + +To see the available databases, run `\list` (or just `\l`). + +To change to another database (for example `synapse`), run `\connect synapse` (or just `\c synapse`). + +You can then proceed to write queries. Example: `SELECT COUNT(*) FROM users;` + +**Be careful**. Modifying the database directly (especially as services are running) is dangerous and may lead to irreversible database corruption. +When in doubt, consider [making a backup](#backing-up-postgresql). + ## Vacuuming PostgreSQL @@ -53,6 +64,8 @@ pg_dumpall -h matrix-postgres \ If you are using an [external Postgres server](configuring-playbook-external-postgres.md), the above command will not work, because the credentials file (`/matrix/postgres/env-postgres-psql`) is not available. +If your server is on the ARM32 [architecture](alternative-architectures.md), you may need to remove the `-alpine` suffix from the image name in the command above. + Restoring a backup made this way can be done by [importing it](importing-postgres.md). diff --git a/docs/prerequisites.md b/docs/prerequisites.md index ae9a992f..e678a0bd 100644 --- a/docs/prerequisites.md +++ b/docs/prerequisites.md @@ -1,12 +1,14 @@ # Prerequisites -- An **x86** server running one of these operating systems: +To install Matrix services using this Ansible playbook, you need: + +- (Recommended) An **x86** server ([What kind of server specs do I need?](faq.md#what-kind-of-server-specs-do-i-need)) running one of these operating systems: - **CentOS** (7 only for now; [8 is not yet supported](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/300)) - - **Debian** (9/Stretch+) - - **Ubuntu** (16.04+, although [20.04 may be problematic](ansible.md#supported-ansible-versions)) + - **Debian** (9/Stretch or newer) + - **Ubuntu** (16.04 or newer, although [20.04 may be problematic](ansible.md#supported-ansible-versions)) - **Archlinux** -We only strive to support released stable versions of distributions, not betas or pre-releases. This playbook can take over your whole server or co-exist with other services that you have there. +Generally, newer is better. We only strive to support released stable versions of distributions, not betas or pre-releases. This playbook can take over your whole server or co-exist with other services that you have there. This playbook somewhat supports running on non-`amd64` architectures like ARM. See [Alternative Architectures](alternative-architectures.md). diff --git a/docs/self-building.md b/docs/self-building.md index d8dc03e0..0d41e419 100644 --- a/docs/self-building.md +++ b/docs/self-building.md @@ -2,13 +2,14 @@ **Caution: self-building does not have to be used on its own. See the [Alternative Architectures](alternative-architectures.md) page.** -The playbook supports the self-building of various components, which don't have a container image for your architecture. For `amd64`, self-building is not required. +The playbook supports self-building of various components, which don't have a container image for your architecture (see the [container images we use](container-images.md)). For `amd64`, self-building is not required. For other architectures (e.g. `arm32`, `arm64`), ready-made container images are used when available. If there's no ready-made image for a specific component and said component supports self-building, an image will be built on the host. Building images like this takes more time and resources (some build tools need to get installed by the playbook to assist building). -To make use of self-building, you don't need to do anything besides change your architecture variable (e.g. `matrix_architecture: arm64`). If a component has an image for the specified architecture, the playbook will use it. If not, it will build the image. +To make use of self-building, you don't need to do anything besides change your architecture variable (e.g. `matrix_architecture: arm64`). If a component has an image for the specified architecture, the playbook will use it directly. If not, it will build the image on the server itself. Note that **not all components support self-building yet**. + List of roles where self-building the Docker image is currently possible: - `matrix-synapse` - `matrix-synapse-admin` @@ -18,6 +19,7 @@ List of roles where self-building the Docker image is currently possible: - `matrix-corporal` - `matrix-ma1sd` - `matrix-mailer` +- `matrix-bridge-appservice-irc` - `matrix-bridge-appservice-slack` - `matrix-bridge-mautrix-facebook` - `matrix-bridge-mautrix-hangouts` diff --git a/examples/hosts b/examples/hosts index daf2cfc5..ba08107b 100644 --- a/examples/hosts +++ b/examples/hosts @@ -9,10 +9,11 @@ # to the host line below or by adding `ansible_ssh_pipelining: False` to your variables file. # # If you're running this Ansible playbook on the same server as the one you're installing to, -# consider adding an additional `ansible_connection=local` argument below. +# consider adding an additional `ansible_connection=local` argument to the host line below. # # Ansible may fail to discover which Python interpreter to use on the host for some distros (like Ubuntu 20.04). -# You may sometimes need to explicitly add `ansible_python_interpreter=/usr/bin/python3` to lines below. +# You may sometimes need to explicitly add the argument `ansible_python_interpreter=/usr/bin/python3` +# to the host line below. [matrix_servers] matrix. ansible_host= ansible_ssh_user=root diff --git a/examples/host-vars.yml b/examples/vars.yml similarity index 75% rename from examples/host-vars.yml rename to examples/vars.yml index 409f344a..f79e5e3c 100644 --- a/examples/host-vars.yml +++ b/examples/vars.yml @@ -4,7 +4,7 @@ # Note: this playbook does not touch the server referenced here. # Installation happens on another server ("matrix."). # -# If you've deployed using the wrong domain, you'll have to run the Uninstalling step, +# If you've deployed using the wrong domain, you'll have to run the Uninstalling step, # because you can't change the Domain after deployment. # # Example value: example.com @@ -18,12 +18,18 @@ matrix_domain: YOUR_BARE_DOMAIN_NAME_HERE # you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`). # # Example value: someone@example.com -matrix_ssl_lets_encrypt_support_email: YOUR_EMAIL_ADDRESS_HERE +matrix_ssl_lets_encrypt_support_email: '' # A shared secret (between Coturn and Synapse) used for authentication. # You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`). -matrix_coturn_turn_static_auth_secret: "" +matrix_coturn_turn_static_auth_secret: '' # A secret used to protect access keys issued by the server. # You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`). -matrix_synapse_macaroon_secret_key: "" +matrix_synapse_macaroon_secret_key: '' + +# A Postgres password to use for the superuser Postgres user (called `matrix` by default). +# +# The playbook creates additional Postgres users and databases (one for each enabled service) +# using this superuser account. +matrix_postgres_connection_password: '' diff --git a/group_vars/all/vault.yml b/group_vars/all/vault.yml index 4c6a75fa..d1f32a8d 100644 --- a/group_vars/all/vault.yml +++ b/group_vars/all/vault.yml @@ -1,118 +1,124 @@ $ANSIBLE_VAULT;1.1;AES256 -66633865333931323532653061623366633536306531613738666236333563383462393236393565 -3030643933643865303563613966663935616632306163380a353739306336386239346262373736 -66353963373834393465363763643335623563303236366239643062366132623433623639643431 -3432313830613565610a383938386133623966386631306530313865323234386534613665616262 -35383266663431353736393337313631663739363634396364393135646562633432623330303434 -32623238366437333237376437636333386130326636346462663165366465303232353937646161 -36396466626630333963323131303336663337333061613531346563316666626461646138353931 -65346135396266623434333362376162616166373163303734303033303634353031303762393338 -32353833646439303233343230323131323765623863366663666337316332613339393337353634 -62383131653363613938363834376235316330343064623234393062626134396365393761383339 -35343035646133343733373339353763646562343765363135346233393466666239306333346264 -37353962626132653032343332633936326166323138643134326563306263613234643636393938 -30343566663135343037663138633566653762323037373936336663376132663266663731613731 -64623237326563366439353065326531326136666564396537383462633132373530393131396666 -37356132333764383239306261643163616164383466616463643135326365663763393437623837 -61353134623336666233636563373538623562623132636264663930643661336561346535326262 -34643832303365653839626161633162666565636536656338346332623461373935313030663964 -64323366613130333264313136343135396562643561383235376130666334363536333265616361 -31656165653838313964313864396439323266303439383838323931363161343132326163643264 -31333933386164663438343739616138313064353338616366353237393131353234663634643964 -31336437343261356636396162386434313332356461646561306435363734306634336264653839 -39366138373162393163656437626634373533656137353266313633316239616337633066343863 -32396138393731386131653131633566616161326363613638623635646136633234623664656631 -65633063663330386330343461336133663165323337303534303435646330333066306230636464 -36373061393663646262303832376235366635353535613730633835663236656138653534353134 -31313334623530326534376332616562643139303265623761666438616362613134663533383230 -64653961306565333337353731323135393965656232626630666431633435333531626431633832 -33393138653431613830353635323165383265636661316430336461353034353536623164653764 -39303332323066343266643966336633353561386234643139663539653435653036646665666163 -38636261393361366437336535666563393632316332623366363536373564356238353436326639 -61393533343936613531333439386162386162313138386536663037663936646135643765396664 -34393533363137613039306335313462643437623861396566356338653734356361656233323332 -63353163653162646131356439306662376266636438393435663961303239666339616266383864 -66393837363739353665353234336534333965333735353335316531383963336233623430393432 -66316531663661363362383965643461303339666436376637633233303733333636346365353239 -64363433366135303530636461656165396530353134623536323631646564656236666161303065 -65393438313864623633303365356430626133363165323433353837646139646566316431666233 -35393562316137346331353637626134376138373266373233346136643336356238393065303265 -38396432346539373066313763643137643165613563346536663164363635326635623835373736 -62623231376562343138383563633266303733323937633738303330353436653434353838303563 -34383963323331303932666566666137383366663364646334326632366136663763386138666461 -36653764363931386465313134343638373935323061303063323262626133346562326535663330 -64333662663837333931373038663165633730636332366235636364363335376233373565303330 -61663432363233626236643130303135386334613464343861363338626632613731373231653337 -61353933313935623965333461333662346364343639333362616165656634373031363532373764 -38373139633038623934663639653430633564303830633864366134323835643039386439323961 -37373733626534303764393362633561363933326366323265373335653135653432633664383838 -39613964646361303761663535363837326263396432656565326232313135643335386239653030 -38326562333533363738366161343936303732346439363266626365363833663036323065313563 -33306435353632613862356531636331356538613633393932643165386361376361373434616630 -33623431343031623336386136386661303166343838666330383036653462623339613962616266 -33303536666237376135326139376236616138346231383932326566343537656530396464633963 -37626239623230326137323061343961336561303434343963363530613263633466383232613364 -61366263383434633139613831623738653434303162643738663836373931636136613631356237 -64326133313833336364353164363032363539323630616337336663633238336636633130376138 -37623933646230643439323934666564663132393233363135353630646630633065653839316665 -30386439313965633061366232333764393538323862643130633937363239363338663034623333 -37326564373538623661616637373434633961393361383965383439383065316462373763346465 -64626662646538326532643839333435313066333462656634373832396439393236323865646239 -33313032376166633861336366393036333431313565663266643133366631303034666464333566 -37333631393639646438666665323937656262396665336561623866613766656363353762386265 -30396533653231666439613239316564626236303635646538353335383534666339666566633837 -35653961386135353735636434333830323564656466306365663763346633396530646536396135 -36346133666665343334623536383931373435333562653634303538326136363062613732653832 -31373239616337386332346434393339316563366365653933616161316130626639396134306636 -64613037623539343061313961646263316134613632626165623532306563653639373263633032 -39353864393734396537386134633561386262353837653931653362306139323538623639356539 -61623131626662656531366433333363633766636662376366313537383930373434353865306562 -37383462346264323936396461336365646663653732333765666364386331343339633366393835 -34626533613736323962636236306265323665366163306430336261306130663464323931656133 -34653538376439313036316632366231663264323739373032623332343038383364303335366562 -34656133373465333665643238663665643137626261303938646466343463353562363330323638 -61383232383038363961393636316161303366373237626665303532633336336436626662363733 -34356466333330626337643437656239396566666362353130303836353262336534376564363830 -30313737373730636130383363323737313531663961343438343061393138353765393830653566 -39303439383738613662336663353764323932306632633932643165633431623063346166613966 -62326438383037376435613861623530336161386535303566626461393737303263643932343862 -65386537306235363837653139353130643638626662363166636463633030656339613865313738 -36306365616334363933343165386265386163303432303233633333303134633566643561346661 -38386436633061376437313464336333386332376337383136396638643261613832656133313366 -63343134333030646331663466316331643432386363646134306462303664393165383563353264 -37336465323638656664343830383431376361383535663632393363383234663462393332633438 -36383134366236393834663237666138353661306564303631336330396634366338643034346331 -37393962636165616462653632626333346537313131393261346632613166343663393265356537 -39316633633633396236373532313338386536316337373037666661303030356564383632343065 -62373438396666356336383963356263316531333032653666646162363139303336646431303430 -65363662376130616436316435313464353038313338636239383665663432343930396238393764 -38363734386662313061643139626335336462653635373861663633636364363964316162646235 -62336238653337303863303761303262333666366237626464666666666230613863343631386464 -37313234333561633534313564393932363865363635303434653236393036363033666431376261 -63373234376361386166316238646462343765633331366166383864656130613466633435646338 -38656438616238323638663636626430633766623538343735313631303963306663386331396331 -36393333323938356433316132303637373165323532373363323837643866373034356133393832 -34653437623333323835643865633865626663633362303535336565386636316232376164336432 -65333839646264323939636662373035323231663733623034386135653436663261363634613439 -34646331666633376461323430623862393736633635323266343334646135636434373235643263 -63373733346537633662613832373566396163663864346630313766336166373733303565316534 -62373832323433653464653133363764633565326435663230613966383562313638306235336433 -65666636353331323865386437303035376264316633376637663562373739343636633235393232 -33396534623036323831323065393236366662313263633339333034633231313434323238393834 -37636433376465343334396632666636356665383163323236313633383530663632366232613763 -32326134353936343233306262636163393734366631623861626266393765396230653662393434 -66623335363232643933326333646364616532323434353464366138636335333138613636393435 -34643534653766356539396536656336663537346637313437623835306132653532323963393337 -65626462653939336335623262633262383636316565663661663538383330393233396634363531 -65303331663662633236393664633166666661633735376263346235343362363662626232363232 -66373661393031353064323932613361363734373638623531643863396361323232373265643361 -62323033383431323431633235333535663262353938353064303765336139356134656364343433 -39356232316436633165643361383135373962626536633662663230366662333262626530663633 -66376630633036313537326634616337346466616333396665663063316334636436636438633938 -66393934306562663431396665323238633761383333316665316535656161393862346535393436 -36316462323636623539346666636639626438343539646230653934373764316539366465383135 -36306664623039316336646135633530343235356630613161623638616339306262373466323830 -35666531626530653435326333383831363239383564633531383437376566666232366431343865 -37353430623138393664346237623839656666316638366532633933383534313734373166336464 -37613139313338323962633735636332373265356132643437663137383035646332366330366163 -32623538336565663061326237643763663637643735633431333232356330646533 +34346166323534643061393764656165333234303064313338623564303233666666323136666331 +3331353237396233323933373433343962356231346461370a383835396234396636343961663932 +66383935653634633465383831636538653331393030323639306265333065376461323932396639 +3537383434303562610a373930363836636262623031393561326135323738393033336239623831 +66646563393232353738633539323664646635313961353534353238343866646139386634626562 +38326239386539386362393532666434323561316437633464626465343936356138326664316662 +34643133306639653237343135313835643236636631643034366263653164383535393037326365 +31663166623737356430386665303236343333316362356361306339363664303166316134343834 +35303064666163663431663432316366653961666464383965613837366666366638666262653237 +32366338393966386661623734643633363532636135623330323835313064333962333066343033 +62396362613564393035663063393038643238663263323464333333306538353634636663383962 +61616331393034623365353636613030303037663663386663656237663263386334393730393964 +62623433346264633332363764303964343836306365646466346137383031366261333065383162 +63633039313932656363613632613839316437373634316462326136373537386461623064343565 +33386161373764636661663166613339313438356232336466366230336237653931386562663535 +64646630613738646666303663656132366537336563626265303230326662623366613138383436 +64323431333139373639356438623832363335323632306661386436366562313732613135653439 +66653337366263666433653837373761616365303435363366623661616464353736643365303962 +39313836633839316461323837373232636639376438316338383039313931646130663831306333 +38663039306464653863636437323730663731623535653239316266656561646639383561626337 +31393131316463636266303537663135323238316262346435343437343635363932623738316466 +63366364643534653365373565613038306230346131623061616263323035346139333836643439 +37316565303262643762633765666163376663666137356431313963353639313738643932363166 +62313837646164326133616362313238633334366238613136396165366635323564363434303130 +64323532393234613461393436613364366331633364333265386532353062373338663563323033 +35346332623762386430626238333830393339353766383663306632333639396139393834326239 +32333965373861363232316162306132646435336562346563653361356139376539343266613264 +33303765316137613232623136343938633534396235393936356664613864333961656536333636 +39383730366439633861633166383830376134633835643435333039313237333862666665373136 +33646330386434663539366135643132383333316531346264653731323536333464623133383835 +34373936653033333565386335363538623930373539666236376362376433396337643762356535 +37383039373334343230663338623339666365343865643137316531373839623035353937333562 +34373439353761386230393936626164346437613565653835366438313732666466303131666638 +39663664333362626163366339623738636133373262626366636364333735393730643231613061 +34666434626461643837633861613736303239343133663663656265363465663038356165323562 +64353731613931633536373262623137313535323262373532653233326564623264663032636666 +36663537303930363132353463393735666438396131613536353938643039636238653465323066 +65633261396433333332633465643239343930366631613436366361613366326434353534336165 +39663535653638323334376333383031343832653132343162326137356435616262346330353462 +39663266633235353831326462623333626631383364303039336631323131393265346335613036 +61333430303837323135393631653233633066343264636439636561633465326333313838333737 +38316364643262356535366438353830666364386635353835656539383434666633343831393331 +61613132363062633133663631623134393465393561346235653763326631343935373330373832 +39613935326535386463653563316331313638356165646461353931653439313163393166366463 +38623262373939373066626239656130666164373837303665356566313636323830356539613339 +66616263363738653138393532653539643766633162343838376362636362616335383365366633 +33303038303332386239396534663164386564653637333338383838323232336337646330373761 +62626562306239396163323934383566373831323933383231333365626436383135653431336364 +39343637666638613438376162633365613365343837633863373931346664366163346335353333 +30613265323034653461376461313231663731343438623331393739323961363862326233653931 +36353739346134363261303536616235363331623539313432623062326433346238393663316139 +61656530373164616564626236656232343036363337333362353532363132303066303032656630 +61303431643763373839656135383730653533396434626166666365656262373231383265303564 +34613065356538636339626137663964343462633135376463366262636338373666383861383035 +65313139316430316533396365616138653536656561346636316565616636363434303762333364 +30366331393566663436343839323562323432393631383939633631366435666363303037336536 +66623736316539303836353935303738666366383533653863323733636365613233363634306438 +35353332653134646663633838316264313434303639643937333531386335613537626161663035 +33323935323035386364613238336335616532626462656265643531656464626232666133343439 +39376263353764613033326363386134386338303734373761396361613266316466333663346636 +38393136666333336437313731336533633133663631643837313366396531353164313264386238 +39306532303161643864643434656538343030366334663737643036643565303534663130613566 +38393661336637396433346166633731663265393864323565353464613337363238646532353765 +61666633313235613839363762663737333234316562643837636164373536306435383634646161 +35663134333837663964366237613035663733666463396132396533623833383663386532313830 +61326436363431653934383234353034346434643433363933646538636636663661363362663662 +62306636303139616138323365646437643135386131396237396132643362346566626463323937 +39623731303766626431313062356335633135373031643266333438373964653538386265326637 +35613730346137366664636433353062613735303035636265313566343131643834613861343762 +61356237626533613366333462323632363731363066616361393536383064663566386536373533 +62376631303666323031313666363631326636313938346564363137656532626330666362663065 +35346662373164363031356430383838356136663764336336393532363561353535373838626135 +65633339653837336238386331653030613362376133643038373839643361633036653039356235 +37656136313839656264616637386538373734343962643261323031666366643837633863353433 +39306236376230613566323562356631613763366130623964336364666132363863303032363863 +63613138303735666539306464303564356234643266636432396561373566386265633531316363 +64373334323837386531386233363935393035316362303661343535303033643739343235303563 +36343064383337306534396565393164306436303861396639343937366330326536633935323136 +36323633363634356265353333396166376634663161366438333035613162366366636239336366 +63336635366438396534366234643332343231626466636464653761333763383062626234336635 +37346332323363313138656539653264363531633331623435353666303536616561623863363962 +35653732353239306437626265356434393930316534393365383931396337303166613237613433 +39376531303862343630336461373839323436626236313535663339636638386566356535356430 +35663331633137323166623864373437383265613030336261306563326232383839623965346239 +37666135646631663763333035616637616661623034633530333033323266663234336130383632 +30613437323365333938313364363331393364636435306231313535363237643735303539633036 +34313833336661616334366265383565393332396131303866333462356638646139343739663838 +30366630343233393161643431356339323565653431313166613836636635363764313261343066 +35653934323030633262323333613265393461383034303266323934353334663832616461393564 +36646663613062333137666635623137336163666363663237363064646437643835646164623665 +65656465393133663663646536663332633732343635393637646634663632613864343334353933 +61393238383336353635633334616530363831396431393337666438663163383538613137636238 +39303261306339303633396166363838316330643238613939656335643431323065623764336465 +30636330363036343164306564323732393331653465343038663466643031303235643130323339 +37343334646435376564663030653032393165313337363165356466323035343530343261623632 +33353533626635346433306130613862653061323438306439303664376563306565623736643034 +63623432363031346639363234303866346462353536346431303539396632636332643739653934 +33313836666637383363353166663363353535316535343334346164653939323632396263306637 +39653663643238636633313736336236333531343362353430623964646366346261633663353930 +32636364376436303264383562373462393461393432333366623834333164646136666133306337 +31663733636162653466363134393163343035393533386132346235626435363336323163663863 +31303435323032613462303163663836623135373330393631356334623730643266373063616133 +35393862643738626530616365636665636462663235336337613339313431616435313535303365 +66383332336662623964333434356333393636393134663034306432633638616439383739366634 +61373864646634383563623632326131626430653334323362363637343238663836343136353930 +63363239343539636365386566303933393431353064313064346663613930373737336632363935 +38616131383034643363346236396564303936303437363133613432656239356463646130316438 +66333739356565666433323839646436666630613132373164303933303431363231653630356262 +38313766623133613837633232323137626164333138643834316431313666396462373532633338 +62616561626236633136313261326563316662306632633438336139653338386361386632386362 +65333933313832616562623739633037616634343333393731336265636266333934653638663039 +35396334633932636163346662623961613463376135323938353333653631613064316539656164 +32653833623965366135656635343465373935333534366466396132336561663831613661643862 +61373037323766633539306161303234373539626133323763386664306261386634366136623832 +64633266333634356132343462336236623637633539626531363866336230373138333836653566 +62656231366234626561636564666331313863326266313630623734363132363663663365643666 +33333033356334333061613539366163653731306639653664336635386130303339303737626338 +38643630356666363065373734333031633033666530303763663461326135363336653334363933 +38363336333164363633333431303935333336323237303536363062656237353262383739323063 +31303662633033353639306634336338616539333932343164383732303437353766623532663232 +36333961323232663661323439343330393061666661306130633738633331363935396363353533 +63313965333435633865373137383031333738353263346538363236636639373265306634326266 +363532356361373037656632383034323936 diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index e5517084..07900778 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -18,6 +18,10 @@ matrix_identity_server_url: "{{ ('https://' + matrix_server_fqn_matrix) if matrix_ma1sd_enabled else None }}" +# If Synapse workers are enabled and matrix-nginx-proxy is disabled, certain APIs may not work over 'http://matrix-synapse:8008'. +# This is because we explicitly disable them for the main Synapse process. +matrix_homeserver_container_url: "{{ 'http://matrix-nginx-proxy:12080' if matrix_nginx_proxy_enabled else 'http://matrix-synapse:8008' }}" + ###################################################################### # # /matrix-base @@ -149,6 +153,8 @@ matrix_appservice_slack_database_password: "{{ matrix_synapse_macaroon_secret_ke # We don't enable bridges by default. matrix_appservice_irc_enabled: false +matrix_appservice_irc_container_self_build: "{{ matrix_architecture != 'amd64' }}" + # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-appservice-irc over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-appservice-irc's client-server port to the local host. @@ -208,7 +214,8 @@ matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_pro matrix_mautrix_facebook_bridge_presence: "{{ matrix_synapse_use_presence if matrix_synapse_enabled else true }}" -# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +# We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain +# and point them to a migration path. matrix_mautrix_facebook_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" matrix_mautrix_facebook_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.fb.db') | to_uuid }}" @@ -258,6 +265,46 @@ matrix_mautrix_hangouts_database_password: "{{ matrix_synapse_macaroon_secret_ke ###################################################################### +###################################################################### +# +# matrix-bridge-mautrix-instagram +# +###################################################################### + +# We don't enable bridges by default. +matrix_mautrix_instagram_enabled: false + +matrix_mautrix_instagram_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm64'] }}" + +matrix_mautrix_instagram_systemd_required_services_list: | + {{ + ['docker.service'] + + + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + }} + +matrix_mautrix_instagram_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'ig.as.token') | to_uuid }}" + +matrix_mautrix_instagram_homeserver_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'ig.hs.token') | to_uuid }}" + +matrix_mautrix_instagram_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" + +matrix_mautrix_instagram_bridge_presence: "{{ matrix_synapse_use_presence if matrix_synapse_enabled else true }}" + +# We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain +# and point them to a migration path. +matrix_mautrix_instagram_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mautrix_instagram_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.ig.db') | to_uuid }}" + +###################################################################### +# +# /matrix-bridge-mautrix-instagram +# +###################################################################### + + ###################################################################### # # matrix-bridge-mautrix-signal @@ -280,7 +327,7 @@ matrix_mautrix_signal_systemd_required_services_list: | matrix_mautrix_signal_homeserver_domain: '{{ matrix_domain }}' -matrix_mautrix_signal_homeserver_address: "{{ 'http://matrix-synapse:8008' if matrix_synapse_enabled else '' }}" +matrix_mautrix_signal_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mautrix_signal_homeserver_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'si.hs.token') | to_uuid }}" @@ -619,6 +666,41 @@ matrix_mx_puppet_steam_database_password: "{{ matrix_synapse_macaroon_secret_key # ###################################################################### +###################################################################### +# +# matrix-bridge-mx-puppet-groupme +# +###################################################################### + +# We don't enable bridges by default. +matrix_mx_puppet_groupme_enabled: false + +matrix_mx_puppet_groupme_container_image_self_build: "{{ matrix_architecture != 'amd64'}}" + +matrix_mx_puppet_groupme_systemd_required_services_list: | + {{ + ['docker.service'] + + + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + }} + +matrix_mx_puppet_groupme_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxgro.as.tok') | to_uuid }}" + +matrix_mx_puppet_groupme_homeserver_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxgro.hs.tok') | to_uuid }}" + +matrix_mx_puppet_groupme_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" + +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mx_puppet_groupme_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mx_puppet_groupme_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxpup.groupme.db') | to_uuid }}" + +###################################################################### +# +# /matrix-bridge-mx-puppet-groupme +# +###################################################################### ###################################################################### # @@ -670,7 +752,8 @@ matrix_corporal_systemd_required_services_list: | (['matrix-synapse.service']) }} -matrix_corporal_matrix_homeserver_api_endpoint: "http://matrix-synapse:8008" +# This goes to Synapse's vhost +matrix_corporal_matrix_homeserver_api_endpoint: "{{ matrix_homeserver_container_url }}" matrix_corporal_matrix_auth_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret }}" @@ -754,7 +837,30 @@ matrix_dimension_database_password: "{{ matrix_synapse_macaroon_secret_key | pas # ###################################################################### +###################################################################### +# +# matrix-etherpad +# +###################################################################### + +matrix_etherpad_enabled: false + +matrix_etherpad_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9001' }}" + +matrix_etherpad_systemd_required_services_list: | + {{ + ['docker.service'] + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + }} +matrix_etherpad_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'etherpad.db') | to_uuid }}" + +###################################################################### +# +# /matrix-etherpad +# +###################################################################### ###################################################################### # @@ -817,6 +923,13 @@ matrix_jitsi_web_stun_servers: | else [ 'stun:meet-jit-si-turnrelay.jitsi.net:443'] }} +# If the self-hosted Etherpad instance is available, it will also show up in Jitsi conferences, +# unless explicitly disabled by setting `matrix_jitsi_etherpad_enabled` to false. +# Falls back to the scalar.vector.im etherpad in case someone sets `matrix_jitsi_etherpad_enabled` to true, +# while also setting `matrix_etherpad_enabled` to false. +matrix_jitsi_etherpad_enabled: "{{ matrix_etherpad_enabled }}" +matrix_jitsi_etherpad_base: "{{ matrix_etherpad_base_url if matrix_etherpad_enabled else 'https://scalar.vector.im/etherpad' }}" + ###################################################################### # # /matrix-jitsi @@ -882,7 +995,7 @@ matrix_ma1sd_synapsesql_connection: //{{ matrix_synapse_database_host }}/{{ matr matrix_ma1sd_dns_overwrite_enabled: true matrix_ma1sd_dns_overwrite_homeserver_client_name: "{{ matrix_server_fqn_matrix }}" -matrix_ma1sd_dns_overwrite_homeserver_client_value: "http://{{ 'matrix-corporal:41080' if matrix_corporal_enabled else 'matrix-synapse:8008' }}" +matrix_ma1sd_dns_overwrite_homeserver_client_value: "http://{{ matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container }}" # By default, we send mail through the `matrix-mailer` service. matrix_ma1sd_threepid_medium_email_identity_from: "{{ matrix_mailer_sender_address }}" @@ -929,8 +1042,8 @@ matrix_ma1sd_database_password: "{{ matrix_synapse_macaroon_secret_key | passwor # If that's not the case, you may wish to disable this and take care of proxying yourself. matrix_nginx_proxy_enabled: true -matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "{{ 'matrix-corporal:41080' if matrix_corporal_enabled else 'matrix-synapse:8008' }}" -matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "{{ '127.0.0.1:41080' if matrix_corporal_enabled else '127.0.0.1:8008' }}" +matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "{{ 'matrix-corporal:41080' if matrix_corporal_enabled else 'matrix-nginx-proxy:12080' }}" +matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "{{ '127.0.0.1:41080' if matrix_corporal_enabled else '127.0.0.1:12080' }}" matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb: "{{ matrix_synapse_max_upload_size_mb }}" matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_admin_api_enabled: "{{ matrix_synapse_admin_enabled }}" @@ -941,6 +1054,7 @@ matrix_nginx_proxy_proxy_matrix_enabled: true matrix_nginx_proxy_proxy_element_enabled: "{{ matrix_client_element_enabled }}" matrix_nginx_proxy_proxy_dimension_enabled: "{{ matrix_dimension_enabled }}" matrix_nginx_proxy_proxy_jitsi_enabled: "{{ matrix_jitsi_enabled }}" +matrix_nginx_proxy_proxy_grafana_enabled: "{{ matrix_grafana_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: "{{ matrix_corporal_enabled and matrix_corporal_http_api_enabled }}" matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corporal:41081" @@ -953,12 +1067,19 @@ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:809 # By default, we do TLS termination for the Matrix Federation API (port 8448) at matrix-nginx-proxy. # Unless this is handled there OR Synapse's federation listener port is disabled, we'll reverse-proxy. matrix_nginx_proxy_proxy_matrix_federation_api_enabled: "{{ matrix_synapse_federation_port_enabled and not matrix_synapse_tls_federation_listener_enabled }}" -matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-synapse:8048" -matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "127.0.0.1:8048" +matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-nginx-proxy:12088" +matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "127.0.0.1:12088" + +# Settings controlling matrix-synapse-proxy.conf +matrix_nginx_proxy_proxy_synapse_enabled: "{{ matrix_synapse_enabled }}" +matrix_nginx_proxy_proxy_synapse_federation_api_enabled: "{{ matrix_nginx_proxy_proxy_matrix_federation_api_enabled }}" matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port }}" -matrix_nginx_proxy_proxy_synapse_metrics: "{{ matrix_synapse_metrics_enabled }}" +# This used to be hooked to `matrix_synapse_metrics_enabled`, but we don't do it anymore. +# The fact that someone wishes to enable Synapse metrics does not necessarily mean they want to make them public. +# A local Prometheus can consume them over the container network. +matrix_nginx_proxy_proxy_synapse_metrics: false matrix_nginx_proxy_proxy_synapse_metrics_addr_with_container: "matrix-synapse:{{ matrix_synapse_metrics_port }}" matrix_nginx_proxy_proxy_synapse_metrics_addr_sans_container: "127.0.0.1:{{ matrix_synapse_metrics_port }}" @@ -968,6 +1089,16 @@ matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "{{ m matrix_nginx_proxy_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_nginx_proxy_synapse_presence_disabled: "{{ not matrix_synapse_use_presence }}" + +matrix_nginx_proxy_synapse_workers_enabled: "{{ matrix_synapse_workers_enabled }}" +matrix_nginx_proxy_synapse_workers_list: "{{ matrix_synapse_workers_enabled_list }}" +matrix_nginx_proxy_synapse_generic_worker_client_server_locations: "{{ matrix_synapse_workers_generic_worker_client_server_endpoints }}" +matrix_nginx_proxy_synapse_generic_worker_federation_locations: "{{ matrix_synapse_workers_generic_worker_federation_endpoints }}" +matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers_media_repository_endpoints|default([]) }}" +matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}" +matrix_nginx_proxy_synapse_frontend_proxy_locations: "{{ matrix_synapse_workers_frontend_proxy_endpoints|default([]) }}" + matrix_nginx_proxy_systemd_wanted_services_list: | {{ (['matrix-synapse.service']) @@ -991,7 +1122,11 @@ matrix_ssl_domains_to_obtain_certificates_for: | + ([matrix_server_fqn_jitsi] if matrix_jitsi_enabled else []) + + ([matrix_server_fqn_grafana] if matrix_grafana_enabled else []) + + ([matrix_domain] if matrix_nginx_proxy_base_domain_serving_enabled else []) + + + matrix_ssl_additional_domains_to_obtain_certificates_for }} matrix_ssl_architecture: "{{ @@ -1020,16 +1155,22 @@ matrix_ssl_pre_obtaining_required_service_name: "{{ 'matrix-dynamic-dns' if matr matrix_postgres_enabled: true -matrix_postgres_connection_hostname: "matrix-postgres" -matrix_postgres_connection_username: "synapse" -# Please note that the max length of the password is 99 characters -matrix_postgres_connection_password: "synapse-password" -matrix_postgres_db_name: "homeserver" +matrix_postgres_architecture: "{{ matrix_architecture }}" + +# We unset this if internal Postgres disabled, which will cascade to some other variables +# and tell users they need to set it (either here or in those variables). +matrix_postgres_connection_hostname: "{{ 'matrix-postgres' if matrix_postgres_enabled else '' }}" matrix_postgres_pgloader_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" matrix_postgres_additional_databases: | {{ + ([{ + 'name': matrix_synapse_database_database, + 'username': matrix_synapse_database_user, + 'password': matrix_synapse_database_password, + }] if (matrix_synapse_enabled and matrix_synapse_database_database != matrix_postgres_db_name and matrix_synapse_database_host == 'matrix-postgres') else []) + + ([{ 'name': matrix_ma1sd_database_name, 'username': matrix_ma1sd_database_username, @@ -1078,6 +1219,12 @@ matrix_postgres_additional_databases: | 'password': matrix_mautrix_hangouts_database_password, }] if (matrix_mautrix_hangouts_enabled and matrix_mautrix_hangouts_database_engine == 'postgres' and matrix_mautrix_hangouts_database_hostname == 'matrix-postgres') else []) + + ([{ + 'name': matrix_mautrix_instagram_database_name, + 'username': matrix_mautrix_instagram_database_username, + 'password': matrix_mautrix_instagram_database_password, + }] if (matrix_mautrix_instagram_enabled and matrix_mautrix_instagram_database_engine == 'postgres' and matrix_mautrix_instagram_database_hostname == 'matrix-postgres') else []) + + ([{ 'name': matrix_mautrix_signal_database_name, 'username': matrix_mautrix_signal_database_username, @@ -1132,11 +1279,23 @@ matrix_postgres_additional_databases: | 'password': matrix_mx_puppet_steam_database_password, }] if (matrix_mx_puppet_steam_enabled and matrix_mx_puppet_steam_database_engine == 'postgres' and matrix_mx_puppet_steam_database_hostname == 'matrix-postgres') else []) + + ([{ + 'name': matrix_mx_puppet_groupme_database_name, + 'username': matrix_mx_puppet_groupme_database_username, + 'password': matrix_mx_puppet_groupme_database_password, + }] if (matrix_mx_puppet_groupme_enabled and matrix_mx_puppet_groupme_database_engine == 'postgres' and matrix_mx_puppet_groupme_database_hostname == 'matrix-postgres') else []) + + ([{ 'name': matrix_dimension_database_name, 'username': matrix_dimension_database_username, 'password': matrix_dimension_database_password, }] if (matrix_dimension_enabled and matrix_dimension_database_engine == 'postgres' and matrix_dimension_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_etherpad_database_name, + 'username': matrix_etherpad_database_username, + 'password': matrix_etherpad_database_password, + }] if (matrix_etherpad_enabled and matrix_etherpad_database_engine == 'postgres' and matrix_etherpad_database_hostname == 'matrix-postgres') else []) }} matrix_postgres_import_roles_to_ignore: | @@ -1161,6 +1320,22 @@ matrix_postgres_import_databases_to_ignore: | +###################################################################### +# +# matrix-redis +# +###################################################################### + +matrix_redis_enabled: "{{ matrix_synapse_workers_enabled }}" + +###################################################################### +# +# /matrix-redis +# +###################################################################### + + + ###################################################################### # # matrix-client-element @@ -1216,7 +1391,7 @@ matrix_client_element_jitsi_preferredDomain: "{{ matrix_server_fqn_jitsi if matr # ###################################################################### -matrix_synapse_container_image_self_build: "{{ matrix_architecture not in ['arm32', 'arm64', 'amd64'] }}" +matrix_synapse_container_image_self_build: "{{ matrix_architecture not in ['arm64', 'amd64'] }}" # When ma1sd is enabled, we can use it to validate email addresses and phone numbers. # Synapse can validate email addresses by itself as well, but it's probably not what we want by default when we have an identity server. @@ -1241,11 +1416,11 @@ matrix_synapse_container_metrics_api_host_bind_port: "{{ '127.0.0.1:9100' if (ma # # For exposing the Synapse Manhole port (plain HTTP) to the local host. matrix_synapse_container_manhole_api_host_bind_port: "{{ '127.0.0.1:9000' if matrix_synapse_manhole_enabled else '' }}" +# +# For exposing the Synapse worker (and metrics) ports to the local host. +matrix_synapse_workers_container_host_bind_address: "{{ '127.0.0.1' if (matrix_synapse_workers_enabled and not matrix_nginx_proxy_enabled) else '' }}" -matrix_synapse_database_host: "{{ matrix_postgres_connection_hostname }}" -matrix_synapse_database_user: "{{ matrix_postgres_connection_username }}" -matrix_synapse_database_password: "{{ matrix_postgres_connection_password }}" -matrix_synapse_database_database: "{{ matrix_postgres_db_name }}" +matrix_synapse_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'synapse.db') | to_uuid }}" # We do not enable TLS in Synapse by default. # TLS is handled by the matrix-nginx-proxy, which proxies the requests to Synapse. @@ -1255,6 +1430,9 @@ matrix_synapse_tls_private_key_path: ~ matrix_synapse_federation_port_openid_resource_required: "{{ not matrix_synapse_federation_enabled and (matrix_dimension_enabled or matrix_ma1sd_enabled) }}" +# If someone instals Prometheus via the playbook, they most likely wish to monitor Synapse. +matrix_synapse_metrics_enabled: "{{ matrix_prometheus_enabled }}" + matrix_synapse_email_enabled: "{{ matrix_mailer_enabled }}" matrix_synapse_email_smtp_host: "matrix-mailer" matrix_synapse_email_smtp_port: 8025 @@ -1295,6 +1473,11 @@ matrix_synapse_systemd_wanted_services_list: | (['matrix-mailer.service'] if matrix_mailer_enabled else []) }} +# Synapse workers (used for parallel load-scaling) need Redis for IPC. +matrix_synapse_redis_enabled: "{{ matrix_redis_enabled }}" +matrix_synapse_redis_host: "{{ 'matrix-redis' if matrix_redis_enabled else '' }}" +matrix_synapse_redis_password: "{{ matrix_redis_connection_password if matrix_redis_enabled else '' }}" + ###################################################################### # # /matrix-synapse @@ -1326,6 +1509,75 @@ matrix_synapse_admin_container_self_build: "{{ matrix_architecture != 'amd64' }} +###################################################################### +# +# matrix-prometheus-node-exporter +# +###################################################################### + +matrix_prometheus_node_exporter_enabled: false + +# Normally, matrix-nginx-proxy is enabled and nginx can reach Prometheus Node Exporter over the container network. +# If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose +# Prometheus' HTTP port to the local host. +matrix_prometheus_node_exporter_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9100' }}" + +###################################################################### +# +# /matrix-prometheus-node-exporter +# +###################################################################### + + + +###################################################################### +# +# matrix-prometheus +# +###################################################################### + +matrix_prometheus_enabled: false + +# Normally, matrix-nginx-proxy is enabled and nginx can reach Prometheus over the container network. +# If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose +# Prometheus' HTTP port to the local host. +matrix_prometheus_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:9090' }}" + +matrix_prometheus_scraper_synapse_enabled: "{{ matrix_synapse_enabled and matrix_synapse_metrics_enabled }}" +matrix_prometheus_scraper_synapse_targets: ['matrix-synapse:{{ matrix_synapse_metrics_port }}'] +matrix_prometheus_scraper_synapse_rules_synapse_tag: "{{ matrix_synapse_docker_image_tag }}" + +matrix_prometheus_scraper_node_enabled: "{{ matrix_prometheus_node_exporter_enabled }}" + +###################################################################### +# +# /matrix-prometheus +# +###################################################################### + + + +###################################################################### +# +# matrix-grafana +# +###################################################################### + +matrix_grafana_enabled: false + +# Normally, matrix-nginx-proxy is enabled and nginx can reach Grafana over the container network. +# If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose +# Grafana's HTTP port to the local host. +matrix_grafana_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:3000' }}" + +###################################################################### +# +# /matrix-grafana +# +###################################################################### + + + ###################################################################### # # matrix-registration @@ -1343,7 +1595,7 @@ matrix_registration_riot_instance: "{{ ('https://' + matrix_server_fqn_element) matrix_registration_shared_secret: "{{ matrix_synapse_registration_shared_secret if matrix_synapse_enabled else '' }}" -matrix_registration_server_location: "{{ 'http://matrix-synapse:8008' if matrix_synapse_enabled else '' }}" +matrix_registration_server_location: "{{ matrix_homeserver_container_url }}" matrix_registration_api_validate_certs: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" diff --git a/inventory/host_vars/matrix.awful.club/vars.yml b/inventory/host_vars/matrix.awful.club/vars.yml index f4e31127..3d2dbafb 100644 --- a/inventory/host_vars/matrix.awful.club/vars.yml +++ b/inventory/host_vars/matrix.awful.club/vars.yml @@ -89,3 +89,55 @@ matrix_mx_puppet_twitter_consumer_secret: "{{ MATRIX_MX_PUPPET_TWITTER_CONSUMER_ matrix_mx_puppet_twitter_access_token: "{{ MATRIX_MX_PUPPET_TWITTER_ACCESS_TOKEN }}" matrix_mx_puppet_twitter_access_token_secret: "{{ MATRIX_MX_PUPPET_TWITTER_ACCESS_TOKEN_SECRET }}" matrix_mx_puppet_twitter_environment: "{{ MATRIX_MX_PUPPET_TWITTER_ENVIRONMENT }}" + +# added by jlj -- irc bullshit +matrix_appservice_irc_enabled: true +matrix_appservice_irc_ircService_servers: + bouncer.awful.club: + name: "awful bouncer" + port: 5000 + ssl: true + sasl: false + allowExpiredCerts: true + sendConnectionMessages: true + botConfig: + enabled: true + nick: "blindidiotgod" + joinChannelsIfNoUsers: true + privateMessages: + enabled: true + federate: false + dynamicChannels: + enabled: true + createAlias: true + published: true + joinRule: public + groupId: +myircnetwork:localhost + federate: true + aliasTemplate: "#irc_$CHANNEL" + membershipLists: + enabled: false + floodDelayMs: 10000 + global: + ircToMatrix: + initial: false + incremental: false + matrixToIrc: + initial: false + incremental: false + matrixClients: + userTemplate: "@irc_$NICK" + displayName: "$NICK (IRC)" + joinAttempts: -1 + ircClients: + nickTemplate: "$DISPLAY" + allowNickChanges: true + maxClients: 30 + idleTimeout: 0 + reconnectIntervalMs: 5000 + concurrentReconnectLimit: 50 + lineLimit: 3 + +# added by jlj 2/20/21 - synapse / psql updates require manual intervention +matrix_postgres_connection_password: "{{ vault_matrix_postgres_connection_password }}" + diff --git a/roles/matrix-base/defaults/main.yml b/roles/matrix-base/defaults/main.yml index d8285e1c..39a8cffc 100644 --- a/roles/matrix-base/defaults/main.yml +++ b/roles/matrix-base/defaults/main.yml @@ -21,6 +21,9 @@ matrix_server_fqn_dimension: "dimension.{{ matrix_domain }}" # This is where you access Jitsi. matrix_server_fqn_jitsi: "jitsi.{{ matrix_domain }}" +# This is where you access Grafana. +matrix_server_fqn_grafana: "stats.{{ matrix_domain }}" + matrix_federation_public_port: 8448 # The architecture that your server runs. @@ -66,12 +69,18 @@ matrix_host_command_chown: "/usr/bin/env chown" matrix_host_command_fusermount: "/usr/bin/env fusermount" matrix_host_command_openssl: "/usr/bin/env openssl" matrix_host_command_systemctl: "/usr/bin/env systemctl" +matrix_host_command_sh: "/usr/bin/env sh" matrix_ntpd_package: "ntp" matrix_ntpd_service: "{{ 'ntpd' if ansible_os_family == 'RedHat' or ansible_distribution == 'Archlinux' else 'ntp' }}" matrix_homeserver_url: "https://{{ matrix_server_fqn_matrix }}" +# Specifies where the homeserver is on the container network. +# Where this is depends on whether there's a reverse-proxy in front of it, etc. +# This likely gets overriden elsewhere. +matrix_homeserver_container_url: "http://matrix-synapse:8008" + matrix_identity_server_url: ~ matrix_integration_manager_rest_url: ~ diff --git a/roles/matrix-base/tasks/server_base/setup_debian.yml b/roles/matrix-base/tasks/server_base/setup_debian.yml index 37706d1f..54e52c1b 100644 --- a/roles/matrix-base/tasks/server_base/setup_debian.yml +++ b/roles/matrix-base/tasks/server_base/setup_debian.yml @@ -23,7 +23,14 @@ repo: "deb [arch={{ matrix_debian_arch }}] https://download.docker.com/linux/{{ ansible_distribution|lower }} {{ ansible_distribution_release }} stable" state: present update_cache: yes - when: matrix_docker_installation_enabled|bool and matrix_docker_package_name == 'docker-ce' + when: matrix_docker_installation_enabled|bool and matrix_docker_package_name == 'docker-ce' and not ansible_distribution_release == 'bullseye' + +- name: Ensure Docker repository is enabled (using Debian Buster on Debian Bullseye, for which there is no Docker yet) + apt_repository: + repo: "deb [arch={{ matrix_debian_arch }}] https://download.docker.com/linux/{{ ansible_distribution|lower }} buster stable" + state: present + update_cache: yes + when: matrix_docker_installation_enabled|bool and matrix_docker_package_name == 'docker-ce' and ansible_distribution_release == 'bullseye' - name: Ensure APT packages are installed apt: diff --git a/roles/matrix-base/templates/usr-local-bin/matrix-remove-all.j2 b/roles/matrix-base/templates/usr-local-bin/matrix-remove-all.j2 index 2a647aba..f4b23b44 100644 --- a/roles/matrix-base/templates/usr-local-bin/matrix-remove-all.j2 +++ b/roles/matrix-base/templates/usr-local-bin/matrix-remove-all.j2 @@ -15,11 +15,14 @@ if [ "$sure" != "Yes, I really want to remove everything!" ]; then exit 0 else echo "Stop and remove matrix services" - for s in $(find {{ matrix_systemd_path }}/ -name "matrix-*" -printf "%f\n"); do - systemctl stop $s + + for s in $(find {{ matrix_systemd_path }}/ -type f -name "matrix-*" -printf "%f\n"); do + systemctl disable --now $s rm -f {{ matrix_systemd_path }}/$s done + systemctl daemon-reload + echo "Remove matrix scripts" find {{ matrix_local_bin_path }}/ -name "matrix-*" -delete echo "Remove unused Docker images and resources" diff --git a/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml b/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml index 29bc8307..c3deb2f2 100644 --- a/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml +++ b/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml @@ -58,7 +58,7 @@ matrix_bot_matrix_reminder_bot_matrix_user_id: '@{{ matrix_bot_matrix_reminder_b # The password that the bot uses to authenticate. matrix_bot_matrix_reminder_bot_matrix_user_password: '' -matrix_bot_matrix_reminder_bot_matrix_homeserver_url: 'http://matrix-synapse:8008' +matrix_bot_matrix_reminder_bot_matrix_homeserver_url: "{{ matrix_homeserver_container_url }}" # The timezone to use when creating reminders. # Examples: 'Europe/London', 'Etc/UTC' diff --git a/roles/matrix-bot-matrix-reminder-bot/tasks/setup_uninstall.yml b/roles/matrix-bot-matrix-reminder-bot/tasks/setup_uninstall.yml index 744f474d..141e61ba 100644 --- a/roles/matrix-bot-matrix-reminder-bot/tasks/setup_uninstall.yml +++ b/roles/matrix-bot-matrix-reminder-bot/tasks/setup_uninstall.yml @@ -7,7 +7,7 @@ - name: Ensure matrix-matrix-reminder-bot is stopped service: - name: matrix-matrix-reminder-bot + name: matrix-bot-matrix-reminder-bot state: stopped daemon_reload: yes register: stopping_result diff --git a/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 b/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 index 825072e8..b1fe3c32 100644 --- a/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 +++ b/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-bot-matrix-reminder-bot -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-bot-matrix-reminder-bot +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-bot-matrix-reminder-bot 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-bot-matrix-reminder-bot 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-bot-matrix-reminder-bot \ --log-driver=none \ @@ -32,8 +32,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-bot-matrix-rem {{ matrix_bot_matrix_reminder_bot_docker_image }} \ -c "matrix-reminder-bot /config/config.yaml" -ExecStop=-{{ matrix_host_command_docker }} kill matrix-bot-matrix-reminder-bot -ExecStop=-{{ matrix_host_command_docker }} rm matrix-bot-matrix-reminder-bot +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-bot-matrix-reminder-bot 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-bot-matrix-reminder-bot 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-bot-matrix-reminder-bot diff --git a/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 b/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 index b2ecd198..6286a5d4 100644 --- a/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 +++ b/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 @@ -1,10 +1,10 @@ #jinja2: lstrip_blocks: "True" bridge: # Domain part of the bridge, e.g. matrix.org - domain: {{ matrix_appservice_discord_bridge_domain }} + domain: {{ matrix_appservice_discord_bridge_domain|to_json }} # This should be your publically facing URL because Discord may use it to # fetch media from the media store. - homeserverUrl: {{ matrix_appservice_discord_bridge_homeserverUrl }} + homeserverUrl: {{ matrix_appservice_discord_bridge_homeserverUrl|to_json }} # Interval at which to process users in the 'presence queue'. If you have # 5 users, one user will be processed every 500 milliseconds according to the # value below. This has a minimum value of 250. @@ -33,7 +33,7 @@ bridge: # Authentication configuration for the discord bot. auth: clientID: {{ matrix_appservice_discord_client_id|string|to_json }} - botToken: {{ matrix_appservice_discord_bot_token }} + botToken: {{ matrix_appservice_discord_bot_token|to_json }} # You must enable "Privileged Gateway Intents" in your bot settings on discord.com (e.g. https://discord.com/developers/applications/12345/bot) # for this to work usePrivilegedIntents: {{ matrix_appservice_discord_auth_usePrivilegedIntents|to_json }} diff --git a/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 b/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 index 412b4a3d..84dee801 100644 --- a/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 +++ b/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-appservice-discord -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-appservice-discord +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-discord 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-discord 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -35,8 +35,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-appservice-dis {{ matrix_appservice_discord_docker_image }} \ node /build/src/discordas.js -p 9005 -c /cfg/config.yaml -f /cfg/registration.yaml -ExecStop=-{{ matrix_host_command_docker }} kill matrix-appservice-discord -ExecStop=-{{ matrix_host_command_docker }} rm matrix-appservice-discord +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-discord 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-discord 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-appservice-discord diff --git a/roles/matrix-bridge-appservice-irc/defaults/main.yml b/roles/matrix-bridge-appservice-irc/defaults/main.yml index 0b671e76..ead4e8de 100644 --- a/roles/matrix-bridge-appservice-irc/defaults/main.yml +++ b/roles/matrix-bridge-appservice-irc/defaults/main.yml @@ -3,14 +3,18 @@ matrix_appservice_irc_enabled: true -matrix_appservice_irc_docker_image: "docker.io/matrixdotorg/matrix-appservice-irc:release-0.17.1" +matrix_appservice_irc_container_self_build: false +matrix_appservice_irc_docker_repo: "https://github.com/matrix-org/matrix-appservice-irc.git" +matrix_appservice_irc_docker_src_files_path: "{{ matrix_base_data_path }}/appservice-irc/docker-src" + +matrix_appservice_irc_docker_image: "docker.io/matrixdotorg/matrix-appservice-irc:release-0.23.0" matrix_appservice_irc_docker_image_force_pull: "{{ matrix_appservice_irc_docker_image.endswith(':latest') }}" matrix_appservice_irc_base_path: "{{ matrix_base_data_path }}/appservice-irc" matrix_appservice_irc_config_path: "{{ matrix_appservice_irc_base_path }}/config" matrix_appservice_irc_data_path: "{{ matrix_appservice_irc_base_path }}/data" -matrix_appservice_irc_homeserver_url: 'http://matrix-synapse:8008' +matrix_appservice_irc_homeserver_url: "{{ matrix_homeserver_container_url }}" matrix_appservice_irc_homeserver_media_url: 'https://{{ matrix_server_fqn_matrix }}' matrix_appservice_irc_homeserver_domain: '{{ matrix_domain }}' matrix_appservice_irc_homeserver_enablePresence: true diff --git a/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml b/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml index 00568c0d..09e1d4ba 100644 --- a/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml +++ b/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml @@ -2,15 +2,17 @@ - name: Ensure Appservice IRC paths exist file: - path: "{{ item }}" + path: "{{ item.path }}" state: directory mode: 0750 owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" with_items: - - "{{ matrix_appservice_irc_base_path }}" - - "{{ matrix_appservice_irc_config_path }}" - - "{{ matrix_appservice_irc_data_path }}" + - { path: "{{ matrix_appservice_irc_base_path }}", when: true } + - { path: "{{ matrix_appservice_irc_config_path }}", when: true } + - { path: "{{ matrix_appservice_irc_data_path }}", when: true } + - { path: "{{ matrix_appservice_irc_docker_src_files_path }}", when: "{{ matrix_appservice_irc_container_self_build }}" } + when: item.when|bool - name: Check if an old passkey file already exists stat: @@ -59,6 +61,26 @@ source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" force_source: "{{ matrix_appservice_irc_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_appservice_irc_docker_image_force_pull }}" + when: "matrix_appservice_irc_enabled|bool and not matrix_appservice_irc_container_self_build|bool" + +- name: Ensure matrix-appservice-irc repository is present when self-building + git: + repo: "{{ matrix_appservice_irc_docker_repo }}" + dest: "{{ matrix_appservice_irc_docker_src_files_path }}" + force: "yes" + register: matrix_appservice_irc_git_pull_results + when: "matrix_appservice_irc_enabled|bool and matrix_appservice_irc_container_self_build|bool" + +- name: Ensure matrix-appservice-irc Docker image is build + docker_image: + name: "{{ matrix_appservice_irc_docker_image }}" + source: build + force_source: yes + build: + dockerfile: Dockerfile + path: "{{ matrix_appservice_irc_docker_src_files_path }}" + pull: yes + when: "matrix_appservice_irc_enabled|bool and matrix_appservice_irc_container_self_build|bool and matrix_appservice_irc_git_pull_results.changed" - name: Ensure Matrix Appservice IRC config installed copy: diff --git a/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 b/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 index 2287a774..8650bd8d 100644 --- a/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 +++ b/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-appservice-irc -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-appservice-irc +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-irc 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-irc 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -36,8 +36,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-appservice-irc {{ matrix_appservice_irc_docker_image }} \ -c 'node app.js -c /config/config.yaml -f /config/registration.yaml -p 9999' -ExecStop=-{{ matrix_host_command_docker }} kill matrix-appservice-irc -ExecStop=-{{ matrix_host_command_docker }} rm matrix-appservice-irc +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-irc 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-irc 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-appservice-irc diff --git a/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml b/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml index 721a5d6b..703d3fab 100644 --- a/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml +++ b/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml @@ -2,7 +2,7 @@ - name: Ensure AppService Slack paths exist file: - path: "{{ item }}" + path: "{{ item.path }}" state: directory mode: 0750 owner: "{{ matrix_user_username }}" diff --git a/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 b/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 index bf7a12ed..21ba27ef 100644 --- a/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 +++ b/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-appservice-slack -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-appservice-slack +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-slack 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-slack 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -35,8 +35,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-appservice-sla {{ matrix_appservice_slack_docker_image }} \ node app.js -p {{matrix_appservice_slack_matrix_port}} -c /config/config.yaml -f /config/slack-registration.yaml -ExecStop=-{{ matrix_host_command_docker }} kill matrix-appservice-slack -ExecStop=-{{ matrix_host_command_docker }} rm matrix-appservice-slack +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-slack 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-slack 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-appservice-slack diff --git a/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 b/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 index 667cfd73..f27111b3 100644 --- a/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 +++ b/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-appservice-webhooks -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-appservice-webhooks +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-webhooks 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-webhooks 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -35,8 +35,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-appservice-web {{ matrix_appservice_webhooks_docker_image }} \ node index.js -p {{ matrix_appservice_webhooks_matrix_port }} -c /config/config.yaml -f /config/webhooks-registration.yaml -ExecStop=-{{ matrix_host_command_docker }} kill matrix-appservice-webhooks -ExecStop=-{{ matrix_host_command_docker }} rm matrix-appservice-webhooks +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-appservice-webhooks 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-appservice-webhooks 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-appservice-webhooks diff --git a/roles/matrix-bridge-mautrix-facebook/defaults/main.yml b/roles/matrix-bridge-mautrix-facebook/defaults/main.yml index 04a81c75..acd3ee58 100644 --- a/roles/matrix-bridge-mautrix-facebook/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-facebook/defaults/main.yml @@ -7,7 +7,7 @@ matrix_mautrix_facebook_container_image_self_build: false matrix_mautrix_facebook_container_image_self_build_repo: "https://github.com/tulir/mautrix-facebook.git" # See: https://mau.dev/tulir/mautrix-facebook/container_registry -matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:da1b4ec596e334325a1589e70829dea46e73064b" +matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:latest" matrix_mautrix_facebook_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_facebook_container_image_self_build else 'dock.mau.dev/' }}" matrix_mautrix_facebook_docker_image_force_pull: "{{ matrix_mautrix_facebook_docker_image.endswith(':latest') }}" @@ -16,7 +16,7 @@ matrix_mautrix_facebook_config_path: "{{ matrix_mautrix_facebook_base_path }}/co matrix_mautrix_facebook_data_path: "{{ matrix_mautrix_facebook_base_path }}/data" matrix_mautrix_facebook_docker_src_files_path: "{{ matrix_mautrix_facebook_base_path }}/docker-src" -matrix_mautrix_facebook_homeserver_address: 'http://matrix-synapse:8008' +matrix_mautrix_facebook_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mautrix_facebook_homeserver_domain: '{{ matrix_domain }}' matrix_mautrix_facebook_appservice_address: 'http://matrix-mautrix-facebook:29319' @@ -35,12 +35,15 @@ matrix_mautrix_facebook_homeserver_token: '' # Database-related configuration fields. # -# To use SQLite, stick to these defaults. +# To use SQLite: +# - change the engine (`matrix_mautrix_facebook_database_engine: 'sqlite'`) +# - change to the last bridge version that supported SQLite: +# `matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:da1b4ec596e334325a1589e70829dea46e73064b"` +# - plan your migration to Postgres, as this bridge does not support SQLite anymore (and neither will the playbook in the future). # # To use Postgres: -# - change the engine (`matrix_mautrix_facebook_database_engine: 'postgres'`) # - adjust your database credentials via the `matrix_mautrix_facebook_postgres_*` variables -matrix_mautrix_facebook_database_engine: 'sqlite' +matrix_mautrix_facebook_database_engine: 'postgres' matrix_mautrix_facebook_sqlite_database_path_local: "{{ matrix_mautrix_facebook_data_path }}/mautrix-facebook.db" matrix_mautrix_facebook_sqlite_database_path_in_container: "/data/mautrix-facebook.db" @@ -66,6 +69,8 @@ matrix_mautrix_facebook_login_shared_secret: '' matrix_mautrix_facebook_bridge_login_shared_secret_map: "{{ {matrix_mautrix_facebook_homeserver_domain: matrix_mautrix_facebook_login_shared_secret} if matrix_mautrix_facebook_login_shared_secret else {} }}" +matrix_mautrix_facebook_appservice_bot_username: facebookbot + matrix_mautrix_facebook_bridge_presence: true # Default configuration template which covers the generic use case. @@ -98,8 +103,11 @@ matrix_mautrix_facebook_registration_yaml: | users: - exclusive: true regex: '^@facebook_.+:{{ matrix_mautrix_facebook_homeserver_domain|regex_escape }}$' + - exclusive: true + regex: '^@{{ matrix_mautrix_facebook_appservice_bot_username|regex_escape }}:{{ matrix_mautrix_facebook_homeserver_domain|regex_escape }}$' url: {{ matrix_mautrix_facebook_appservice_address }} - sender_localpart: facebookbot + # See https://github.com/tulir/mautrix-signal/issues/43 + sender_localpart: _bot_{{ matrix_mautrix_facebook_appservice_bot_username }} rate_limited: false matrix_mautrix_facebook_registration: "{{ matrix_mautrix_facebook_registration_yaml|from_yaml }}" diff --git a/roles/matrix-bridge-mautrix-facebook/tasks/validate_config.yml b/roles/matrix-bridge-mautrix-facebook/tasks/validate_config.yml index dfbe072b..0879bad9 100644 --- a/roles/matrix-bridge-mautrix-facebook/tasks/validate_config.yml +++ b/roles/matrix-bridge-mautrix-facebook/tasks/validate_config.yml @@ -8,3 +8,24 @@ with_items: - "matrix_mautrix_facebook_appservice_token" - "matrix_mautrix_facebook_homeserver_token" + +- block: + - name: Fail if on SQLite, unless on the last version supporting SQLite + fail: + msg: >- + You're trying to use the mautrix-facebook bridge with an SQLite database. + Going forward, this bridge only supports Postgres. + To learn more about this, see our changelog: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#breaking-change-the-mautrix-facebook-bridge-now-requires-a-postgres-database + when: "not matrix_mautrix_facebook_docker_image.endswith(':da1b4ec596e334325a1589e70829dea46e73064b')" + + - name: Inject warning if still on SQLite + set_fact: + matrix_playbook_runtime_results: | + {{ + matrix_playbook_runtime_results|default([]) + + + [ + "NOTE: Your mautrix-facebook bridge setup is still on SQLite. Your bridge is not getting any updates and will likely stop working at some point. To learn more about this, see our changelog: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#breaking-change-the-mautrix-facebook-bridge-now-requires-a-postgres-database" + ] + }} + when: "matrix_mautrix_facebook_database_engine == 'sqlite'" diff --git a/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 index 09287362..628db713 100644 --- a/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 @@ -8,6 +8,10 @@ homeserver: # Whether or not to verify the SSL certificate of the homeserver. # Only applies if address starts with https:// verify_ssl: true + # Whether or not the homeserver supports asmux-specific endpoints, + # such as /_matrix/client/unstable/net.maunium.asmux/dms for atomically + # updating m.direct. + asmux: false # Application service host/registration related details # Changing these values requires regeneration of the registration. @@ -22,11 +26,7 @@ appservice: # Usually 1 is enough, but on high-traffic bridges you might need to increase this to avoid 413s max_body_size: 1 - # The full URI to the database. SQLite and Postgres are fully supported. - # Other DBMSes supported by SQLAlchemy may or may not work. - # Format examples: - # SQLite: sqlite:///filename.db - # Postgres: postgres://username:password@hostname/dbname + # The full URI to the database. Only Postgres is currently supported. database: {{ matrix_mautrix_facebook_appservice_database|to_json }} # Public part of web server for out-of-Matrix interaction with the bridge. @@ -38,20 +38,29 @@ appservice: # The base URL where the public-facing endpoints are available. The prefix is not added # implicitly. external: https://example.com/public + # Shared secret for integration managers such as mautrix-manager. + # If set to "generate", a random string will be generated on the next startup. + # If null, integration manager access to the API will not be possible. + shared_secret: generate # The unique ID of this appservice. id: facebook # Username of the appservice bot. - bot_username: facebookbot + bot_username: {{ matrix_mautrix_facebook_appservice_bot_username|to_json }} # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty # to leave display name/avatar as-is. bot_displayname: Facebook bridge bot - bot_avatar: mxc://maunium.net/ddtNPZSKMNqaUzqrHuWvUADv + bot_avatar: mxc://maunium.net/ygtkteZsXnGJLJHRchUwYWak # Authentication tokens for AS <-> HS communication. as_token: "{{ matrix_mautrix_facebook_appservice_token }}" hs_token: "{{ matrix_mautrix_facebook_homeserver_token }}" +# Prometheus telemetry config. Requires prometheus-client to be installed. +metrics: + enabled: false + listen_port: 8000 + # Bridge config bridge: # Localpart template of MXIDs for Facebook users. @@ -76,6 +85,7 @@ bridge: # "own_nickname" (user-specific!) displayname_preference: - name + - first_name # The prefix for commands. Only required in non-management rooms. command_prefix: "!fb" @@ -120,6 +130,18 @@ bridge: # Default to encryption, force-enable encryption in all portals the bridge creates # This will cause the bridge bot to be in private chats for the encryption to work properly. default: false + # Options for automatic key sharing. + key_sharing: + # Enable key sharing? If enabled, key requests for rooms where users are in will be fulfilled. + # You must use a client that supports requesting keys from other users to use this feature. + allow: false + # Require the requesting device to have a valid cross-signing signature? + # This doesn't require that the bridge has verified the device, only that the user has verified it. + # Not yet implemented. + require_cross_signing: false + # Require devices to be verified by the bridge? + # Verification by the bridge is not yet implemented. + require_verification: true # Whether or not the bridge should send a read receipt from the bridge bot when a message has # been sent to Facebook. delivery_receipts: false @@ -161,6 +183,10 @@ bridge: # Whether or not the bridge should try to "refresh" the connection if a normal reconnection # attempt fails. refresh_on_reconnection_fail: false + # Set this to true to tell the bridge to re-send m.bridge events to all rooms on the next run. + # This field will automatically be changed back to false after it, + # except if the config file is not writable. + resend_bridge_info: false # Permissions for using the bridge. # Permitted values: @@ -192,9 +218,7 @@ logging: loggers: mau: level: DEBUG - fbchat: - level: DEBUG - hbmqtt: + paho: level: INFO aiohttp: level: INFO diff --git a/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 b/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 index 52e28859..acd2c885 100644 --- a/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 +++ b/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 @@ -13,8 +13,13 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mautrix-facebook -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mautrix-facebook +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-facebook 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-facebook 2>/dev/null' + +# This bridge uses another mechanism for migrations now (migrations happen automatically during regular startup), +# so going forward, running this alembic stuff will not necessary. +# People who are upgrading from an older version of the bridge should go through this migration +# first though, so we're keeping it around for now. ExecStartPre={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-facebook-db \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ @@ -39,10 +44,10 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-facebo {{ arg }} \ {% endfor %} {{ matrix_mautrix_facebook_docker_image }} \ - python3 -m mautrix_facebook -c /config/config.yaml + python3 -m mautrix_facebook -c /config/config.yaml --no-update -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mautrix-facebook -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mautrix-facebook +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-facebook 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-facebook 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mautrix-facebook diff --git a/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml b/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml index 8dfee030..0ed519cd 100644 --- a/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml @@ -18,7 +18,7 @@ matrix_mautrix_hangouts_docker_src_files_path: "{{ matrix_mautrix_hangouts_base_ matrix_mautrix_hangouts_public_endpoint: '/mautrix-hangouts' -matrix_mautrix_hangouts_homeserver_address: 'http://matrix-synapse:8008' +matrix_mautrix_hangouts_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mautrix_hangouts_homeserver_domain: '{{ matrix_domain }}' matrix_mautrix_hangouts_appservice_address: 'http://matrix-mautrix-hangouts:8080' @@ -71,6 +71,8 @@ matrix_mautrix_hangouts_appservice_database: "{{ # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mautrix_hangouts_login_shared_secret: '' +matrix_mautrix_hangouts_appservice_bot_username: hangoutsbot + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # @@ -101,8 +103,11 @@ matrix_mautrix_hangouts_registration_yaml: | users: - exclusive: true regex: '^@hangouts_.+:{{ matrix_mautrix_hangouts_homeserver_domain|regex_escape }}$' + - exclusive: true + regex: '^@{{ matrix_mautrix_hangouts_appservice_bot_username|regex_escape }}:{{ matrix_mautrix_hangouts_homeserver_domain|regex_escape }}$' url: {{ matrix_mautrix_hangouts_appservice_address }} - sender_localpart: hangoutsbot + # See https://github.com/tulir/mautrix-signal/issues/43 + sender_localpart: _bot_{{ matrix_mautrix_hangouts_appservice_bot_username }} rate_limited: false matrix_mautrix_hangouts_registration: "{{ matrix_mautrix_hangouts_registration_yaml|from_yaml }}" diff --git a/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 index cc2ca90b..7ff7d539 100644 --- a/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 @@ -32,7 +32,7 @@ appservice: # The unique ID of this appservice. id: hangouts # Username of the appservice bot. - bot_username: hangoutsbot + bot_username: {{ matrix_mautrix_hangouts_appservice_bot_username|to_json }} # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty # to leave display name/avatar as-is. bot_displayname: Hangouts bridge bot diff --git a/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 b/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 index f6b16bea..60f0e055 100644 --- a/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 +++ b/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mautrix-hangouts matrix-mautrix-hangouts-db -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mautrix-hangouts matrix-mautrix-hangouts-db +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-hangouts matrix-mautrix-hangouts-db 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-hangouts matrix-mautrix-hangouts-db 2>/dev/null' ExecStartPre={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-hangouts-db \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ @@ -42,10 +42,10 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-hangou {{ arg }} \ {% endfor %} {{ matrix_mautrix_hangouts_docker_image }} \ - python3 -m mautrix_hangouts -c /config/config.yaml + python3 -m mautrix_hangouts -c /config/config.yaml --no-update -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mautrix-hangouts -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mautrix-hangouts +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-hangouts 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-hangouts 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mautrix-hangouts diff --git a/roles/matrix-bridge-mautrix-instagram/defaults/main.yml b/roles/matrix-bridge-mautrix-instagram/defaults/main.yml new file mode 100644 index 00000000..411ec7ed --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/defaults/main.yml @@ -0,0 +1,103 @@ +# mautrix-instagram is a Matrix <-> Instagram bridge +# See: https://github.com/tulir/mautrix-instagram + +matrix_mautrix_instagram_enabled: true + +matrix_mautrix_instagram_container_image_self_build: false +matrix_mautrix_instagram_container_image_self_build_repo: "https://github.com/tulir/mautrix-instagram.git" + +# See: https://mau.dev/tulir/mautrix-instagram/container_registry +matrix_mautrix_instagram_docker_image: "{{ matrix_mautrix_instagram_docker_image_name_prefix }}tulir/mautrix-instagram:latest" +matrix_mautrix_instagram_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_instagram_container_image_self_build else 'dock.mau.dev/' }}" +matrix_mautrix_instagram_docker_image_force_pull: "{{ matrix_mautrix_instagram_docker_image.endswith(':latest') }}" + +matrix_mautrix_instagram_base_path: "{{ matrix_base_data_path }}/mautrix-instagram" +matrix_mautrix_instagram_config_path: "{{ matrix_mautrix_instagram_base_path }}/config" +matrix_mautrix_instagram_data_path: "{{ matrix_mautrix_instagram_base_path }}/data" +matrix_mautrix_instagram_docker_src_files_path: "{{ matrix_mautrix_instagram_base_path }}/docker-src" + +matrix_mautrix_instagram_homeserver_address: 'http://matrix-synapse:8008' +matrix_mautrix_instagram_homeserver_domain: '{{ matrix_domain }}' +matrix_mautrix_instagram_appservice_address: 'http://matrix-mautrix-instagram:29330' + +# A list of extra arguments to pass to the container +matrix_mautrix_instagram_container_extra_arguments: [] + +# List of systemd services that matrix-mautrix-instagram.service depends on. +matrix_mautrix_instagram_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-mautrix-instagram.service wants +matrix_mautrix_instagram_systemd_wanted_services_list: [] + +matrix_mautrix_instagram_appservice_token: '' +matrix_mautrix_instagram_homeserver_token: '' + + +# Database-related configuration fields. +# +# To use Postgres: +# - adjust your database credentials via the `matrix_mautrix_instagram_postgres_*` variables +matrix_mautrix_instagram_database_engine: 'postgres' + +matrix_mautrix_instagram_database_username: 'matrix_mautrix_instagram' +matrix_mautrix_instagram_database_password: 'some-password' +matrix_mautrix_instagram_database_hostname: 'matrix-postgres' +matrix_mautrix_instagram_database_port: 5432 +matrix_mautrix_instagram_database_name: 'matrix_mautrix_instagram' + +matrix_mautrix_instagram_database_connection_string: 'postgres://{{ matrix_mautrix_instagram_database_username }}:{{ matrix_mautrix_instagram_database_password }}@{{ matrix_mautrix_instagram_database_hostname }}:{{ matrix_mautrix_instagram_database_port }}/{{ matrix_mautrix_instagram_database_name }}' + +matrix_mautrix_instagram_appservice_database: "{{ + { + 'postgres': matrix_mautrix_instagram_database_connection_string, + }[matrix_mautrix_instagram_database_engine] +}}" + + +# Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). +matrix_mautrix_instagram_login_shared_secret: '' + +matrix_mautrix_instagram_bridge_login_shared_secret_map: "{{ {matrix_mautrix_instagram_homeserver_domain: matrix_mautrix_instagram_login_shared_secret} if matrix_mautrix_instagram_login_shared_secret else {} }}" + +matrix_mautrix_instagram_appservice_bot_username: instagrambot + +matrix_mautrix_instagram_bridge_presence: true + +# Default configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_mautrix_instagram_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_mautrix_instagram_configuration_yaml: "{{ lookup('template', 'templates/config.yaml.j2') }}" + +matrix_mautrix_instagram_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_mautrix_instagram_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_mautrix_instagram_configuration_yaml`. + +matrix_mautrix_instagram_configuration_extension: "{{ matrix_mautrix_instagram_configuration_extension_yaml|from_yaml if matrix_mautrix_instagram_configuration_extension_yaml|from_yaml is mapping else {} }}" + +# Holds the final configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_mautrix_instagram_configuration_yaml`. +matrix_mautrix_instagram_configuration: "{{ matrix_mautrix_instagram_configuration_yaml|from_yaml|combine(matrix_mautrix_instagram_configuration_extension, recursive=True) }}" + +matrix_mautrix_instagram_registration_yaml: | + id: instagram + as_token: "{{ matrix_mautrix_instagram_appservice_token }}" + hs_token: "{{ matrix_mautrix_instagram_homeserver_token }}" + namespaces: + users: + - exclusive: true + regex: '^@instagram_.+:{{ matrix_mautrix_instagram_homeserver_domain|regex_escape }}$' + - exclusive: true + regex: '^@{{ matrix_mautrix_instagram_appservice_bot_username|regex_escape }}:{{ matrix_mautrix_instagram_homeserver_domain|regex_escape }}$' + url: {{ matrix_mautrix_instagram_appservice_address }} + # See https://github.com/tulir/mautrix-signal/issues/43 + sender_localpart: _bot_{{ matrix_mautrix_instagram_appservice_bot_username }} + rate_limited: false + +matrix_mautrix_instagram_registration: "{{ matrix_mautrix_instagram_registration_yaml|from_yaml }}" diff --git a/roles/matrix-bridge-mautrix-instagram/tasks/init.yml b/roles/matrix-bridge-mautrix-instagram/tasks/init.yml new file mode 100644 index 00000000..2b407358 --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/tasks/init.yml @@ -0,0 +1,23 @@ +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-mautrix-instagram.service'] }}" + when: matrix_mautrix_instagram_enabled|bool + +# If the matrix-synapse role is not used, these variables may not exist. +- set_fact: + matrix_synapse_container_extra_arguments: > + {{ matrix_synapse_container_extra_arguments|default([]) }} + + + ["--mount type=bind,src={{ matrix_mautrix_instagram_config_path }}/registration.yaml,dst=/matrix-mautrix-instagram-registration.yaml,ro"] + + matrix_synapse_app_service_config_files: > + {{ matrix_synapse_app_service_config_files|default([]) }} + + + {{ ["/matrix-mautrix-instagram-registration.yaml"] }} + when: matrix_mautrix_instagram_enabled|bool + +# ansible lower than 2.8, does not support docker_image build parameters +# for self buildig it is explicitly needed, so we rather fail here +- name: Fail if running on Ansible lower than 2.8 and trying self building + fail: + msg: "To self build Mautrix instagram image, you should usa ansible 2.8 or higher. E.g. pip contains such packages." + when: "ansible_version.major == 2 and ansible_version.minor < 8 and matrix_mautrix_instagram_container_image_self_build" diff --git a/roles/matrix-bridge-mautrix-instagram/tasks/main.yml b/roles/matrix-bridge-mautrix-instagram/tasks/main.yml new file mode 100644 index 00000000..7326e22d --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/tasks/main.yml @@ -0,0 +1,21 @@ +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/validate_config.yml" + when: "run_setup|bool and matrix_mautrix_instagram_enabled|bool" + tags: + - setup-all + - setup-mautrix-instagram + +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: "run_setup|bool and matrix_mautrix_instagram_enabled|bool" + tags: + - setup-all + - setup-mautrix-instagram + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: "run_setup|bool and not matrix_mautrix_instagram_enabled|bool" + tags: + - setup-all + - setup-mautrix-instagram diff --git a/roles/matrix-bridge-mautrix-instagram/tasks/setup_install.yml b/roles/matrix-bridge-mautrix-instagram/tasks/setup_install.yml new file mode 100644 index 00000000..b83deab3 --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/tasks/setup_install.yml @@ -0,0 +1,80 @@ +--- +# If the matrix-synapse role is not used, `matrix_synapse_role_executed` won't exist. +# We don't want to fail in such cases. +- name: Fail if matrix-synapse role already executed + fail: + msg: >- + The matrix-bridge-mautrix-instagram role needs to execute before the matrix-synapse role. + when: "matrix_synapse_role_executed|default(False)" + +- name: Ensure Mautrix instagram image is pulled + docker_image: + name: "{{ matrix_mautrix_instagram_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mautrix_instagram_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mautrix_instagram_docker_image_force_pull }}" + when: matrix_mautrix_instagram_enabled|bool and not matrix_mautrix_instagram_container_image_self_build + +- name: Ensure Mautrix instagram paths exist + file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - { path: "{{ matrix_mautrix_instagram_base_path }}", when: true } + - { path: "{{ matrix_mautrix_instagram_config_path }}", when: true } + - { path: "{{ matrix_mautrix_instagram_data_path }}", when: true } + - { + path: "{{ matrix_mautrix_instagram_docker_src_files_path }}", + when: "{{ matrix_mautrix_instagram_container_image_self_build }}", + } + when: item.when|bool + +- name: Ensure Mautrix instagram repository is present on self-build + git: + repo: "{{ matrix_mautrix_instagram_container_image_self_build_repo }}" + dest: "{{ matrix_mautrix_instagram_docker_src_files_path }}" + force: "yes" + register: matrix_mautrix_instagram_git_pull_results + when: "matrix_mautrix_instagram_enabled|bool and matrix_mautrix_instagram_container_image_self_build" + +- name: Ensure Mautrix instagram Docker image is built + docker_image: + name: "{{ matrix_mautrix_instagram_docker_image }}" + source: build + force_source: "{{ matrix_mautrix_instagram_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_mautrix_instagram_docker_src_files_path }}" + pull: yes + when: "matrix_mautrix_instagram_enabled|bool and matrix_mautrix_instagram_container_image_self_build|bool" + +- name: Ensure mautrix-instagram config.yaml installed + copy: + content: "{{ matrix_mautrix_instagram_configuration|to_nice_yaml }}" + dest: "{{ matrix_mautrix_instagram_config_path }}/config.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure mautrix-instagram registration.yaml installed + copy: + content: "{{ matrix_mautrix_instagram_registration|to_nice_yaml }}" + dest: "{{ matrix_mautrix_instagram_config_path }}/registration.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-mautrix-instagram.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-mautrix-instagram.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-mautrix-instagram.service" + mode: 0644 + register: matrix_mautrix_instagram_systemd_service_result + +- name: Ensure systemd reloaded after matrix-mautrix-instagram.service installation + service: + daemon_reload: yes + when: "matrix_mautrix_instagram_systemd_service_result.changed" diff --git a/roles/matrix-bridge-mautrix-instagram/tasks/setup_uninstall.yml b/roles/matrix-bridge-mautrix-instagram/tasks/setup_uninstall.yml new file mode 100644 index 00000000..c5c8a3e6 --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/tasks/setup_uninstall.yml @@ -0,0 +1,23 @@ +--- +- name: Check existence of matrix-mautrix-instagram service + stat: + path: "{{ matrix_systemd_path }}/matrix-mautrix-instagram.service" + register: matrix_mautrix_instagram_service_stat + +- name: Ensure matrix-mautrix-instagram is stopped + service: + name: matrix-mautrix-instagram + state: stopped + daemon_reload: yes + when: "matrix_mautrix_instagram_service_stat.stat.exists" + +- name: Ensure matrix-mautrix-instagram.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-mautrix-instagram.service" + state: absent + when: "matrix_mautrix_instagram_service_stat.stat.exists" + +- name: Ensure systemd reloaded after matrix-mautrix-instagram.service removal + service: + daemon_reload: yes + when: "matrix_mautrix_instagram_service_stat.stat.exists" diff --git a/roles/matrix-bridge-mautrix-instagram/tasks/validate_config.yml b/roles/matrix-bridge-mautrix-instagram/tasks/validate_config.yml new file mode 100644 index 00000000..24992ff5 --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/tasks/validate_config.yml @@ -0,0 +1,9 @@ +--- +- name: Fail if required settings not defined + fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - "matrix_mautrix_instagram_appservice_token" + - "matrix_mautrix_instagram_homeserver_token" diff --git a/roles/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 new file mode 100644 index 00000000..db57bd0d --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/templates/config.yaml.j2 @@ -0,0 +1,234 @@ +#jinja2: lstrip_blocks: "True" +# Homeserver details +homeserver: + # The address that this appservice can use to connect to the homeserver. + address: {{ matrix_mautrix_instagram_homeserver_address }} + # The domain of the homeserver (for MXIDs, etc). + domain: {{ matrix_mautrix_instagram_homeserver_domain }} + # Whether or not to verify the SSL certificate of the homeserver. + # Only applies if address starts with https:// + verify_ssl: true + # Whether or not the homeserver supports asmux-specific endpoints, + # such as /_matrix/client/unstable/net.maunium.asmux/dms for atomically + # updating m.direct. + asmux: false + +# Application service host/registration related details +# Changing these values requires regeneration of the registration. +appservice: + # The address that the homeserver can use to connect to this appservice. + address: {{ matrix_mautrix_instagram_appservice_address }} + # When using https:// the TLS certificate and key files for the address. + tls_cert: false + tls_key: false + + # The hostname and port where this appservice should listen. + hostname: 0.0.0.0 + port: 29330 + # The maximum body size of appservice API requests (from the homeserver) in mebibytes + # Usually 1 is enough, but on high-traffic bridges you might need to increase this to avoid 413s + max_body_size: 1 + + # The full URI to the database. Only Postgres is currently supported. + database: {{ matrix_mautrix_instagram_appservice_database|to_json }} + # Additional arguments for asyncpg.create_pool() + # https://magicstack.github.io/asyncpg/current/api/index.html#asyncpg.pool.create_pool + database_opts: + min_size: 5 + max_size: 10 + + # The unique ID of this appservice. + id: instagram + # Username of the appservice bot. + bot_username: {{ matrix_mautrix_instagram_appservice_bot_username|to_json }} + # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty + # to leave display name/avatar as-is. + bot_displayname: instagram bridge bot + bot_avatar: mxc://maunium.net/JxjlbZUlCPULEeHZSwleUXQv + + # Community ID for bridged users (changes registration file) and rooms. + # Must be created manually. + # + # Example: "+instagram:example.com". Set to false to disable. + community_id: false + + # Whether or not to receive ephemeral events via appservice transactions. + # Requires MSC2409 support (i.e. Synapse 1.22+). + # You should disable bridge -> sync_with_custom_puppets when this is enabled. + ephemeral_events: false + + # Authentication tokens for AS <-> HS communication. + as_token: "{{ matrix_mautrix_instagram_appservice_token }}" + hs_token: "{{ matrix_mautrix_instagram_homeserver_token }}" + +# Prometheus telemetry config. Requires prometheus-client to be installed. +metrics: + enabled: false + listen_port: 8000 + +instagram: + # Seed for generating devices. This is secret because the seed is used to generate + # device IDs, which can apparently be used to bypass two-factor authentication after + # logging out, because Instagram is insecure. + device_seed: generate + +# Bridge config +bridge: + # Localpart template of MXIDs for Instagram users. + # {userid} is replaced with the user ID of the Instagram user. + username_template: "instagram_{userid}" + # Displayname template for Instagram users. + # {displayname} is replaced with the display name of the Instagram user. + # {username} is replaced with the username of the Instagram user. + displayname_template: "{username} (Instagram)" + + # Maximum length of displayname + displayname_max_length: 100 + + # Maximum number of seconds since the last activity in a chat to automatically create portals. + portal_create_max_age: 86400 + # Maximum number of chats to fetch for startup sync + chat_sync_limit: 100 + # Whether or not to use /sync to get read receipts and typing notifications + # when double puppeting is enabled + sync_with_custom_puppets: true + # Whether or not to update the m.direct account data event when double puppeting is enabled. + # Note that updating the m.direct event is not atomic (except with mautrix-asmux) + # and is therefore prone to race conditions. + sync_direct_chat_list: false + # Allow using double puppeting from any server with a valid client .well-known file. + double_puppet_allow_discovery: false + # Servers to allow double puppeting from, even if double_puppet_allow_discovery is false. + double_puppet_server_map: {} + # example.com: https://example.com + # Allow using double puppeting from any server with a valid client .well-known file. + double_puppet_allow_discovery: false + # Shared secrets for https://github.com/devture/matrix-synapse-shared-secret-auth + # + # If set, custom puppets will be enabled automatically for local users + # instead of users having to find an access token and run `login-matrix` + # manually. + # If using this for other servers than the bridge's server, + # you must also set the URL in the double_puppet_server_map. + login_shared_secret_map: + {{ matrix_mautrix_instagram_bridge_login_shared_secret_map|to_json }} + # Whether or not to update avatars when syncing all contacts at startup. + update_avatar_initial_sync: true + # Whether or not created rooms should have federation enabled. + # If false, created portal rooms will never be federated. + federate_rooms: true + # Settings for backfilling messages from Instagram. + backfill: + # Whether or not the Instagram users of logged in Matrix users should be + # invited to private chats when backfilling history from Instagram. This is + # usually needed to prevent rate limits and to allow timestamp massaging. + invite_own_puppet: true + # Maximum number of messages to backfill initially. + # Set to 0 to disable backfilling when creating portal. + initial_limit: 0 + # Maximum number of messages to backfill if messages were missed while + # the bridge was disconnected. + # Set to 0 to disable backfilling missed messages. + missed_limit: 1000 + # If using double puppeting, should notifications be disabled + # while the initial backfill is in progress? + disable_notifications: false + periodic_reconnect: + # Interval in seconds in which to automatically reconnect all users. + # This can be used to automatically mitigate the bug where Instagram stops sending messages. + # Set to -1 to disable periodic reconnections entirely. + interval: -1 + # Whether or not the bridge should backfill chats when reconnecting. + resync: true + # Should even disconnected users be reconnected? + always: false + # End-to-bridge encryption support options. These require matrix-nio to be installed with pip + # and login_shared_secret to be configured in order to get a device for the bridge bot. + # + # Additionally, https://github.com/matrix-org/synapse/pull/5758 is required if using a normal + # application service. + encryption: + # Allow encryption, work in group chat rooms with e2ee enabled + allow: false + # Default to encryption, force-enable encryption in all portals the bridge creates + # This will cause the bridge bot to be in private chats for the encryption to work properly. + default: false + # Options for automatic key sharing. + key_sharing: + # Enable key sharing? If enabled, key requests for rooms where users are in will be fulfilled. + # You must use a client that supports requesting keys from other users to use this feature. + allow: false + # Require the requesting device to have a valid cross-signing signature? + # This doesn't require that the bridge has verified the device, only that the user has verified it. + # Not yet implemented. + require_cross_signing: false + # Require devices to be verified by the bridge? + # Verification by the bridge is not yet implemented. + require_verification: true + # Whether or not to explicitly set the avatar and room name for private + # chat portal rooms. This will be implicitly enabled if encryption.default is true. + private_chat_portal_meta: false + # Whether or not the bridge should send a read receipt from the bridge bot when a message has + # been sent to Instagram. + delivery_receipts: false + # Whether or not delivery errors should be reported as messages in the Matrix room. + delivery_error_reports: false + # Set this to true to tell the bridge to re-send m.bridge events to all rooms on the next run. + # This field will automatically be changed back to false after it, + # except if the config file is not writable. + resend_bridge_info: false + # Whether or not unimportant bridge notices should be sent to the user. + # (e.g. connected, disconnected but will retry) + unimportant_bridge_notices: true + + # The prefix for commands. Only required in non-management rooms. + command_prefix: "!ig" + # Permissions for using the bridge. + # Permitted values: + # user - Use the bridge with puppeting. + # admin - Use and administrate the bridge. + # Permitted keys: + # * - All Matrix users + # domain - All users on that homeserver + # mxid - Specific user + permissions: + "{{ matrix_mautrix_instagram_homeserver_domain }}": user + # Provisioning API part of the web server for automated portal creation and fetching information. + # Used by things like mautrix-manager (https://github.com/tulir/mautrix-manager). + provisioning: + # Whether or not the provisioning API should be enabled. + enabled: true + # The prefix to use in the provisioning API endpoints. + prefix: /_matrix/provision/v1 + # The shared secret to authorize users of the API. + # Set to "generate" to generate and save a new token. + shared_secret: generate + +# Python logging configuration. +# +# See section 16.7.2 of the Python documentation for more info: +# https://docs.python.org/3.6/library/logging.config.html#configuration-dictionary-schema +logging: + version: 1 + formatters: + colored: + (): mautrix_instagram.util.ColorFormatter + format: "[%(asctime)s] [%(levelname)s@%(name)s] %(message)s" + normal: + format: "[%(asctime)s] [%(levelname)s@%(name)s] %(message)s" + handlers: + console: + class: logging.StreamHandler + formatter: colored + loggers: + mau: + level: DEBUG + mauigpapi: + level: DEBUG + paho: + level: INFO + aiohttp: + level: INFO + root: + level: DEBUG + handlers: [console] diff --git a/roles/matrix-bridge-mautrix-instagram/templates/systemd/matrix-mautrix-instagram.service.j2 b/roles/matrix-bridge-mautrix-instagram/templates/systemd/matrix-mautrix-instagram.service.j2 new file mode 100644 index 00000000..33a5bab3 --- /dev/null +++ b/roles/matrix-bridge-mautrix-instagram/templates/systemd/matrix-mautrix-instagram.service.j2 @@ -0,0 +1,42 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Mautrix Instagram bridge +{% for service in matrix_mautrix_instagram_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_mautrix_instagram_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-instagram 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-instagram 2>/dev/null' + +# Intentional delay, so that the homeserver (we likely depend on) can manage to start. +ExecStartPre={{ matrix_host_command_sleep }} 5 + +ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-instagram \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --network={{ matrix_docker_network }} \ + -v {{ matrix_mautrix_instagram_config_path }}:/config:z \ + -v {{ matrix_mautrix_instagram_data_path }}:/data:z \ + {% for arg in matrix_mautrix_instagram_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_mautrix_instagram_docker_image }} \ + python3 -m mautrix_instagram -c /config/config.yaml --no-update + +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-instagram 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-instagram 2>/dev/null' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-mautrix-instagram + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-bridge-mautrix-signal/defaults/main.yml b/roles/matrix-bridge-mautrix-signal/defaults/main.yml index 962140c9..aaa0a166 100644 --- a/roles/matrix-bridge-mautrix-signal/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-signal/defaults/main.yml @@ -43,6 +43,8 @@ matrix_mautrix_signal_daemon_systemd_wanted_services_list: [] matrix_mautrix_signal_appservice_token: '' matrix_mautrix_signal_homeserver_token: '' +matrix_mautrix_signal_appservice_bot_username: signalbot + # Database-related configuration fields # # This bridge only supports postgres. @@ -59,7 +61,7 @@ matrix_mautrix_signal_database_connection_string: 'postgres://{{ matrix_mautrix_ matrix_mautrix_signal_appservice_database: "{{ { - 'postgres': matrix_mautrix_facebook_database_connection_string, + 'postgres': matrix_mautrix_signal_database_connection_string, }[matrix_mautrix_signal_database_engine] }}" diff --git a/roles/matrix-bridge-mautrix-signal/tasks/setup_install.yml b/roles/matrix-bridge-mautrix-signal/tasks/setup_install.yml index 29555116..61c6adff 100644 --- a/roles/matrix-bridge-mautrix-signal/tasks/setup_install.yml +++ b/roles/matrix-bridge-mautrix-signal/tasks/setup_install.yml @@ -21,7 +21,7 @@ name: "{{ matrix_mautrix_signal_daemon_docker_image }}" source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" force_source: "{{ matrix_mautrix_signal_daemon_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mautrix_signal_docker_image_force_pull }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mautrix_signal_daemon_docker_image_force_pull }}" when: matrix_mautrix_signal_enabled|bool - name: Ensure Mautrix Signal paths exist @@ -35,6 +35,9 @@ - "{{ matrix_mautrix_signal_base_path }}" - "{{ matrix_mautrix_signal_config_path }}" - "{{ matrix_mautrix_signal_daemon_path }}" + - "{{ matrix_mautrix_signal_daemon_path }}/avatars" + - "{{ matrix_mautrix_signal_daemon_path }}/attachments" + - "{{ matrix_mautrix_signal_daemon_path }}/data" - name: Ensure mautrix-signal config.yaml installed copy: diff --git a/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 index 28fff6f0..dc2cff36 100644 --- a/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-signal/templates/config.yaml.j2 @@ -43,7 +43,7 @@ appservice: # The unique ID of this appservice. id: signal # Username of the appservice bot. - bot_username: signalbot + bot_username: {{ matrix_mautrix_signal_appservice_bot_username|to_json }} # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty # to leave display name/avatar as-is. bot_displayname: Signal bridge bot diff --git a/roles/matrix-bridge-mautrix-signal/templates/registration.yaml.j2 b/roles/matrix-bridge-mautrix-signal/templates/registration.yaml.j2 index 45cc5a0f..6891c2b5 100644 --- a/roles/matrix-bridge-mautrix-signal/templates/registration.yaml.j2 +++ b/roles/matrix-bridge-mautrix-signal/templates/registration.yaml.j2 @@ -6,9 +6,12 @@ namespaces: users: - exclusive: true regex: '^@signal_.+:{{ matrix_mautrix_signal_homeserver_domain|regex_escape }}$' + - exclusive: true + regex: '^@{{ matrix_mautrix_signal_appservice_bot_username|regex_escape }}:{{ matrix_mautrix_signal_homeserver_domain|regex_escape }}$' aliases: - exclusive: true regex: '^#signal_.+:{{ matrix_mautrix_signal_homeserver_domain|regex_escape }}$' url: {{ matrix_mautrix_signal_appservice_address }} -sender_localpart: signalbot +# See https://github.com/tulir/mautrix-signal/issues/43 +sender_localpart: _bot_{{ matrix_mautrix_signal_appservice_bot_username }} rate_limited: false diff --git a/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal-daemon.service.j2 b/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal-daemon.service.j2 index 35120317..6f128da3 100644 --- a/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal-daemon.service.j2 +++ b/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal-daemon.service.j2 @@ -15,21 +15,23 @@ Wants={{ service }} Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mautrix-signal-daemon -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mautrix-signal-daemon +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-signal-daemon 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-signal-daemon 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 +# We can't use `--read-only` for this bridge. ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-signal-daemon \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ --network={{ matrix_docker_network }} \ -v {{ matrix_mautrix_signal_daemon_path }}:/signald:z \ {{ matrix_mautrix_signal_daemon_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mautrix-signal-daemon -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mautrix-signal-daemon +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-signal-daemon 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-signal-daemon 2>/dev/null' Restart=always RestartSec=30 diff --git a/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal.service.j2 b/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal.service.j2 index 223f6dac..e3e02424 100644 --- a/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal.service.j2 +++ b/roles/matrix-bridge-mautrix-signal/templates/systemd/matrix-mautrix-signal.service.j2 @@ -14,8 +14,8 @@ Wants={{ service }} [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mautrix-signal -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mautrix-signal +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-signal 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-signal 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -23,19 +23,22 @@ ExecStartPre={{ matrix_host_command_sleep }} 5 ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-signal \ --log-driver=none \ --network={{ matrix_docker_network }} \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ {% if matrix_mautrix_signal_container_http_host_bind_port %} -p {{ matrix_mautrix_signal_container_http_host_bind_port }}:29328 \ {% endif %} -v {{ matrix_mautrix_signal_daemon_path }}:/signald:z \ - -v {{ matrix_mautrix_signal_config_path }}:/data:z \ + -v {{ matrix_mautrix_signal_config_path }}:/config:z \ {% for arg in matrix_mautrix_signal_container_extra_arguments %} {{ arg }} \ {% endfor %} {{ matrix_mautrix_signal_docker_image }} \ - python3 -m mautrix_signal -c /data/config.yaml + python3 -m mautrix_signal -c /config/config.yaml --no-update -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mautrix-signal -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mautrix-signal +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-signal 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-signal 2>/dev/null' Restart=always RestartSec=30 diff --git a/roles/matrix-bridge-mautrix-telegram/defaults/main.yml b/roles/matrix-bridge-mautrix-telegram/defaults/main.yml index 7e072b5a..b1f1b8fc 100644 --- a/roles/matrix-bridge-mautrix-telegram/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-telegram/defaults/main.yml @@ -25,11 +25,13 @@ matrix_mautrix_telegram_bot_token: disabled # Example: /741a0483-ba17-4682-9900-30bd7269f1cc matrix_mautrix_telegram_public_endpoint: '' -matrix_mautrix_telegram_homeserver_address: 'http://matrix-synapse:8008' +matrix_mautrix_telegram_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mautrix_telegram_homeserver_domain: '{{ matrix_domain }}' matrix_mautrix_telegram_appservice_address: 'http://matrix-mautrix-telegram:8080' matrix_mautrix_telegram_appservice_public_external: 'https://{{ matrix_server_fqn_matrix }}{{ matrix_mautrix_telegram_public_endpoint }}' +matrix_mautrix_telegram_appservice_bot_username: telegrambot + # Controls whether the matrix-mautrix-telegram container exposes its HTTP port (tcp/8080 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:9006"), or empty string to not expose. @@ -109,11 +111,14 @@ matrix_mautrix_telegram_registration_yaml: | users: - exclusive: true regex: '^@telegram_.+:{{ matrix_mautrix_telegram_homeserver_domain|regex_escape }}$' + - exclusive: true + regex: '^@{{ matrix_mautrix_telegram_appservice_bot_username|regex_escape }}:{{ matrix_mautrix_telegram_homeserver_domain|regex_escape }}$' aliases: - exclusive: true regex: '^#telegram_.+:{{ matrix_mautrix_telegram_homeserver_domain|regex_escape }}$' + # See https://github.com/tulir/mautrix-signal/issues/43 + sender_localpart: _bot_{{ matrix_mautrix_telegram_appservice_bot_username }} url: {{ matrix_mautrix_telegram_appservice_address }} - sender_localpart: telegrambot rate_limited: false matrix_mautrix_telegram_registration: "{{ matrix_mautrix_telegram_registration_yaml|from_yaml }}" diff --git a/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 index 52efba02..39a18462 100644 --- a/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 @@ -13,7 +13,7 @@ homeserver: # Changing these values requires regeneration of the registration. appservice: # The address that the homeserver can use to connect to this appservice. - address: {{ matrix_mautrix_telegram_appservice_address }} + address: {{ matrix_mautrix_telegram_appservice_address|to_json }} # The hostname and port where this appservice should listen. hostname: 0.0.0.0 @@ -36,10 +36,10 @@ appservice: # Whether or not the public-facing endpoints should be enabled. enabled: true # The prefix to use in the public-facing endpoints. - prefix: {{ matrix_mautrix_telegram_public_endpoint }} + prefix: {{ matrix_mautrix_telegram_public_endpoint|to_json }} # The base URL where the public-facing endpoints are available. The prefix is not added # implicitly. - external: {{ matrix_mautrix_telegram_appservice_public_external }} + external: {{ matrix_mautrix_telegram_appservice_public_external|to_json }} # Provisioning API part of the web server for automated portal creation and fetching information. # Used by things like Dimension (https://dimension.t2bot.io/). @@ -55,15 +55,15 @@ appservice: # The unique ID of this appservice. id: telegram # Username of the appservice bot. - bot_username: telegrambot + bot_username: {{ matrix_mautrix_telegram_appservice_bot_username|to_json }} # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty # to leave display name/avatar as-is. bot_displayname: Telegram bridge bot bot_avatar: mxc://maunium.net/tJCRmUyJDsgRNgqhOgoiHWbX # Authentication tokens for AS <-> HS communication. - as_token: "{{ matrix_mautrix_telegram_appservice_token }}" - hs_token: "{{ matrix_mautrix_telegram_homeserver_token }}" + as_token: {{ matrix_mautrix_telegram_appservice_token|to_json }} + hs_token: {{ matrix_mautrix_telegram_homeserver_token|to_json }} # Bridge config bridge: @@ -330,10 +330,10 @@ bridge: # Telegram config telegram: # Get your own API keys at https://my.telegram.org/apps - api_id: {{ matrix_mautrix_telegram_api_id }} - api_hash: {{ matrix_mautrix_telegram_api_hash }} + api_id: {{ matrix_mautrix_telegram_api_id|to_json }} + api_hash: {{ matrix_mautrix_telegram_api_hash|to_json }} # (Optional) Create your own bot at https://t.me/BotFather - bot_token: {{ matrix_mautrix_telegram_bot_token }} + bot_token: {{ matrix_mautrix_telegram_bot_token|to_json }} # Telethon connection options. connection: diff --git a/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 b/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 index ef4440bc..ae1ac675 100644 --- a/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 +++ b/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mautrix-telegram -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mautrix-telegram +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-telegram 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-telegram 2>/dev/null' ExecStartPre={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-telegram-db \ --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ @@ -42,10 +42,10 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-telegr {{ arg }} \ {% endfor %} {{ matrix_mautrix_telegram_docker_image }} \ - python3 -m mautrix_telegram -c /config/config.yaml + python3 -m mautrix_telegram -c /config/config.yaml --no-update -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mautrix-telegram -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mautrix-telegram +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-telegram 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-telegram 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mautrix-telegram diff --git a/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml index beda6d7d..581d47de 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -27,6 +27,8 @@ matrix_mautrix_whatsapp_systemd_wanted_services_list: [] matrix_mautrix_whatsapp_appservice_token: '' matrix_mautrix_whatsapp_homeserver_token: '' +matrix_mautrix_whatsapp_appservice_bot_username: whatsappbot + # Database-related configuration fields. # @@ -93,11 +95,14 @@ matrix_mautrix_whatsapp_registration_yaml: | url: {{ matrix_mautrix_whatsapp_appservice_address }} as_token: "{{ matrix_mautrix_whatsapp_appservice_token }}" hs_token: "{{ matrix_mautrix_whatsapp_homeserver_token }}" - sender_localpart: whatsappbot + # See https://github.com/tulir/mautrix-signal/issues/43 + sender_localpart: _bot_{{ matrix_mautrix_whatsapp_appservice_bot_username }} rate_limited: false namespaces: users: - regex: '^@whatsapp_[0-9]+:{{ matrix_mautrix_whatsapp_homeserver_domain|regex_escape }}$' exclusive: true + - exclusive: true + regex: '^@{{ matrix_mautrix_whatsapp_appservice_bot_username|regex_escape }}:{{ matrix_mautrix_whatsapp_homeserver_domain|regex_escape }}$' matrix_mautrix_whatsapp_registration: "{{ matrix_mautrix_whatsapp_registration_yaml|from_yaml }}" diff --git a/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 index 89216695..b3b1caf1 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 @@ -36,7 +36,7 @@ appservice: # Appservice bot details. bot: # Username of the appservice bot. - username: whatsappbot + username: {{ matrix_mautrix_whatsapp_appservice_bot_username|to_json }} # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty # to leave display name/avatar as-is. displayname: WhatsApp bridge bot diff --git a/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 b/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 index 22384fbd..4a492492 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 +++ b/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mautrix-whatsapp -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mautrix-whatsapp +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-whatsapp 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-whatsapp 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -33,8 +33,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-whatsa {{ matrix_mautrix_whatsapp_docker_image }} \ /usr/bin/mautrix-whatsapp -c /config/config.yaml -r /config/registration.yaml -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mautrix-whatsapp -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mautrix-whatsapp +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mautrix-whatsapp 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mautrix-whatsapp 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mautrix-whatsapp diff --git a/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml b/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml index 097a0916..8e278041 100644 --- a/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml @@ -22,7 +22,7 @@ matrix_mx_puppet_discord_docker_src_files_path: "{{ matrix_mx_puppet_discord_bas matrix_mx_puppet_discord_appservice_port: "8432" -matrix_mx_puppet_discord_homeserver_address: 'http://matrix-synapse:8008' +matrix_mx_puppet_discord_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mx_puppet_discord_homeserver_domain: '{{ matrix_domain }}' matrix_mx_puppet_discord_appservice_address: 'http://matrix-mx-puppet-discord:{{ matrix_mx_puppet_discord_appservice_port }}' diff --git a/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 index 28329e97..7c23fc61 100644 --- a/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 @@ -122,20 +122,4 @@ logging: lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files # Log files are rotated daily by default - files: - # Log file path - - file: "/data/bridge.log" - # Log level for this file - # Allowed values starting with most verbose: - # silly, debug, verbose, info, warn, error - level: info - # Date and time formatting - datePattern: YYYY-MM-DD - # Maximum number of logs to keep. - # This can be a number of files or number of days. - # If using days, add 'd' as a suffix - maxFiles: 14d - # Maximum size of the file after which it will rotate. This can be a - # number of bytes, or units of kb, mb, and gb. If using the units, add - # 'k', 'm', or 'g' as the suffix - maxSize: 50m + files: [] diff --git a/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 b/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 index 4f195ef6..6ffb87cd 100644 --- a/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 +++ b/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-discord -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-discord +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-discord 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-discord 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -33,8 +33,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mx-puppet-disc {% endfor %} {{ matrix_mx_puppet_discord_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-discord -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-discord +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-discord 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-discord 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mx-puppet-discord diff --git a/roles/matrix-bridge-mx-puppet-groupme/defaults/main.yml b/roles/matrix-bridge-mx-puppet-groupme/defaults/main.yml new file mode 100644 index 00000000..c0bafcf0 --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/defaults/main.yml @@ -0,0 +1,110 @@ +# Mx Puppet GroupMe is a Matrix <-> GroupMe bridge +# See: https://gitlab.com/robintown/mx-puppet-groupme + +matrix_mx_puppet_groupme_enabled: true + +matrix_mx_puppet_groupme_container_image_self_build: false +matrix_mx_puppet_groupme_container_image_self_build_repo: "https://gitlab.com/robintown/mx-puppet-groupme" + +# Controls whether the mx-puppet-groupme container exposes its HTTP port (tcp/8437 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:8437"), or empty string to not expose. +matrix_mx_puppet_groupme_container_http_host_bind_port: '' + +matrix_mx_puppet_groupme_docker_image: "{{ matrix_mx_puppet_groupme_docker_image_name_prefix }}xangelix/mx-puppet-groupme:latest" +matrix_mx_puppet_groupme_docker_image_name_prefix: "{{ 'localhost/' if matrix_mx_puppet_groupme_container_image_self_build else 'docker.io/' }}" +matrix_mx_puppet_groupme_docker_image_force_pull: "{{ matrix_mx_puppet_groupme_docker_image.endswith(':latest') }}" + +matrix_mx_puppet_groupme_base_path: "{{ matrix_base_data_path }}/mx-puppet-groupme" +matrix_mx_puppet_groupme_config_path: "{{ matrix_mx_puppet_groupme_base_path }}/config" +matrix_mx_puppet_groupme_data_path: "{{ matrix_mx_puppet_groupme_base_path }}/data" +matrix_mx_puppet_groupme_docker_src_files_path: "{{ matrix_mx_puppet_groupme_base_path }}/docker-src" + +matrix_mx_puppet_groupme_appservice_port: "8437" + +matrix_mx_puppet_groupme_homeserver_address: 'http://matrix-synapse:8008' +matrix_mx_puppet_groupme_homeserver_domain: '{{ matrix_domain }}' +matrix_mx_puppet_groupme_appservice_address: 'http://matrix-mx-puppet-groupme:{{ matrix_mx_puppet_groupme_appservice_port }}' + +matrix_mx_puppet_groupme_client_id: '' +matrix_mx_puppet_groupme_client_secret: '' + +# "@user:server.com" to allow specific user +# "@.*:yourserver.com" to allow users on a specific homeserver +# "@.*" to allow anyone +matrix_mx_puppet_groupme_provisioning_whitelist: + - "@.*:{{ matrix_domain|regex_escape }}" + +# Leave empty to disable blacklist +# "@user:server.com" disallow a specific user +# "@.*:yourserver.com" disallow users on a specific homeserver +matrix_mx_puppet_groupme_provisioning_blacklist: [] + +# A list of extra arguments to pass to the container +matrix_mx_puppet_groupme_container_extra_arguments: [] + +# List of systemd services that matrix-puppet-groupme.service depends on. +matrix_mx_puppet_groupme_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-puppet-groupme.service wants +matrix_mx_puppet_groupme_systemd_wanted_services_list: [] + +matrix_mx_puppet_groupme_appservice_token: '' +matrix_mx_puppet_groupme_homeserver_token: '' + +# Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). +matrix_mx_puppet_groupme_login_shared_secret: '' + +matrix_mx_puppet_groupme_database_engine: sqlite + +matrix_mx_puppet_groupme_sqlite_database_path_local: "{{ matrix_mx_puppet_groupme_data_path }}/database.db" +matrix_mx_puppet_groupme_sqlite_database_path_in_container: "/data/database.db" + +matrix_mx_puppet_groupme_database_username: matrix_mx_puppet_groupme +matrix_mx_puppet_groupme_database_password: ~ +matrix_mx_puppet_groupme_database_hostname: 'matrix-postgres' +matrix_mx_puppet_groupme_database_port: 5432 +matrix_mx_puppet_groupme_database_name: matrix_mx_puppet_groupme + +matrix_mx_puppet_groupme_database_connection_string: 'postgresql://{{ matrix_mx_puppet_groupme_database_username }}:{{ matrix_mx_puppet_groupme_database_password }}@{{ matrix_mx_puppet_groupme_database_hostname }}:{{ matrix_mx_puppet_groupme_database_port }}/{{ matrix_mx_puppet_groupme_database_name }}?sslmode=disable' + +# Default configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_mx_puppet_groupme_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_mx_puppet_groupme_configuration_yaml: "{{ lookup('template', 'templates/config.yaml.j2') }}" + +matrix_mx_puppet_groupme_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_mx_puppet_groupme_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_mx_puppet_groupme_configuration_yaml`. + +matrix_mx_puppet_groupme_configuration_extension: "{{ matrix_mx_puppet_groupme_configuration_extension_yaml|from_yaml if matrix_mx_puppet_groupme_configuration_extension_yaml|from_yaml is mapping else {} }}" + +# Holds the final configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_mx_puppet_groupme_configuration_yaml`. +matrix_mx_puppet_groupme_configuration: "{{ matrix_mx_puppet_groupme_configuration_yaml|from_yaml|combine(matrix_mx_puppet_groupme_configuration_extension, recursive=True) }}" + +matrix_mx_puppet_groupme_registration_yaml: | + as_token: "{{ matrix_mx_puppet_groupme_appservice_token }}" + hs_token: "{{ matrix_mx_puppet_groupme_homeserver_token }}" + id: groupme-puppet + namespaces: + users: + - exclusive: true + regex: '@_groupmepuppet_.*:{{ matrix_mx_puppet_groupme_homeserver_domain|regex_escape }}' + rooms: [] + aliases: + - exclusive: true + regex: '#_groupmepuppet_.*:{{ matrix_mx_puppet_groupme_homeserver_domain|regex_escape }}' + protocols: [] + rate_limited: false + sender_localpart: _groupmepuppet_bot + url: {{ matrix_mx_puppet_groupme_appservice_address }} + +matrix_mx_puppet_groupme_registration: "{{ matrix_mx_puppet_groupme_registration_yaml|from_yaml }}" diff --git a/roles/matrix-bridge-mx-puppet-groupme/tasks/init.yml b/roles/matrix-bridge-mx-puppet-groupme/tasks/init.yml new file mode 100644 index 00000000..1f00e8a5 --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/tasks/init.yml @@ -0,0 +1,23 @@ +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-mx-puppet-groupme.service'] }}" + when: matrix_mx_puppet_groupme_enabled|bool + +# If the matrix-synapse role is not used, these variables may not exist. +- set_fact: + matrix_synapse_container_extra_arguments: > + {{ matrix_synapse_container_extra_arguments|default([]) }} + + + ["--mount type=bind,src={{ matrix_mx_puppet_groupme_config_path }}/registration.yaml,dst=/matrix-mx-puppet-groupme-registration.yaml,ro"] + + matrix_synapse_app_service_config_files: > + {{ matrix_synapse_app_service_config_files|default([]) }} + + + {{ ["/matrix-mx-puppet-groupme-registration.yaml"] }} + when: matrix_mx_puppet_groupme_enabled|bool + +# ansible lower than 2.8, does not support docker_image build parameters +# for self buildig it is explicitly needed, so we rather fail here +- name: Fail if running on Ansible lower than 2.8 and trying self building + fail: + msg: "To self build Puppet Slack image, you should usa ansible 2.8 or higher. E.g. pip contains such packages." + when: "ansible_version.major == 2 and ansible_version.minor < 8 and matrix_mx_puppet_groupme_container_image_self_build" diff --git a/roles/matrix-bridge-mx-puppet-groupme/tasks/main.yml b/roles/matrix-bridge-mx-puppet-groupme/tasks/main.yml new file mode 100644 index 00000000..994e7e45 --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/tasks/main.yml @@ -0,0 +1,21 @@ +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/validate_config.yml" + when: "run_setup|bool and matrix_mx_puppet_groupme_enabled|bool" + tags: + - setup-all + - setup-mx-puppet-groupme + +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: "run_setup|bool and matrix_mx_puppet_groupme_enabled|bool" + tags: + - setup-all + - setup-mx-puppet-groupme + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: "run_setup|bool and not matrix_mx_puppet_groupme_enabled|bool" + tags: + - setup-all + - setup-mx-puppet-groupme diff --git a/roles/matrix-bridge-mx-puppet-groupme/tasks/setup_install.yml b/roles/matrix-bridge-mx-puppet-groupme/tasks/setup_install.yml new file mode 100644 index 00000000..58fe9485 --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/tasks/setup_install.yml @@ -0,0 +1,127 @@ +--- + +# If the matrix-synapse role is not used, `matrix_synapse_role_executed` won't exist. +# We don't want to fail in such cases. +- name: Fail if matrix-synapse role already executed + fail: + msg: >- + The matrix-bridge-mx-puppet-groupme role needs to execute before the matrix-synapse role. + when: "matrix_synapse_role_executed|default(False)" + +- name: Ensure MX Puppet Groupme paths exist + file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - { path: "{{ matrix_mx_puppet_groupme_base_path }}", when: true } + - { path: "{{ matrix_mx_puppet_groupme_config_path }}", when: true } + - { path: "{{ matrix_mx_puppet_groupme_data_path }}", when: true } + - { path: "{{ matrix_mx_puppet_groupme_docker_src_files_path }}", when: "{{ matrix_mx_puppet_groupme_container_image_self_build }}" } + when: matrix_mx_puppet_groupme_enabled|bool and item.when|bool + +- name: Check if an old database file already exists + stat: + path: "{{ matrix_mx_puppet_groupme_base_path }}/database.db" + register: matrix_mx_puppet_groupme_stat_database + +- name: (Data relocation) Ensure matrix-mx-puppet-groupme.service is stopped + service: + name: matrix-mx-puppet-groupme + state: stopped + daemon_reload: yes + failed_when: false + when: "matrix_mx_puppet_groupme_stat_database.stat.exists" + +- name: (Data relocation) Move mx-puppet-groupme database file to ./data directory + command: "mv {{ matrix_mx_puppet_groupme_base_path }}/database.db {{ matrix_mx_puppet_groupme_data_path }}/database.db" + when: "matrix_mx_puppet_groupme_stat_database.stat.exists" + +- set_fact: + matrix_mx_puppet_groupme_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mx_puppet_groupme_sqlite_database_path_local }}" + register: matrix_mx_puppet_groupme_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mx_puppet_groupme_sqlite_database_path_local }}" + dst: "{{ matrix_mx_puppet_groupme_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mx_puppet_groupme_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mx-puppet-groupme.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mx_puppet_groupme_requires_restart: true + when: "matrix_mx_puppet_groupme_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mx_puppet_groupme_database_engine == 'postgres'" + +- name: Ensure MX Puppet Groupme image is pulled + docker_image: + name: "{{ matrix_mx_puppet_groupme_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mx_puppet_groupme_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_groupme_docker_image_force_pull }}" + when: matrix_mx_puppet_groupme_enabled|bool and not matrix_mx_puppet_groupme_container_image_self_build + +- name: Ensure MX Puppet Groupme repository is present on self build + git: + repo: "{{ matrix_mx_puppet_groupme_container_image_self_build_repo }}" + dest: "{{ matrix_mx_puppet_groupme_docker_src_files_path }}" + force: "yes" + register: matrix_mx_puppet_groupme_git_pull_results + when: "matrix_mx_puppet_groupme_enabled|bool and matrix_mx_puppet_groupme_container_image_self_build" + +- name: Ensure MX Puppet Groupme Docker image is built + docker_image: + name: "{{ matrix_mx_puppet_groupme_docker_image }}" + source: build + force_source: "{{ matrix_mx_puppet_groupme_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_mx_puppet_groupme_docker_src_files_path }}" + pull: yes + when: "matrix_mx_puppet_groupme_enabled|bool and matrix_mx_puppet_groupme_container_image_self_build" + +- name: Ensure mx-puppet-groupme config.yaml installed + copy: + content: "{{ matrix_mx_puppet_groupme_configuration|to_nice_yaml }}" + dest: "{{ matrix_mx_puppet_groupme_config_path }}/config.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure mx-puppet-groupme groupme-registration.yaml installed + copy: + content: "{{ matrix_mx_puppet_groupme_registration|to_nice_yaml }}" + dest: "{{ matrix_mx_puppet_groupme_config_path }}/registration.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-mx-puppet-groupme.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-mx-puppet-groupme.service.j2" + dest: "/etc/systemd/system/matrix-mx-puppet-groupme.service" + mode: 0644 + register: matrix_mx_puppet_groupme_systemd_service_result + +- name: Ensure systemd reloaded after matrix-mx-puppet-groupme.service installation + service: + daemon_reload: yes + when: "matrix_mx_puppet_groupme_systemd_service_result.changed" + +- name: Ensure matrix-mx-puppet-groupme.service restarted, if necessary + service: + name: "matrix-mx-puppet-groupme.service" + state: restarted + when: "matrix_mx_puppet_groupme_requires_restart|bool" diff --git a/roles/matrix-bridge-mx-puppet-groupme/tasks/setup_uninstall.yml b/roles/matrix-bridge-mx-puppet-groupme/tasks/setup_uninstall.yml new file mode 100644 index 00000000..cc4fdfa5 --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/tasks/setup_uninstall.yml @@ -0,0 +1,24 @@ +--- + +- name: Check existence of matrix-mx-puppet-groupme service + stat: + path: "/etc/systemd/system/matrix-mx-puppet-groupme.service" + register: matrix_mx_puppet_groupme_service_stat + +- name: Ensure matrix-mx-puppet-groupme is stopped + service: + name: matrix-mx-puppet-groupme + state: stopped + daemon_reload: yes + when: "matrix_mx_puppet_groupme_service_stat.stat.exists" + +- name: Ensure matrix-mx-puppet-groupme.service doesn't exist + file: + path: "/etc/systemd/system/matrix-mx-puppet-groupme.service" + state: absent + when: "matrix_mx_puppet_groupme_service_stat.stat.exists" + +- name: Ensure systemd reloaded after matrix-mx-puppet-groupme.service removal + service: + daemon_reload: yes + when: "matrix_mx_puppet_groupme_service_stat.stat.exists" diff --git a/roles/matrix-bridge-mx-puppet-groupme/tasks/validate_config.yml b/roles/matrix-bridge-mx-puppet-groupme/tasks/validate_config.yml new file mode 100644 index 00000000..5c5463ce --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/tasks/validate_config.yml @@ -0,0 +1,10 @@ +--- + +- name: Fail if required settings not defined + fail: + msg: >- + You need to define a required configuration setting (`{{ item }}`). + when: "vars[item] == ''" + with_items: + - "matrix_mx_puppet_groupme_appservice_token" + - "matrix_mx_puppet_groupme_homeserver_token" diff --git a/roles/matrix-bridge-mx-puppet-groupme/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-groupme/templates/config.yaml.j2 new file mode 100644 index 00000000..a9ab7701 --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/templates/config.yaml.j2 @@ -0,0 +1,86 @@ +#jinja2: lstrip_blocks: "True" +bridge: + # Port to host the bridge on + # Used for communication between the homeserver and the bridge + port: {{ matrix_mx_puppet_groupme_appservice_port }} + # The host connections to the bridge's webserver are allowed from + bindAddress: 0.0.0.0 + # Public domain of the homeserver + domain: {{ matrix_mx_puppet_groupme_homeserver_domain }} + # Reachable URL of the Matrix homeserver + homeserverUrl: {{ matrix_mx_puppet_groupme_homeserver_address }} + {% if matrix_mx_puppet_groupme_login_shared_secret != '' %} + loginSharedSecretMap: + {{ matrix_domain }}: {{ matrix_mx_puppet_groupme_login_shared_secret }} + {% endif %} + # Display name of the bridge bot + displayname: GroupMe Puppet Bridge + # Optionally specify a different media URL used for the media store + # + # This is where GroupMe will download user profile pictures and media + # from + #mediaUrl: https://external-url.org + +presence: + # Bridge GroupMe online/offline status + enabled: true + # How often to send status to the homeserver in milliseconds + interval: 5000 + +provisioning: + # Regex of Matrix IDs allowed to use the puppet bridge + whitelist: {{ matrix_mx_puppet_groupme_provisioning_whitelist|to_json }} + # Allow a specific user + #- "@user:server\\.com" + # Allow users on a specific homeserver + #- "@.*:yourserver\\.com" + # Allow anyone + #- ".*" + # Regex of Matrix IDs forbidden from using the puppet bridge + #blacklist: + # Disallow a specific user + #- "@user:server\\.com" + # Disallow users on a specific homeserver + #- "@.*:yourserver\\.com" + blacklist: {{ matrix_mx_puppet_groupme_provisioning_blacklist|to_json }} + +relay: + # Regex of Matrix IDs who are allowed to use the bridge in relay mode. + # Relay mode is when a single GroupMe bot account relays messages of + # multiple Matrix users + # + # Same format as in provisioning + whitelist: {{ matrix_mx_puppet_groupme_provisioning_whitelist|to_json }} + blacklist: {{ matrix_mx_puppet_groupme_provisioning_blacklist|to_json }} + +selfService: + # Regex of Matrix IDs who are allowed to use bridge self-servicing (plumbed rooms) + # + # Same format as in provisioning + whitelist: {{ matrix_mx_puppet_groupme_provisioning_whitelist|to_json }} + blacklist: {{ matrix_mx_puppet_groupme_provisioning_blacklist|to_json }} + +database: +{% if matrix_mx_puppet_groupme_database_engine == 'postgres' %} + # Use Postgres as a database backend + # If set, will be used instead of SQLite3 + # Connection string to connect to the Postgres instance + # with username "user", password "pass", host "localhost" and database name "dbname". + # Modify each value as necessary + connString: {{ matrix_mx_puppet_groupme_database_connection_string|to_json }} +{% else %} + # Use SQLite3 as a database backend + # The name of the database file + filename: {{ matrix_mx_puppet_groupme_sqlite_database_path_in_container|to_json }} +{% endif %} + +logging: + # Log level of console output + # Allowed values starting with most verbose: + # silly, debug, verbose, info, warn, error + console: info + # Date and time formatting + lineDateFormat: MMM-D HH:mm:ss.SSS + # Logging files + # Log files are rotated daily by default + files: [] diff --git a/roles/matrix-bridge-mx-puppet-groupme/templates/systemd/matrix-mx-puppet-groupme.service.j2 b/roles/matrix-bridge-mx-puppet-groupme/templates/systemd/matrix-mx-puppet-groupme.service.j2 new file mode 100644 index 00000000..dabafd18 --- /dev/null +++ b/roles/matrix-bridge-mx-puppet-groupme/templates/systemd/matrix-mx-puppet-groupme.service.j2 @@ -0,0 +1,43 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Mx Puppet Groupme bridge +{% for service in matrix_mx_puppet_groupme_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_mx_puppet_groupme_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-groupme 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-groupme 2>/dev/null' + +# Intentional delay, so that the homeserver (we likely depend on) can manage to start. +ExecStartPre={{ matrix_host_command_sleep }} 5 + +ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mx-puppet-groupme \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --network={{ matrix_docker_network }} \ + -e CONFIG_PATH=/config/config.yaml \ + -e REGISTRATION_PATH=/config/registration.yaml \ + -v {{ matrix_mx_puppet_groupme_config_path }}:/config:z \ + -v {{ matrix_mx_puppet_groupme_data_path }}:/data:z \ + {% for arg in matrix_mx_puppet_groupme_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_mx_puppet_groupme_docker_image }} + +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-groupme 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-groupme 2>/dev/null' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-mx-puppet-groupme + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml b/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml index cd08c010..4c9fbd98 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml @@ -16,7 +16,7 @@ matrix_mx_puppet_instagram_data_path: "{{ matrix_mx_puppet_instagram_base_path } matrix_mx_puppet_instagram_docker_src_files_path: "{{ matrix_mx_puppet_instagram_base_path }}/docker-src" matrix_mx_puppet_instagram_appservice_port: "8440" -matrix_mx_puppet_instagram_homeserver_address: 'http://matrix-synapse:8008' +matrix_mx_puppet_instagram_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mx_puppet_instagram_homeserver_domain: '{{ matrix_domain }}' matrix_mx_puppet_instagram_appservice_address: 'http://matrix-mx-puppet-instagram:{{ matrix_mx_puppet_instagram_appservice_port }}' diff --git a/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 index b830da2b..1c4bb1bd 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 @@ -66,20 +66,4 @@ logging: lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files # Log files are rotated daily by default - files: - # Log file path - - file: "/data/bridge.log" - # Log level for this file - # Allowed values starting with most verbose: - # silly, debug, verbose, info, warn, error - level: info - # Date and time formatting - datePattern: YYYY-MM-DD - # Maximum number of logs to keep. - # This can be a number of files or number of days. - # If using days, add 'd' as a suffix - maxFiles: 14d - # Maximum size of the file after which it will rotate. This can be a - # number of bytes, or units of kb, mb, and gb. If using the units, add - # 'k', 'm', or 'g' as the suffix - maxSize: 50m + files: [] diff --git a/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 b/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 index 6eb28da0..965bb41c 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 +++ b/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-instagram -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-instagram +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-instagram 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-instagram 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -33,8 +33,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mx-puppet-inst {% endfor %} {{ matrix_mx_puppet_instagram_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-instagram -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-instagram +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-instagram 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-instagram 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mx-puppet-instagram diff --git a/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml b/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml index 83cd3dc5..53c8e379 100644 --- a/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml @@ -17,7 +17,7 @@ matrix_mx_puppet_skype_docker_src_files_path: "{{ matrix_mx_puppet_skype_base_pa matrix_mx_puppet_skype_appservice_port: "8438" -matrix_mx_puppet_skype_homeserver_address: 'http://matrix-synapse:8008' +matrix_mx_puppet_skype_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mx_puppet_skype_appservice_address: 'http://matrix-mx-puppet-skype:{{ matrix_mx_puppet_skype_appservice_port }}' # "@user:server.com" to allow specific user diff --git a/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 index d41d3a23..1d6d4828 100644 --- a/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 @@ -42,30 +42,7 @@ logging: lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files # Log files are rotated daily by default - files: - # Log file path - - file: "/data/bridge.log" - # Log level for this file - # Allowed values starting with most verbose: - # silly, debug, verbose, info, warn, error - level: info - # Date and time formatting - datePattern: YYYY-MM-DD - # Maximum number of logs to keep. - # This can be a number of files or number of days. - # If using days, add 'd' as a suffix - maxFiles: 14d - # Maximum size of the file after which it will rotate. This can be a - # number of bytes, or units of kb, mb, and gb. If using the units, add - # 'k', 'm', or 'g' as the suffix - maxSize: 50m - # Optionally enable/disable logging for certain modules - #disabled: - # - PresenceHandler - # - module: bot-sdk-MatrixLiteClient - # regex: /_matrix/client/r0/presence/ # this regex needs to match to disable the log - #enabled: - # - Store + files: [] database: {% if matrix_mx_puppet_skype_database_engine == 'postgres' %} diff --git a/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 b/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 index e61a369c..9a7986e4 100644 --- a/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 +++ b/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-skype -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-skype +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-skype 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-skype 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -33,8 +33,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mx-puppet-skyp {% endfor %} {{ matrix_mx_puppet_skype_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-skype -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-skype +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-skype 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-skype 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mx-puppet-skype diff --git a/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml b/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml index 70b98ece..b1fb7487 100644 --- a/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml @@ -22,7 +22,7 @@ matrix_mx_puppet_slack_docker_src_files_path: "{{ matrix_mx_puppet_slack_base_pa matrix_mx_puppet_slack_appservice_port: "8432" -matrix_mx_puppet_slack_homeserver_address: 'http://matrix-synapse:8008' +matrix_mx_puppet_slack_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mx_puppet_slack_homeserver_domain: '{{ matrix_domain }}' matrix_mx_puppet_slack_appservice_address: 'http://matrix-mx-puppet-slack:{{ matrix_mx_puppet_slack_appservice_port }}' diff --git a/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 index af6b5cb8..01714cb3 100644 --- a/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 @@ -80,20 +80,4 @@ logging: lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files # Log files are rotated daily by default - files: - # Log file path - - file: "/data/bridge.log" - # Log level for this file - # Allowed values starting with most verbose: - # silly, debug, verbose, info, warn, error - level: info - # Date and time formatting - datePattern: YYYY-MM-DD - # Maximum number of logs to keep. - # This can be a number of files or number of days. - # If using days, add 'd' as a suffix - maxFiles: 14d - # Maximum size of the file after which it will rotate. This can be a - # number of bytes, or units of kb, mb, and gb. If using the units, add - # 'k', 'm', or 'g' as the suffix - maxSize: 50m + files: [] diff --git a/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 b/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 index b564c3b3..973771b3 100644 --- a/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 +++ b/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-slack -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-slack +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-slack 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-slack 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -36,8 +36,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mx-puppet-slac {% endfor %} {{ matrix_mx_puppet_slack_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-slack -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-slack +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-slack 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-slack 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mx-puppet-slack diff --git a/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml b/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml index 15fa889f..c3ac977e 100644 --- a/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml @@ -22,7 +22,7 @@ matrix_mx_puppet_steam_docker_src_files_path: "{{ matrix_mx_puppet_steam_base_pa matrix_mx_puppet_steam_appservice_port: "8432" -matrix_mx_puppet_steam_homeserver_address: 'http://matrix-synapse:8008' +matrix_mx_puppet_steam_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mx_puppet_steam_homeserver_domain: '{{ matrix_domain }}' matrix_mx_puppet_steam_appservice_address: 'http://matrix-mx-puppet-steam:{{ matrix_mx_puppet_steam_appservice_port }}' diff --git a/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 index 149e08b6..fd59471d 100644 --- a/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 @@ -83,20 +83,4 @@ logging: lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files # Log files are rotated daily by default - files: - # Log file path - - file: "/data/bridge.log" - # Log level for this file - # Allowed values starting with most verbose: - # silly, debug, verbose, info, warn, error - level: info - # Date and time formatting - datePattern: YYYY-MM-DD - # Maximum number of logs to keep. - # This can be a number of files or number of days. - # If using days, add 'd' as a suffix - maxFiles: 14d - # Maximum size of the file after which it will rotate. This can be a - # number of bytes, or units of kb, mb, and gb. If using the units, add - # 'k', 'm', or 'g' as the suffix - maxSize: 50m + files: [] diff --git a/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 b/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 index 498b6ad3..0772872b 100644 --- a/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 +++ b/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-steam -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-steam +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-steam 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-steam 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -33,8 +33,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mx-puppet-stea {% endfor %} {{ matrix_mx_puppet_steam_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-steam -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-steam +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-steam 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-steam 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mx-puppet-steam diff --git a/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml b/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml index 28639fda..d8582e53 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml @@ -22,7 +22,7 @@ matrix_mx_puppet_twitter_docker_src_files_path: "{{ matrix_mx_puppet_twitter_bas matrix_mx_puppet_twitter_appservice_port: "8432" -matrix_mx_puppet_twitter_homeserver_address: 'http://matrix-synapse:8008' +matrix_mx_puppet_twitter_homeserver_address: "{{ matrix_homeserver_container_url }}" matrix_mx_puppet_twitter_homeserver_domain: '{{ matrix_domain }}' matrix_mx_puppet_twitter_appservice_address: 'http://matrix-mx-puppet-twitter:{{ matrix_mx_puppet_twitter_appservice_port }}' diff --git a/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 index bdecf1dc..1d269057 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 @@ -76,20 +76,4 @@ logging: lineDateFormat: MMM-D HH:mm:ss.SSS # Logging files # Log files are rotated daily by default - files: - # Log file path - - file: "/data/bridge.log" - # Log level for this file - # Allowed values starting with most verbose: - # silly, debug, verbose, info, warn, error - level: info - # Date and time formatting - datePattern: YYYY-MM-DD - # Maximum number of logs to keep. - # This can be a number of files or number of days. - # If using days, add 'd' as a suffix - maxFiles: 14d - # Maximum size of the file after which it will rotate. This can be a - # number of bytes, or units of kb, mb, and gb. If using the units, add - # 'k', 'm', or 'g' as the suffix - maxSize: 50m + files: [] diff --git a/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 b/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 index 77424bfa..7e1b1c32 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 +++ b/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-twitter -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-twitter +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-twitter 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-twitter 2>/dev/null' # Intentional delay, so that the homeserver (we likely depend on) can manage to start. ExecStartPre={{ matrix_host_command_sleep }} 5 @@ -36,8 +36,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mx-puppet-twit {% endfor %} {{ matrix_mx_puppet_twitter_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mx-puppet-twitter -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mx-puppet-twitter +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mx-puppet-twitter 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mx-puppet-twitter 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mx-puppet-twitter diff --git a/roles/matrix-client-element/defaults/main.yml b/roles/matrix-client-element/defaults/main.yml index d0297193..db2e7945 100644 --- a/roles/matrix-client-element/defaults/main.yml +++ b/roles/matrix-client-element/defaults/main.yml @@ -3,7 +3,7 @@ matrix_client_element_enabled: true matrix_client_element_container_image_self_build: false matrix_client_element_container_image_self_build_repo: "https://github.com/vector-im/riot-web.git" -matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:v1.7.16" +matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:v1.7.21" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else 'docker.io/' }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" @@ -59,6 +59,8 @@ matrix_client_element_branding_authHeaderLogoUrl: "{{ matrix_client_element_welc # URL to Wallpaper, shown in background of welcome page matrix_client_element_branding_welcomeBackgroundUrl: ~ +matrix_client_element_page_template_welcome_path: "{{ role_path }}/templates/welcome.html.j2" + # By default, there's no Element homepage (when logged in). If you wish to have one, # point this to a `home.html` template file on your local filesystem. matrix_client_element_embedded_pages_home_path: ~ diff --git a/roles/matrix-client-element/tasks/setup.yml b/roles/matrix-client-element/tasks/setup.yml index 3b542b14..c4ed0847 100644 --- a/roles/matrix-client-element/tasks/setup.yml +++ b/roles/matrix-client-element/tasks/setup.yml @@ -62,7 +62,7 @@ group: "{{ matrix_user_groupname }}" with_items: - {src: "{{ role_path }}/templates/nginx.conf.j2", name: "nginx.conf"} - - {src: "{{ role_path }}/templates/welcome.html.j2", name: "welcome.html"} + - {src: "{{ matrix_client_element_page_template_welcome_path }}", name: "welcome.html"} - {src: "{{ matrix_client_element_embedded_pages_home_path }}", name: "home.html"} when: "matrix_client_element_enabled|bool and item.src is not none" diff --git a/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 b/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 index f1f9eb3f..fe2a3a86 100644 --- a/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 +++ b/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 @@ -10,8 +10,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-client-element -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-client-element +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-client-element 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-client-element 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-client-element \ --log-driver=none \ @@ -35,8 +35,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-client-element {% endfor %} {{ matrix_client_element_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-client-element -ExecStop=-{{ matrix_host_command_docker }} rm matrix-client-element +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-client-element 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-client-element 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-client-element diff --git a/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 b/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 index cc9c4587..262e2e77 100644 --- a/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 +++ b/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 @@ -10,8 +10,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-corporal -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-corporal +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-corporal 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-corporal 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-corporal \ --log-driver=none \ @@ -34,8 +34,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-corporal \ {{ matrix_corporal_docker_image }} \ /matrix-corporal -config=/etc/matrix-corporal/config.json -ExecStop=-{{ matrix_host_command_docker }} kill matrix-corporal -ExecStop=-{{ matrix_host_command_docker }} rm matrix-corporal +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-corporal 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-corporal 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-corporal diff --git a/roles/matrix-coturn/tasks/setup_uninstall.yml b/roles/matrix-coturn/tasks/setup_uninstall.yml index 99a7080e..4674903f 100644 --- a/roles/matrix-coturn/tasks/setup_uninstall.yml +++ b/roles/matrix-coturn/tasks/setup_uninstall.yml @@ -41,7 +41,5 @@ path: "{{ matrix_coturn_base_path }}" state: absent -- name: Ensure coturn Docker image doesn't exist - docker_image: - name: "{{ matrix_coturn_docker_image }}" - state: absent +# Intentionally not removing the Docker image when uninstalling. +# We can't be sure it had been pulled by us in the first place. diff --git a/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 b/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 index f8550e36..930db7c1 100644 --- a/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 +++ b/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 @@ -10,8 +10,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-coturn -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-coturn +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-coturn 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-coturn 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-coturn \ --log-driver=none \ @@ -42,8 +42,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-coturn \ {{ matrix_coturn_docker_image }} \ -c /turnserver.conf -ExecStop=-{{ matrix_host_command_docker }} kill matrix-coturn -ExecStop=-{{ matrix_host_command_docker }} rm matrix-coturn +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-coturn 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-coturn 2>/dev/null' # This only reloads certificates (not other configuration). # See: https://github.com/coturn/coturn/pull/236 diff --git a/roles/matrix-dimension/templates/config.yaml.j2 b/roles/matrix-dimension/templates/config.yaml.j2 index 200871e7..39721d71 100644 --- a/roles/matrix-dimension/templates/config.yaml.j2 +++ b/roles/matrix-dimension/templates/config.yaml.j2 @@ -13,7 +13,7 @@ homeserver: # The URL that Dimension, go-neb, and other services provisioned by Dimension should # use to access the homeserver with. - clientServerUrl: "http://matrix-synapse:8008" + clientServerUrl: "{{ matrix_homeserver_container_url }}" # The URL that Dimension should use when trying to communicate with federated APIs on # the homeserver. If not supplied or left empty Dimension will try to resolve the address diff --git a/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 b/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 index 30d78d0d..e27a5558 100644 --- a/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 +++ b/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-dimension -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-dimension +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-dimension 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-dimension 2>/dev/null' # Fixup database ownership if it got changed somehow (during a server migration, etc.) {% if matrix_dimension_database_engine == 'sqlite' %} @@ -38,8 +38,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-dimension \ {% endfor %} {{ matrix_dimension_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-dimension -ExecStop=-{{ matrix_host_command_docker }} rm matrix-dimension +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-dimension 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-dimension 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-dimension diff --git a/roles/matrix-dynamic-dns/tasks/uninstall.yml b/roles/matrix-dynamic-dns/tasks/uninstall.yml index 98dca0e8..f3caba25 100644 --- a/roles/matrix-dynamic-dns/tasks/uninstall.yml +++ b/roles/matrix-dynamic-dns/tasks/uninstall.yml @@ -22,3 +22,6 @@ service: daemon_reload: yes when: "matrix_dynamic_dns_service_stat.stat.exists" + +# Intentionally not removing the Docker image when uninstalling. +# We can't be sure it had been pulled by us in the first place. diff --git a/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 b/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 index 8dc2443d..dfdd2f72 100644 --- a/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 +++ b/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-dynamic-dns -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-dynamic-dns +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-dynamic-dns 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-dynamic-dns 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-dynamic-dns \ --log-driver=none \ --network={{ matrix_docker_network }} \ @@ -26,8 +26,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-dynamic-dns \ {% endfor %} {{ matrix_dynamic_dns_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-dynamic-dns -ExecStop=-{{ matrix_host_command_docker }} rm matrix-dynamic-dns +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-dynamic-dns 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-dynamic-dns 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-dynamic-dns diff --git a/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 b/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 index 1b9d6642..c9226768 100644 --- a/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 +++ b/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 @@ -8,8 +8,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-email2matrix -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-email2matrix +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-email2matrix 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-email2matrix 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-email2matrix \ --log-driver=none \ @@ -24,8 +24,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-email2matrix \ {% endfor %} {{ matrix_email2matrix_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-email2matrix -ExecStop=-{{ matrix_host_command_docker }} rm matrix-email2matrix +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-email2matrix 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-email2matrix 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-email2matrix diff --git a/roles/matrix-etherpad/defaults/main.yml b/roles/matrix-etherpad/defaults/main.yml new file mode 100644 index 00000000..7c63fe03 --- /dev/null +++ b/roles/matrix-etherpad/defaults/main.yml @@ -0,0 +1,91 @@ +matrix_etherpad_enabled: false + +matrix_etherpad_base_path: "{{ matrix_base_data_path }}/etherpad" + +matrix_etherpad_docker_image: "docker.io/etherpad/etherpad:1.8.7" +matrix_etherpad_docker_image_force_pull: "{{ matrix_etherpad_docker_image.endswith(':latest') }}" + +# List of systemd services that matrix-etherpad.service depends on. +matrix_etherpad_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-etherpad.service wants +matrix_etherpad_systemd_wanted_services_list: [] + +# Container user has to be able to write to the source file directories until this bug is fixed: +# https://github.com/ether/etherpad-lite/issues/2683 +matrix_etherpad_user_uid: '5001' +matrix_etherpad_user_gid: '5001' + +# Controls whether the matrix-etherpad container exposes its HTTP port (tcp/9001 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:9001"), or empty string to not expose. +matrix_etherpad_container_http_host_bind_port: '' + +# A list of extra arguments to pass to the container +# +# We assume that a reverse proxy is used and tell the container to trust it +# Details: https://github.com/ether/etherpad-lite/blob/develop/doc/docker.md +matrix_etherpad_container_extra_arguments: [ + '--env TRUST_PROXY=true' +] + +matrix_etherpad_public_endpoint: '/etherpad' + +# By default, the Etherpad app can be accessed within the Dimension domain +matrix_etherpad_base_url: "https://{{ matrix_server_fqn_dimension }}{{ matrix_etherpad_public_endpoint }}" + +# Database-related configuration fields. +# +# Etherpad requires a dedicated database +matrix_etherpad_database_engine: 'postgres' + +matrix_etherpad_database_username: 'matrix_etherpad' +matrix_etherpad_database_password: 'some-password' +matrix_etherpad_database_hostname: 'matrix-postgres' +matrix_etherpad_database_port: 5432 +matrix_etherpad_database_name: 'matrix_etherpad' + +matrix_etherpad_database_connection_string: 'postgres://{{ matrix_etherpad_database_username }}:{{ matrix_etherpad_database_password }}@{{ matrix_etherpad_database_hostname }}:{{ matrix_etherpad_database_port }}/{{ matrix_etherpad_database_name }}' + +# Variables configuring the etherpad +matrix_etherpad_title: 'Etherpad' +matrix_etherpad_default_pad_text: | + Welcome to Etherpad! + + This pad text is synchronized as you type, so that everyone viewing this page sees the same text. This allows you to collaborate seamlessly on documents! + + Get involved with Etherpad at https://etherpad.org + +# Default Etherpad configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_etherpad_configuration_extension_json`) +# or completely replace this variable with your own template. +matrix_etherpad_configuration_default: "{{ lookup('template', 'templates/settings.json.j2') }}" + +# Your custom JSON configuration for Etherpad goes here. +# This configuration extends the default starting configuration (`matrix_etherpad_configuration_json`). +# +# You can override individual variables from the default configuration, or introduce new ones. +# +# If you need something more special, you can take full control by +# completely redefining `matrix_etherpad_configuration_json`. +# +# Example configuration extension follows: +# +# matrix_etherpad_configuration_extension_json: | +# { +# "loadTest": true, +# "commitRateLimiting": { +# "duration": 1, +# "points": 10 +# } +# } +# +matrix_etherpad_configuration_extension_json: '{}' + +matrix_etherpad_configuration_extension: "{{ matrix_etherpad_configuration_extension_json|from_json if matrix_etherpad_configuration_extension_json|from_json is mapping else {} }}" + +# Holds the final Etherpad configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_etherpad_configuration_json`. +matrix_etherpad_configuration: "{{ matrix_etherpad_configuration_default|combine(matrix_etherpad_configuration_extension, recursive=True) }}" diff --git a/roles/matrix-etherpad/tasks/init.yml b/roles/matrix-etherpad/tasks/init.yml new file mode 100644 index 00000000..081d4c23 --- /dev/null +++ b/roles/matrix-etherpad/tasks/init.yml @@ -0,0 +1,62 @@ +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-etherpad.service'] }}" + when: matrix_etherpad_enabled|bool + +- block: + - name: Fail if matrix-nginx-proxy role already executed + fail: + msg: >- + Trying to append Etherpad's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your plabook, + so that the matrix-nginx-proxy role would run after the matrix-etherpad role. + when: matrix_nginx_proxy_role_executed|default(False)|bool + + - name: Generate Etherpad proxying configuration for matrix-nginx-proxy + set_fact: + matrix_etherpad_matrix_nginx_proxy_configuration: | + rewrite ^{{ matrix_etherpad_public_endpoint }}$ $scheme://$server_name{{ matrix_etherpad_public_endpoint }}/ permanent; + + location {{ matrix_etherpad_public_endpoint }}/ { + {% if matrix_nginx_proxy_enabled|default(False) %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + proxy_pass http://matrix-etherpad:9001/; + {# These are proxy directives needed specifically by Etherpad #} + proxy_buffering off; + proxy_http_version 1.1; # recommended with keepalive connections + proxy_pass_header Server; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Proto $scheme; # for EP to set secure cookie flag when https is used + # WebSocket proxying - from http://nginx.org/en/docs/http/websocket.html + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + {% else %} + {# Generic configuration for use outside of our container setup #} + # A good guide for setting up your Etherpad behind nginx: + # https://docs.gandi.net/en/cloud/tutorials/etherpad_lite.html + proxy_pass http://127.0.0.1:9001/; + {% endif %} + } + + - name: Register Etherpad proxying configuration with matrix-nginx-proxy + set_fact: + matrix_nginx_proxy_proxy_dimension_additional_server_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_dimension_additional_server_configuration_blocks|default([]) + + + [matrix_etherpad_matrix_nginx_proxy_configuration] + }} + tags: + - always + when: matrix_etherpad_enabled|bool + +- name: Warn about reverse-proxying if matrix-nginx-proxy not used + debug: + msg: >- + NOTE: You've enabled the Etherpad tool but are not using the matrix-nginx-proxy + reverse proxy. + Please make sure that you're proxying the `{{ matrix_etherpad_public_endpoint }}` + URL endpoint to the matrix-etherpad container. + You can expose the container's port using the `matrix_etherpad_container_http_host_bind_port` variable. + when: "matrix_etherpad_enabled|bool and matrix_nginx_proxy_enabled is not defined" diff --git a/roles/matrix-etherpad/tasks/main.yml b/roles/matrix-etherpad/tasks/main.yml new file mode 100644 index 00000000..27548aaf --- /dev/null +++ b/roles/matrix-etherpad/tasks/main.yml @@ -0,0 +1,21 @@ +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: run_setup|bool and matrix_etherpad_enabled|bool + tags: + - setup-all + - setup-etherpad + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: run_setup|bool and not matrix_etherpad_enabled|bool + tags: + - setup-all + - setup-etherpad + +- import_tasks: "{{ role_path }}/tasks/validate_config.yml" + when: run_setup|bool and matrix_etherpad_enabled|bool + tags: + - setup-all + - setup-etherpad diff --git a/roles/matrix-etherpad/tasks/setup_install.yml b/roles/matrix-etherpad/tasks/setup_install.yml new file mode 100644 index 00000000..a93c28de --- /dev/null +++ b/roles/matrix-etherpad/tasks/setup_install.yml @@ -0,0 +1,36 @@ +--- + +- name: Ensure Etherpad base path exists + file: + path: "{{ matrix_etherpad_base_path }}" + state: directory + mode: 0770 + owner: "{{ matrix_etherpad_user_uid }}" + group: "{{ matrix_etherpad_user_gid }}" + +- name: Ensure Etherpad config installed + copy: + content: "{{ matrix_etherpad_configuration|to_nice_json }}" + dest: "{{ matrix_etherpad_base_path }}/settings.json" + mode: 0640 + owner: "{{ matrix_etherpad_user_uid }}" + group: "{{ matrix_etherpad_user_gid }}" + +- name: Ensure Etherpad image is pulled + docker_image: + name: "{{ matrix_etherpad_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_etherpad_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_etherpad_docker_image_force_pull }}" + +- name: Ensure matrix-etherpad.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-etherpad.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-etherpad.service" + mode: 0644 + register: matrix_etherpad_systemd_service_result + +- name: Ensure systemd reloaded after matrix-etherpad.service installation + service: + daemon_reload: yes + when: "matrix_etherpad_systemd_service_result.changed|bool" diff --git a/roles/matrix-etherpad/tasks/setup_uninstall.yml b/roles/matrix-etherpad/tasks/setup_uninstall.yml new file mode 100644 index 00000000..8f40f420 --- /dev/null +++ b/roles/matrix-etherpad/tasks/setup_uninstall.yml @@ -0,0 +1,35 @@ +--- + +- name: Check existence of matrix-etherpad service + stat: + path: "{{ matrix_systemd_path }}/matrix-etherpad.service" + register: matrix_etherpad_service_stat + +- name: Ensure matrix-etherpad is stopped + service: + name: matrix-etherpad + state: stopped + daemon_reload: yes + register: stopping_result + when: "matrix_etherpad_service_stat.stat.exists|bool" + +- name: Ensure matrix-etherpad.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-etherpad.service" + state: absent + when: "matrix_etherpad_service_stat.stat.exists|bool" + +- name: Ensure systemd reloaded after matrix-etherpad.service removal + service: + daemon_reload: yes + when: "matrix_etherpad_service_stat.stat.exists|bool" + +- name: Ensure Etherpad base directory doesn't exist + file: + path: "{{ matrix_etherpad_base_path }}" + state: absent + +- name: Ensure Etherpad Docker image doesn't exist + docker_image: + name: "{{ matrix_etherpad_docker_image }}" + state: absent diff --git a/roles/matrix-etherpad/tasks/validate_config.yml b/roles/matrix-etherpad/tasks/validate_config.yml new file mode 100644 index 00000000..c76dc3b5 --- /dev/null +++ b/roles/matrix-etherpad/tasks/validate_config.yml @@ -0,0 +1,11 @@ +- name: Fail if Etherpad is enabled without the Dimension integrations manager + fail: + msg: >- + To integrate Etherpad notes with Matrix rooms you need to set "matrix_dimension_enabled" to true + when: "not matrix_dimension_enabled|bool" + +- name: Fail if no database is configured for Etherpad + fail: + msg: >- + Etherpad requires a dedicated Postgres database. Please enable the built in one, or configure an external DB by redefining "matrix_etherpad_database_hostname" + when: matrix_etherpad_database_hostname == "matrix-postgres" and not matrix_postgres_enabled diff --git a/roles/matrix-etherpad/templates/settings.json.j2 b/roles/matrix-etherpad/templates/settings.json.j2 new file mode 100644 index 00000000..b3b87f43 --- /dev/null +++ b/roles/matrix-etherpad/templates/settings.json.j2 @@ -0,0 +1,102 @@ +{ + "title": {{ matrix_etherpad_title|to_json }}, + "favicon": "favicon.ico", + "skinName": "colibris", + "skinVariants": "super-light-toolbar super-light-editor light-background", + "ip": "::", + "port": 9001, + "showSettingsInAdminPage": true, + "dbType": {{ matrix_etherpad_database_engine|to_json }}, + "dbSettings": { + "database": {{ matrix_etherpad_database_name|to_json }}, + "host": {{ matrix_etherpad_database_hostname|to_json }}, + "password": {{ matrix_etherpad_database_password|to_json }}, + "port": {{ matrix_etherpad_database_port|to_json }}, + "user": {{ matrix_etherpad_database_username|to_json }} + }, + "defaultPadText" : {{ matrix_etherpad_default_pad_text|to_json }}, + "suppressErrorsInPadText": false, + "requireSession": false, + "editOnly": false, + "minify": true, + "maxAge": 21600, + "abiword": null, + "soffice": null, + "tidyHtml": null, + "allowUnknownFileEnds": true, + "requireAuthentication": false, + "requireAuthorization": false, + "trustProxy": true, + "cookie": { + "sameSite": "Lax" + }, + "disableIPlogging": true, + "automaticReconnectionTimeout": 0, + "scrollWhenFocusLineIsOutOfViewport": { + "percentage": { + "editionAboveViewport": 0, + "editionBelowViewport": 0 + }, + "duration": 0, + "scrollWhenCaretIsInTheLastLineOfViewport": false, + "percentageToScrollWhenUserPressesArrowUp": 0 + }, + "socketTransportProtocols" : ["xhr-polling", "jsonp-polling", "htmlfile"], + "loadTest": false, + "importExportRateLimiting": { + "windowMs": 90000, + "max": 10 + }, + "importMaxFileSize": 52428800, + "commitRateLimiting": { + "duration": 1, + "points": 10 + }, + "exposeVersion": false, + "padOptions": { + "noColors": false, + "showControls": true, + "showChat": false, + "showLineNumbers": true, + "useMonospaceFont": false, + "userName": false, + "userColor": false, + "rtl": false, + "alwaysShowChat": false, + "chatAndUsers": false, + "lang": "en-gb" + }, + "padShortcutEnabled" : { + "altF9": true, + "altC": true, + "cmdShift2": true, + "delete": true, + "return": true, + "esc": true, + "cmdS": true, + "tab": true, + "cmdZ": true, + "cmdY": true, + "cmdI": true, + "cmdB": true, + "cmdU": true, + "cmd5": true, + "cmdShiftL": true, + "cmdShiftN": true, + "cmdShift1": true, + "cmdShiftC": true, + "cmdH": true, + "ctrlHome": true, + "pageUp": true, + "pageDown": true + }, + "loglevel": "INFO", + "logconfig" : + { "appenders": [ + { "type": "console", + "layout": {"type": "messagePassThrough"} + } + ] + }, + "customLocaleStrings": {} +} diff --git a/roles/matrix-etherpad/templates/systemd/matrix-etherpad.service.j2 b/roles/matrix-etherpad/templates/systemd/matrix-etherpad.service.j2 new file mode 100644 index 00000000..b8a26664 --- /dev/null +++ b/roles/matrix-etherpad/templates/systemd/matrix-etherpad.service.j2 @@ -0,0 +1,44 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Etherpad +{% for service in matrix_etherpad_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_etherpad_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" +ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-etherpad +ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-etherpad + +ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-etherpad \ + --log-driver=none \ + --user={{ matrix_etherpad_user_uid }}:{{ matrix_etherpad_user_gid }} \ + --cap-drop=ALL \ + --network={{ matrix_docker_network }} \ + {% if matrix_etherpad_container_http_host_bind_port %} + -p {{ matrix_etherpad_container_http_host_bind_port }}:9001 \ + {% endif %} + --mount type=bind,src={{ matrix_etherpad_base_path }},dst=/data \ + {% for arg in matrix_etherpad_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_etherpad_docker_image }} \ + node --experimental-worker /opt/etherpad-lite/node_modules/ep_etherpad-lite/node/server.js \ + --settings /data/settings.json --credentials /data/credentials.json \ + --sessionkey /data/sessionkey.json --apikey /data/apijey.json + + +ExecStop=-{{ matrix_host_command_docker }} kill matrix-etherpad +ExecStop=-{{ matrix_host_command_docker }} rm matrix-etherpad +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-etherpad + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-grafana/defaults/main.yml b/roles/matrix-grafana/defaults/main.yml new file mode 100644 index 00000000..00ed947e --- /dev/null +++ b/roles/matrix-grafana/defaults/main.yml @@ -0,0 +1,47 @@ +# matrix-grafana is open source visualization and analytics software +# See: https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md + +matrix_grafana_enabled: false + +matrix_grafana_docker_image: "docker.io/grafana/grafana:7.4.0" +matrix_grafana_docker_image_force_pull: "{{ matrix_grafana_docker_image.endswith(':latest') }}" + +# Not conditional, because when someone disables metrics +# they might still want to look at the old existing data. +# So it would be silly to delete the dashboard in such case. +matrix_grafana_dashboard_download_urls: +- "https://raw.githubusercontent.com/matrix-org/synapse/master/contrib/grafana/synapse.json" +- "https://raw.githubusercontent.com/rfrail3/grafana-dashboards/master/prometheus/node-exporter-full.json" + +matrix_grafana_base_path: "{{ matrix_base_data_path }}/grafana" +matrix_grafana_config_path: "{{ matrix_grafana_base_path }}/config" +matrix_grafana_data_path: "{{ matrix_grafana_base_path }}/data" + +# Allow viewing Grafana without logging in +matrix_grafana_anonymous_access: false + +# specify organization name that should be used for unauthenticated users +# if you change this in the Grafana admin panel, this needs to be updated +# to match to keep anonymous logins working +matrix_grafana_anonymous_access_org_name: 'Main Org.' + + +# default admin credentials, you are asked to change these on first login +matrix_grafana_default_admin_user: admin +matrix_grafana_default_admin_password: admin + +# A list of extra arguments to pass to the container +matrix_grafana_container_extra_arguments: [] + +# List of systemd services that matrix-grafana.service depends on +matrix_grafana_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-grafana.service wants +matrix_grafana_systemd_wanted_services_list: [] + +# Controls whether the matrix-grafana container exposes its HTTP port (tcp/3000 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:3000"), or empty string to not expose. +matrix_grafana_container_http_host_bind_port: '' + + diff --git a/roles/matrix-grafana/tasks/init.yml b/roles/matrix-grafana/tasks/init.yml new file mode 100644 index 00000000..8a22e301 --- /dev/null +++ b/roles/matrix-grafana/tasks/init.yml @@ -0,0 +1,5 @@ +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-grafana.service'] }}" + when: matrix_grafana_enabled|bool + + diff --git a/roles/matrix-grafana/tasks/main.yml b/roles/matrix-grafana/tasks/main.yml new file mode 100644 index 00000000..fb16c394 --- /dev/null +++ b/roles/matrix-grafana/tasks/main.yml @@ -0,0 +1,14 @@ +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/validate_config.yml" + when: "run_setup|bool and matrix_grafana_enabled|bool" + tags: + - setup-all + - setup-grafana + +- import_tasks: "{{ role_path }}/tasks/setup.yml" + tags: + - setup-all + - setup-grafana diff --git a/roles/matrix-grafana/tasks/setup.yml b/roles/matrix-grafana/tasks/setup.yml new file mode 100644 index 00000000..581e6617 --- /dev/null +++ b/roles/matrix-grafana/tasks/setup.yml @@ -0,0 +1,115 @@ +--- + +# +# Tasks related to setting up matrix-grafana +# + +- name: Ensure matrix-grafana image is pulled + docker_image: + name: "{{ matrix_grafana_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_grafana_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_grafana_docker_image_force_pull }}" + when: "matrix_grafana_enabled|bool" + +- name: Ensure grafana paths exists + file: + path: "{{ item }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - "{{ matrix_grafana_base_path }}" + - "{{ matrix_grafana_config_path }}" + - "{{ matrix_grafana_config_path }}/provisioning" + - "{{ matrix_grafana_config_path }}/provisioning/datasources" + - "{{ matrix_grafana_config_path }}/provisioning/dashboards" + - "{{ matrix_grafana_config_path }}/dashboards" + - "{{ matrix_grafana_data_path }}" + when: matrix_grafana_enabled|bool + +- name: Ensure grafana.ini present + template: + src: "{{ role_path }}/templates/grafana.ini.j2" + dest: "{{ matrix_grafana_config_path }}/grafana.ini" + mode: 0440 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: matrix_grafana_enabled|bool + +- name: Ensure provisioning/datasources/default.yaml present + template: + src: "{{ role_path }}/templates/datasources.yaml.j2" + dest: "{{ matrix_grafana_config_path }}/provisioning/datasources/default.yaml" + mode: 0440 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: matrix_grafana_enabled|bool + +- name: Ensure provisioning/dashboards/default.yaml present + template: + src: "{{ role_path }}/templates/dashboards.yaml.j2" + dest: "{{ matrix_grafana_config_path }}/provisioning/dashboards/default.yaml" + mode: 0440 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: matrix_grafana_enabled|bool + +- name: Ensure dashboard(s) downloaded + get_url: + url: "{{ item }}" + dest: "{{ matrix_grafana_config_path }}/dashboards/" + force: true + mode: 0440 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: "{{ matrix_grafana_dashboard_download_urls }}" + when: matrix_grafana_enabled|bool + +- name: Ensure matrix-grafana.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-grafana.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-grafana.service" + mode: 0644 + register: matrix_grafana_systemd_service_result + when: matrix_grafana_enabled|bool + +- name: Ensure systemd reloaded after matrix-grafana.service installation + service: + daemon_reload: yes + when: "matrix_grafana_enabled|bool and matrix_grafana_systemd_service_result.changed" + +# +# Tasks related to getting rid of matrix-grafana (if it was previously enabled) +# + +- name: Check existence of matrix-grafana service + stat: + path: "{{ matrix_systemd_path }}/matrix-grafana.service" + register: matrix_grafana_service_stat + +- name: Ensure matrix-grafana is stopped + service: + name: matrix-grafana + state: stopped + daemon_reload: yes + register: stopping_result + when: "not matrix_grafana_enabled|bool and matrix_grafana_service_stat.stat.exists" + +- name: Ensure matrix-grafana.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-grafana.service" + state: absent + when: "not matrix_grafana_enabled|bool and matrix_grafana_service_stat.stat.exists" + +- name: Ensure systemd reloaded after matrix-grafana.service removal + service: + daemon_reload: yes + when: "not matrix_grafana_enabled|bool and matrix_grafana_service_stat.stat.exists" + +- name: Ensure matrix-grafana Docker image doesn't exist + docker_image: + name: "{{ matrix_grafana_docker_image }}" + state: absent + when: "not matrix_grafana_enabled|bool" diff --git a/roles/matrix-grafana/tasks/validate_config.yml b/roles/matrix-grafana/tasks/validate_config.yml new file mode 100644 index 00000000..63d4919a --- /dev/null +++ b/roles/matrix-grafana/tasks/validate_config.yml @@ -0,0 +1,7 @@ +--- + +- name: Fail if Prometheus not enabled + fail: + msg: > + You need to enable `matrix_prometheus_enabled` to use Prometheus as data source for Grafana. + when: "not matrix_prometheus_enabled" diff --git a/roles/matrix-grafana/templates/dashboards.yaml.j2 b/roles/matrix-grafana/templates/dashboards.yaml.j2 new file mode 100644 index 00000000..aae42ba2 --- /dev/null +++ b/roles/matrix-grafana/templates/dashboards.yaml.j2 @@ -0,0 +1,9 @@ +apiVersion: 1 + +providers: + - name: {{ matrix_server_fqn_matrix }} - Dashboards + folder: '' # The folder where to place the dashboards + type: file + allowUiUpdates: true + options: + path: /etc/grafana/dashboards diff --git a/roles/matrix-grafana/templates/datasources.yaml.j2 b/roles/matrix-grafana/templates/datasources.yaml.j2 new file mode 100644 index 00000000..6ccbe374 --- /dev/null +++ b/roles/matrix-grafana/templates/datasources.yaml.j2 @@ -0,0 +1,8 @@ +apiVersion: 1 + +datasources: + - name: {{ matrix_server_fqn_matrix }} - Prometheus + type: prometheus + # Access mode - proxy (server in the UI) or direct (browser in the UI). + access: proxy + url: http://matrix-prometheus:9090 diff --git a/roles/matrix-grafana/templates/grafana.ini.j2 b/roles/matrix-grafana/templates/grafana.ini.j2 new file mode 100644 index 00000000..694bf7d7 --- /dev/null +++ b/roles/matrix-grafana/templates/grafana.ini.j2 @@ -0,0 +1,20 @@ +[security] +# default admin user, created on startup +admin_user = {{ matrix_grafana_default_admin_user }} + +# default admin password, can be changed before first start of grafana, or in profile settings +admin_password = {{ matrix_grafana_default_admin_password }} + +[auth.anonymous] +# enable anonymous access +enabled = {{ matrix_grafana_anonymous_access }} + +# specify organization name that should be used for unauthenticated users +org_name = {{ matrix_grafana_anonymous_access_org_name }} + +[dashboards] +{% if matrix_synapse_metrics_enabled %} +default_home_dashboard_path = /etc/grafana/dashboards/synapse.json +{% else %} +default_home_dashboard_path = /etc/grafana/dashboards/node-exporter-full.json +{% endif %} diff --git a/roles/matrix-grafana/templates/systemd/matrix-grafana.service.j2 b/roles/matrix-grafana/templates/systemd/matrix-grafana.service.j2 new file mode 100644 index 00000000..a4f81e35 --- /dev/null +++ b/roles/matrix-grafana/templates/systemd/matrix-grafana.service.j2 @@ -0,0 +1,43 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=matrix-grafana +{% for service in matrix_grafana_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_grafana_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-grafana 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-grafana 2>/dev/null' + + +ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-grafana \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --network={{ matrix_docker_network }} \ + {% if matrix_grafana_container_http_host_bind_port %} + -p {{ matrix_grafana_container_http_host_bind_port }}:3000 \ + {% endif %} + -v {{ matrix_grafana_config_path }}:/etc/grafana:z \ + -v {{ matrix_grafana_data_path }}:/var/lib/grafana:z \ + {% for arg in matrix_grafana_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_grafana_docker_image }} + +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-grafana 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-grafana 2>/dev/null' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-grafana + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-jitsi/defaults/main.yml b/roles/matrix-jitsi/defaults/main.yml index 924198b4..028d9c19 100644 --- a/roles/matrix-jitsi/defaults/main.yml +++ b/roles/matrix-jitsi/defaults/main.yml @@ -67,6 +67,9 @@ matrix_jitsi_web_public_url: "https://{{ matrix_server_fqn_jitsi }}" # Addresses need to be prefixed with one of `stun:`, `turn:` or `turns:`. matrix_jitsi_web_stun_servers: ['stun:meet-jit-si-turnrelay.jitsi.net:443'] +# Controls whether Etherpad will be available within Jitsi +matrix_jitsi_etherpad_enabled: false + # Controls whether the matrix-jitsi-web container exposes its HTTP port (tcp/80 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:12080"), or empty string to not expose. diff --git a/roles/matrix-jitsi/tasks/setup_jitsi_jicofo.yml b/roles/matrix-jitsi/tasks/setup_jitsi_jicofo.yml index 63da7fcf..dd2a7bd2 100644 --- a/roles/matrix-jitsi/tasks/setup_jitsi_jicofo.yml +++ b/roles/matrix-jitsi/tasks/setup_jitsi_jicofo.yml @@ -89,8 +89,5 @@ state: absent when: "not matrix_jitsi_enabled|bool" -- name: Ensure jitsi-jicofo Docker image doesn't exist - docker_image: - name: "{{ matrix_jitsi_jicofo_docker_image }}" - state: absent - when: "not matrix_jitsi_enabled|bool" +# Intentionally not removing the Docker image when uninstalling. +# We can't be sure it had been pulled by us in the first place. diff --git a/roles/matrix-jitsi/tasks/setup_jitsi_jvb.yml b/roles/matrix-jitsi/tasks/setup_jitsi_jvb.yml index e4c7f277..b73426db 100644 --- a/roles/matrix-jitsi/tasks/setup_jitsi_jvb.yml +++ b/roles/matrix-jitsi/tasks/setup_jitsi_jvb.yml @@ -89,8 +89,5 @@ state: absent when: "not matrix_jitsi_enabled|bool" -- name: Ensure jitsi-jvb Docker image doesn't exist - docker_image: - name: "{{ matrix_jitsi_jvb_docker_image }}" - state: absent - when: "not matrix_jitsi_enabled|bool" +# Intentionally not removing the Docker image when uninstalling. +# We can't be sure it had been pulled by us in the first place. diff --git a/roles/matrix-jitsi/tasks/setup_jitsi_prosody.yml b/roles/matrix-jitsi/tasks/setup_jitsi_prosody.yml index 66299f64..fd051fda 100644 --- a/roles/matrix-jitsi/tasks/setup_jitsi_prosody.yml +++ b/roles/matrix-jitsi/tasks/setup_jitsi_prosody.yml @@ -80,8 +80,5 @@ state: absent when: "not matrix_jitsi_enabled|bool" -- name: Ensure jitsi-prosody Docker image doesn't exist - docker_image: - name: "{{ matrix_jitsi_prosody_docker_image }}" - state: absent - when: "not matrix_jitsi_enabled|bool" +# Intentionally not removing the Docker image when uninstalling. +# We can't be sure it had been pulled by us in the first place. diff --git a/roles/matrix-jitsi/tasks/setup_jitsi_web.yml b/roles/matrix-jitsi/tasks/setup_jitsi_web.yml index 3dd6f30c..2b8a2cd2 100644 --- a/roles/matrix-jitsi/tasks/setup_jitsi_web.yml +++ b/roles/matrix-jitsi/tasks/setup_jitsi_web.yml @@ -90,8 +90,6 @@ state: absent when: "not matrix_jitsi_enabled|bool" -- name: Ensure jitsi-web Docker image doesn't exist - docker_image: - name: "{{ matrix_jitsi_web_docker_image }}" - state: absent - when: "not matrix_jitsi_enabled|bool" +# Intentionally not removing the Docker image when uninstalling. +# We can't be sure it had been pulled by us in the first place. + diff --git a/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 b/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 index 3d093795..6ecafaa0 100644 --- a/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 +++ b/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 @@ -10,8 +10,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-jitsi-jicofo -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-jitsi-jicofo +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-jicofo 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-jicofo 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-jicofo \ --log-driver=none \ @@ -23,8 +23,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-jicofo \ {% endfor %} {{ matrix_jitsi_jicofo_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-jitsi-jicofo -ExecStop=-{{ matrix_host_command_docker }} rm matrix-jitsi-jicofo +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-jicofo 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-jicofo 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-jitsi-jicofo diff --git a/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 b/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 index c109b600..53c0c83a 100644 --- a/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 +++ b/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 @@ -10,8 +10,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-jitsi-jvb -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-jitsi-jvb +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-jvb 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-jvb 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-jvb \ --log-driver=none \ @@ -32,8 +32,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-jvb \ {% endfor %} {{ matrix_jitsi_jvb_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-jitsi-jvb -ExecStop=-{{ matrix_host_command_docker }} rm matrix-jitsi-jvb +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-jvb 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-jvb 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-jitsi-jvb diff --git a/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 b/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 index c1cd32bc..b3525a74 100644 --- a/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 +++ b/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 @@ -10,8 +10,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-jitsi-prosody -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-jitsi-prosody +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-prosody 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-prosody 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-prosody \ --log-driver=none \ @@ -24,8 +24,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-prosody {% endfor %} {{ matrix_jitsi_prosody_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-jitsi-prosody -ExecStop=-{{ matrix_host_command_docker }} rm matrix-jitsi-prosody +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-prosody 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-prosody 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-jitsi-prosody diff --git a/roles/matrix-jitsi/templates/web/custom-config.js.j2 b/roles/matrix-jitsi/templates/web/custom-config.js.j2 index 02316ca0..bbe85798 100644 --- a/roles/matrix-jitsi/templates/web/custom-config.js.j2 +++ b/roles/matrix-jitsi/templates/web/custom-config.js.j2 @@ -11,5 +11,8 @@ config.p2p.stunServers = [ ]; {% endif %} +{% if matrix_jitsi_etherpad_enabled %} +config.etherpad_base = {{ (matrix_jitsi_etherpad_base + '/p/') |to_json }} +{% endif %} {{ matrix_jitsi_web_custom_config_extension }} diff --git a/roles/matrix-jitsi/templates/web/env.j2 b/roles/matrix-jitsi/templates/web/env.j2 index 353a3d14..7b763a3c 100644 --- a/roles/matrix-jitsi/templates/web/env.j2 +++ b/roles/matrix-jitsi/templates/web/env.j2 @@ -37,4 +37,6 @@ RESOLUTION_WIDTH_MIN={{ matrix_jitsi_web_config_resolution_width_min }} START_AUDIO_MUTED={{ matrix_jitsi_web_config_start_audio_muted_after_nth_participant }} START_VIDEO_MUTED={{ matrix_jitsi_web_config_start_video_muted_after_nth_participant }} +ETHERPAD_URL_BASE={{ (matrix_jitsi_etherpad_base + '/') if matrix_jitsi_etherpad_enabled else ''}} + {{ matrix_jitsi_web_environment_variables_extension }} diff --git a/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 b/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 index 63535f91..6ae2074d 100644 --- a/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 +++ b/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 @@ -10,8 +10,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-jitsi-web -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-jitsi-web +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-web 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-web 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-web \ --log-driver=none \ @@ -27,8 +27,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-jitsi-web \ {% endfor %} {{ matrix_jitsi_web_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-jitsi-web -ExecStop=-{{ matrix_host_command_docker }} rm matrix-jitsi-web +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-jitsi-web 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-jitsi-web 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-jitsi-web diff --git a/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 b/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 index 697b5aba..c2adffc0 100644 --- a/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 +++ b/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-ma1sd -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-ma1sd +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-ma1sd 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-ma1sd 2>/dev/null' # ma1sd writes an SQLite shared library (libsqlitejdbc.so) to /tmp and executes it from there, # so /tmp needs to be mounted with an exec option. @@ -38,8 +38,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-ma1sd \ {% endfor %} {{ matrix_ma1sd_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-ma1sd -ExecStop=-{{ matrix_host_command_docker }} rm matrix-ma1sd +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-ma1sd 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-ma1sd 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-ma1sd diff --git a/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 b/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 index d773d698..14712935 100644 --- a/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 +++ b/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 @@ -8,8 +8,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-mailer -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-mailer +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mailer 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mailer 2>/dev/null' # --hostname gives us a friendlier hostname than the default. # The real hostname is passed via a `HOSTNAME` environment variable though. @@ -28,8 +28,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mailer \ {% endfor %} {{ matrix_mailer_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-mailer -ExecStop=-{{ matrix_host_command_docker }} rm matrix-mailer +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-mailer 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-mailer 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-mailer diff --git a/roles/matrix-nginx-proxy/defaults/main.yml b/roles/matrix-nginx-proxy/defaults/main.yml index 7c383383..d46d5480 100644 --- a/roles/matrix-nginx-proxy/defaults/main.yml +++ b/roles/matrix-nginx-proxy/defaults/main.yml @@ -99,6 +99,10 @@ matrix_nginx_proxy_access_log_enabled: true matrix_nginx_proxy_proxy_riot_compat_redirect_enabled: false matrix_nginx_proxy_proxy_riot_compat_redirect_hostname: "riot.{{ matrix_domain }}" +# Controls whether proxying the Synapse domain should be done. +matrix_nginx_proxy_proxy_synapse_enabled: false +matrix_nginx_proxy_proxy_synapse_hostname: "matrix-nginx-proxy" + # Controls whether proxying the Element domain should be done. matrix_nginx_proxy_proxy_element_enabled: false matrix_nginx_proxy_proxy_element_hostname: "{{ matrix_server_fqn_element }}" @@ -115,6 +119,10 @@ matrix_nginx_proxy_proxy_dimension_hostname: "{{ matrix_server_fqn_dimension }}" matrix_nginx_proxy_proxy_jitsi_enabled: false matrix_nginx_proxy_proxy_jitsi_hostname: "{{ matrix_server_fqn_jitsi }}" +# Controls whether proxying the grafana domain should be done. +matrix_nginx_proxy_proxy_grafana_enabled: false +matrix_nginx_proxy_proxy_grafana_hostname: "{{ matrix_server_fqn_grafana }}" + # Controls whether proxying for the matrix-corporal API (`/_matrix/corporal`) should be done (on the matrix domain) matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: false matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corporal:41081" @@ -146,8 +154,13 @@ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key: "" # The addresses where the Matrix Client API is. # Certain extensions (like matrix-corporal) may override this in order to capture all traffic. -matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "matrix-synapse:8008" -matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "127.0.0.1:8008" +matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "matrix-nginx-proxy:12080" +matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "127.0.0.1:12080" + +# The addresses where the Matrix Client API is, when using Synapse. +matrix_nginx_proxy_proxy_synapse_client_api_addr_with_container: "matrix-synapse:8008" +matrix_nginx_proxy_proxy_synapse_client_api_addr_sans_container: "127.0.0.1:8008" + # This needs to be equal or higher than the maximum upload size accepted by Synapse. matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb: 50 @@ -185,34 +198,44 @@ matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain: "" # Controls whether proxying for the Matrix Federation API should be done. matrix_nginx_proxy_proxy_matrix_federation_api_enabled: false -matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-synapse:8048" -matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "localhost:8048" +matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-nginx-proxy:12088" +matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "localhost:12088" matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb: "{{ (matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb | int) * 3 }}" matrix_nginx_proxy_proxy_matrix_federation_api_ssl_certificate: "{{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/fullchain.pem" matrix_nginx_proxy_proxy_matrix_federation_api_ssl_certificate_key: "{{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/privkey.pem" +# The addresses where the Federation API is, when using Synapse. +matrix_nginx_proxy_proxy_synapse_federation_api_addr_with_container: "matrix-synapse:8048" +matrix_nginx_proxy_proxy_synapse_federation_api_addr_sans_container: "localhost:8048" + # The tmpfs at /tmp needs to be large enough to handle multiple concurrent file uploads. matrix_nginx_proxy_tmp_directory_size_mb: "{{ (matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb | int) * 50 }}" -# A list of strings containing additional configuration blocks to add to the nginx http's server configuration. +# A list of strings containing additional configuration blocks to add to the nginx http's server configuration (nginx-http.conf). matrix_nginx_proxy_proxy_http_additional_server_configuration_blocks: [] -# A list of strings containing additional configuration blocks to add to the matrix synapse's server configuration. +# A list of strings containing additional configuration blocks to add to the base matrix server configuration (matrix-domain.conf). matrix_nginx_proxy_proxy_matrix_additional_server_configuration_blocks: [] -# A list of strings containing additional configuration blocks to add to Riot's server configuration. +# A list of strings containing additional configuration blocks to add to the synapse's server configuration (matrix-synapse.conf). +matrix_nginx_proxy_proxy_synapse_additional_server_configuration_blocks: [] + +# A list of strings containing additional configuration blocks to add to Riot's server configuration (matrix-riot-web.conf). matrix_nginx_proxy_proxy_riot_additional_server_configuration_blocks: [] -# A list of strings containing additional configuration blocks to add to Element's server configuration. +# A list of strings containing additional configuration blocks to add to Element's server configuration (matrix-client-element.conf). matrix_nginx_proxy_proxy_element_additional_server_configuration_blocks: [] -# A list of strings containing additional configuration blocks to add to Dimension's server configuration. +# A list of strings containing additional configuration blocks to add to Dimension's server configuration (matrix-dimension.conf). matrix_nginx_proxy_proxy_dimension_additional_server_configuration_blocks: [] -# A list of strings containing additional configuration blocks to add to Jitsi's server configuration. +# A list of strings containing additional configuration blocks to add to Jitsi's server configuration (matrix-jitsi.conf). matrix_nginx_proxy_proxy_jitsi_additional_server_configuration_blocks: [] -# A list of strings containing additional configuration blocks to add to the base domain server configuration. +# A list of strings containing additional configuration blocks to add to Grafana's server configuration (matrix-grafana.conf). +matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks: [] + +# A list of strings containing additional configuration blocks to add to the base domain server configuration (matrix-base-domain.conf). matrix_nginx_proxy_proxy_domain_additional_server_configuration_blocks: [] # Specifies the SSL configuration that should be used for the SSL protocols and ciphers @@ -290,12 +313,17 @@ matrix_ssl_retrieval_method: "lets-encrypt" matrix_ssl_architecture: "amd64" -# The list of domains that this role will obtain certificates for. -matrix_ssl_domains_to_obtain_certificates_for: [] +# The full list of domains that this role will obtain certificates for. +# This variable is likely redefined outside of the role, to include the domains that are necessary (depending on the services that are enabled). +# To add additional domain names, consider using `matrix_ssl_additional_domains_to_obtain_certificates_for` instead. +matrix_ssl_domains_to_obtain_certificates_for: "{{ matrix_ssl_additional_domains_to_obtain_certificates_for }}" + +# A list of additional domain names to obtain certificates for. +matrix_ssl_additional_domains_to_obtain_certificates_for: [] # Controls whether to obtain production or staging certificates from Let's Encrypt. matrix_ssl_lets_encrypt_staging: false -matrix_ssl_lets_encrypt_certbot_docker_image: "docker.io/certbot/certbot:{{ matrix_ssl_architecture }}-v1.10.1" +matrix_ssl_lets_encrypt_certbot_docker_image: "docker.io/certbot/certbot:{{ matrix_ssl_architecture }}-v1.11.0" matrix_ssl_lets_encrypt_certbot_docker_image_force_pull: "{{ matrix_ssl_lets_encrypt_certbot_docker_image.endswith(':latest') }}" matrix_ssl_lets_encrypt_certbot_standalone_http_port: 2402 matrix_ssl_lets_encrypt_support_email: ~ @@ -319,3 +347,13 @@ matrix_ssl_pre_obtaining_required_service_start_wait_time_seconds: 60 # nginx status page configurations. matrix_nginx_proxy_proxy_matrix_nginx_status_enabled: false matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses: ['{{ ansible_default_ipv4.address }}'] + + +# synapse worker activation and endpoint mappings +matrix_nginx_proxy_synapse_workers_enabled: false +matrix_nginx_proxy_synapse_workers_list: [] +matrix_nginx_proxy_synapse_generic_worker_client_server_locations: [] +matrix_nginx_proxy_synapse_generic_worker_federation_locations: [] +matrix_nginx_proxy_synapse_media_repository_locations: [] +matrix_nginx_proxy_synapse_user_dir_locations: [] +matrix_nginx_proxy_synapse_frontend_proxy_locations: [] diff --git a/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml b/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml index 90f0da73..233fe4a9 100644 --- a/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml +++ b/roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml @@ -45,12 +45,18 @@ mode: 0644 when: matrix_nginx_proxy_enabled|bool -- name: Ensure Matrix nginx-proxy configuration for matrix domain exists +- name: Ensure Matrix nginx-proxy configuration for matrix-synapse exists template: src: "{{ role_path }}/templates/nginx/conf.d/matrix-synapse.conf.j2" dest: "{{ matrix_nginx_proxy_confd_path }}/matrix-synapse.conf" mode: 0644 - when: matrix_nginx_proxy_proxy_matrix_enabled|bool + when: matrix_nginx_proxy_proxy_synapse_enabled|bool + +- name: Ensure Matrix nginx-proxy configuration for matrix-synapse deleted + file: + path: "{{ matrix_nginx_proxy_confd_path }}/matrix-synapse.conf" + state: absent + when: "not matrix_nginx_proxy_proxy_synapse_enabled|bool" - name: Ensure Matrix nginx-proxy configuration for Element domain exists template: @@ -80,6 +86,19 @@ mode: 0644 when: matrix_nginx_proxy_proxy_jitsi_enabled|bool +- name: Ensure Matrix nginx-proxy configuration for grafana domain exists + template: + src: "{{ role_path }}/templates/nginx/conf.d/matrix-grafana.conf.j2" + dest: "{{ matrix_nginx_proxy_confd_path }}/matrix-grafana.conf" + mode: 0644 + when: matrix_nginx_proxy_proxy_grafana_enabled|bool + +- name: Ensure Matrix nginx-proxy configuration for Matrix domain exists + template: + src: "{{ role_path }}/templates/nginx/conf.d/matrix-domain.conf.j2" + dest: "{{ matrix_nginx_proxy_confd_path }}/matrix-domain.conf" + mode: 0644 + - name: Ensure Matrix nginx-proxy data directory for base domain exists file: path: "{{ matrix_nginx_proxy_data_path }}/matrix-domain" @@ -100,8 +119,8 @@ - name: Ensure Matrix nginx-proxy configuration for base domain exists template: - src: "{{ role_path }}/templates/nginx/conf.d/matrix-domain.conf.j2" - dest: "{{ matrix_nginx_proxy_confd_path }}/matrix-domain.conf" + src: "{{ role_path }}/templates/nginx/conf.d/matrix-base-domain.conf.j2" + dest: "{{ matrix_nginx_proxy_confd_path }}/matrix-base-domain.conf" mode: 0644 when: matrix_nginx_proxy_base_domain_serving_enabled|bool @@ -161,7 +180,7 @@ - name: Ensure Matrix nginx-proxy configuration for matrix domain deleted file: - path: "{{ matrix_nginx_proxy_confd_path }}/matrix-synapse.conf" + path: "{{ matrix_nginx_proxy_confd_path }}/matrix-domain.conf" state: absent when: "not matrix_nginx_proxy_proxy_matrix_enabled|bool" @@ -183,6 +202,12 @@ state: absent when: "not matrix_nginx_proxy_proxy_jitsi_enabled|bool" +- name: Ensure Matrix nginx-proxy configuration for grafana domain deleted + file: + path: "{{ matrix_nginx_proxy_confd_path }}/matrix-grafana.conf" + state: absent + when: "not matrix_nginx_proxy_proxy_grafana_enabled|bool" + - name: Ensure Matrix nginx-proxy homepage for base domain deleted file: path: "{{ matrix_nginx_proxy_data_path }}/matrix-domain/index.html" @@ -191,7 +216,7 @@ - name: Ensure Matrix nginx-proxy configuration for base domain deleted file: - path: "{{ matrix_nginx_proxy_confd_path }}/matrix-domain.conf" + path: "{{ matrix_nginx_proxy_confd_path }}/matrix-base-domain.conf" state: absent when: "not matrix_nginx_proxy_base_domain_serving_enabled|bool" diff --git a/roles/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt.yml b/roles/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt.yml index b976923f..f0b14327 100644 --- a/roles/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt.yml +++ b/roles/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt.yml @@ -55,7 +55,7 @@ file: path: "{{ matrix_systemd_path }}/{{ item.name }}" state: absent - when: "{{ not item.applicable }}" + when: "not item.applicable|bool" with_items: "{{ matrix_ssl_renewal_systemd_units_list }}" - name: Ensure Let's Encrypt SSL renewal script removed diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-base-domain.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-base-domain.conf.j2 new file mode 100644 index 00000000..227747a5 --- /dev/null +++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-base-domain.conf.j2 @@ -0,0 +1,70 @@ +#jinja2: lstrip_blocks: "True" + +{% macro render_vhost_directives() %} + root /nginx-data/matrix-domain; + + gzip on; + gzip_types text/plain application/json; + {% for configuration_block in matrix_nginx_proxy_proxy_domain_additional_server_configuration_blocks %} + {{- configuration_block }} + {% endfor %} + + location /.well-known/matrix { + root {{ matrix_static_files_base_path }}; + {# + A somewhat long expires value is used to prevent outages + in case this is unreachable due to network failure. + #} + expires 4h; + default_type application/json; + add_header Access-Control-Allow-Origin *; + } +{% endmacro %} + +server { + listen {{ 8080 if matrix_nginx_proxy_enabled else 80 }}; + + server_name {{ matrix_nginx_proxy_base_domain_hostname }}; + server_tokens off; + + {% if matrix_nginx_proxy_https_enabled %} + location /.well-known/acme-challenge { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-certbot:8080"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://127.0.0.1:{{ matrix_ssl_lets_encrypt_certbot_standalone_http_port }}; + {% endif %} + } + + location / { + return 301 https://$http_host$request_uri; + } + {% else %} + {{ render_vhost_directives() }} + {% endif %} +} + +{% if matrix_nginx_proxy_https_enabled %} +server { + listen {{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; + listen [::]:{{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; + + server_name {{ matrix_nginx_proxy_base_domain_hostname }}; + server_tokens off; + + ssl_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_base_domain_hostname }}/fullchain.pem; + ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_base_domain_hostname }}/privkey.pem; + + ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }}; + {% if matrix_nginx_proxy_ssl_ciphers != '' %} + ssl_ciphers {{ matrix_nginx_proxy_ssl_ciphers }}; + {% endif %} + ssl_prefer_server_ciphers {{ matrix_nginx_proxy_ssl_prefer_server_ciphers }}; + + {{ render_vhost_directives() }} +} +{% endif %} diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 index 227747a5..2ab78a1b 100644 --- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 +++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2 @@ -1,31 +1,148 @@ #jinja2: lstrip_blocks: "True" +{% macro render_nginx_status_location_block(addresses) %} + {# Empty first line to make indentation prettier. #} + + location /nginx_status { + stub_status on; + access_log off; + {% for address in addresses %} + allow {{ address }}; + {% endfor %} + deny all; + } +{% endmacro %} -{% macro render_vhost_directives() %} - root /nginx-data/matrix-domain; +{% macro render_vhost_directives() %} gzip on; gzip_types text/plain application/json; - {% for configuration_block in matrix_nginx_proxy_proxy_domain_additional_server_configuration_blocks %} - {{- configuration_block }} - {% endfor %} location /.well-known/matrix { root {{ matrix_static_files_base_path }}; {# A somewhat long expires value is used to prevent outages - in case this is unreachable due to network failure. + in case this is unreachable due to network failure or + due to the base domain's server completely dying. #} expires 4h; default_type application/json; add_header Access-Control-Allow-Origin *; } + + {% if matrix_nginx_proxy_proxy_matrix_nginx_status_enabled %} + {{ render_nginx_status_location_block(matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses) }} + {% endif %} + + {% if matrix_nginx_proxy_proxy_matrix_corporal_api_enabled %} + location ^~ /_matrix/corporal { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "{{ matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container }}"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_corporal_api_addr_sans_container }}; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endif %} + + {% if matrix_nginx_proxy_proxy_matrix_identity_api_enabled %} + location ^~ /_matrix/identity { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container }}"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container }}; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endif %} + + {% if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled %} + location ^~ /_matrix/client/r0/user_directory/search { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "{{ matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container }}"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container }}; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endif %} + + {% if matrix_nginx_proxy_proxy_matrix_3pid_registration_enabled %} + location ~ ^/_matrix/client/r0/register/(email|msisdn)/requestToken$ { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "{{ matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_with_container }}"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_sans_container }}; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endif %} + + {% for configuration_block in matrix_nginx_proxy_proxy_matrix_additional_server_configuration_blocks %} + {{- configuration_block }} + {% endfor %} + + {# + This handles the Matrix Client API only. + The Matrix Federation API is handled by a separate vhost. + #} + location ~* ^({{ matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_prefix_regexes|join('|') }}) { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "{{ matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container }}"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container }}; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + + client_body_buffer_size 25M; + client_max_body_size {{ matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb }}M; + proxy_max_temp_file_size 0; + } + + location / { + {% if matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain %} + return 302 $scheme://{{ matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain }}$request_uri; + {% else %} + rewrite ^/$ /_matrix/static/ last; + {% endif %} + } {% endmacro %} server { listen {{ 8080 if matrix_nginx_proxy_enabled else 80 }}; + server_name {{ matrix_nginx_proxy_proxy_matrix_hostname }}; - server_name {{ matrix_nginx_proxy_base_domain_hostname }}; server_tokens off; + root /dev/null; {% if matrix_nginx_proxy_https_enabled %} location /.well-known/acme-challenge { @@ -40,6 +157,10 @@ server { {% endif %} } + {% if matrix_nginx_proxy_proxy_matrix_nginx_status_enabled %} + {{ render_nginx_status_location_block(matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses) }} + {% endif %} + location / { return 301 https://$http_host$request_uri; } @@ -53,11 +174,13 @@ server { listen {{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; listen [::]:{{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; - server_name {{ matrix_nginx_proxy_base_domain_hostname }}; + server_name {{ matrix_nginx_proxy_proxy_matrix_hostname }}; + server_tokens off; + root /dev/null; - ssl_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_base_domain_hostname }}/fullchain.pem; - ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_base_domain_hostname }}/privkey.pem; + ssl_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/fullchain.pem; + ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/privkey.pem; ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }}; {% if matrix_nginx_proxy_ssl_ciphers != '' %} @@ -68,3 +191,56 @@ server { {{ render_vhost_directives() }} } {% endif %} + +{% if matrix_nginx_proxy_proxy_matrix_federation_api_enabled %} +{# + This federation vhost is a little special. + It serves federation over HTTP or HTTPS, depending on `matrix_nginx_proxy_https_enabled`. +#} +server { + {% if matrix_nginx_proxy_https_enabled %} + listen 8448 ssl http2; + listen [::]:8448 ssl http2; + {% else %} + listen 8448; + {% endif %} + + server_name {{ matrix_nginx_proxy_proxy_matrix_hostname }}; + server_tokens off; + + root /dev/null; + + gzip on; + gzip_types text/plain application/json; + + {% if matrix_nginx_proxy_https_enabled %} + ssl_certificate {{ matrix_nginx_proxy_proxy_matrix_federation_api_ssl_certificate }}; + ssl_certificate_key {{ matrix_nginx_proxy_proxy_matrix_federation_api_ssl_certificate_key }}; + + ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }}; + {% if matrix_nginx_proxy_ssl_ciphers != '' %} + ssl_ciphers {{ matrix_nginx_proxy_ssl_ciphers }}; + {% endif %} + ssl_prefer_server_ciphers {{ matrix_nginx_proxy_ssl_prefer_server_ciphers }}; + {% endif %} + + location / { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "{{ matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container }}"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container }}; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + + client_body_buffer_size 25M; + client_max_body_size {{ matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb }}M; + proxy_max_temp_file_size 0; + } +} +{% endif %} diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-grafana.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-grafana.conf.j2 new file mode 100644 index 00000000..0e1f1c2d --- /dev/null +++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-grafana.conf.j2 @@ -0,0 +1,79 @@ +#jinja2: lstrip_blocks: "True" + +{% macro render_vhost_directives() %} + gzip on; + gzip_types text/plain application/json application/javascript text/css image/x-icon font/ttf image/gif; + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Content-Type-Options nosniff; + add_header X-Frame-Options SAMEORIGIN; + {% for configuration_block in matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks %} + {{- configuration_block }} + {% endfor %} + + location / { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-grafana:3000"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://127.0.0.1:3000; + {% endif %} + + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } +{% endmacro %} + +server { + listen {{ 8080 if matrix_nginx_proxy_enabled else 80 }}; + + server_name {{ matrix_nginx_proxy_proxy_grafana_hostname }}; + + server_tokens off; + root /dev/null; + + {% if matrix_nginx_proxy_https_enabled %} + location /.well-known/acme-challenge { + {% if matrix_nginx_proxy_enabled %} + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "matrix-certbot:8080"; + proxy_pass http://$backend; + {% else %} + {# Generic configuration for use outside of our container setup #} + proxy_pass http://127.0.0.1:{{ matrix_ssl_lets_encrypt_certbot_standalone_http_port }}; + {% endif %} + } + + location / { + return 301 https://$http_host$request_uri; + } + {% else %} + {{ render_vhost_directives() }} + {% endif %} +} + +{% if matrix_nginx_proxy_https_enabled %} +server { + listen {{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; + listen [::]:{{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; + + server_name {{ matrix_nginx_proxy_proxy_grafana_hostname }}; + + server_tokens off; + root /dev/null; + + ssl_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_grafana_hostname }}/fullchain.pem; + ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_grafana_hostname }}/privkey.pem; + + ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }}; + {% if matrix_nginx_proxy_ssl_ciphers != "" %} + ssl_ciphers {{ matrix_nginx_proxy_ssl_ciphers }}; + {% endif %} + ssl_prefer_server_ciphers {{ matrix_nginx_proxy_ssl_prefer_server_ciphers }}; + + {{ render_vhost_directives() }} +} +{% endif %} diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 index 1b4c0a8d..7041468e 100644 --- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 +++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 @@ -1,107 +1,139 @@ #jinja2: lstrip_blocks: "True" -{% macro render_nginx_status_location_block(addresses) %} - {# Empty first line to make indentation prettier. #} - - location /nginx_status { - stub_status on; - access_log off; - {% for address in addresses %} - allow {{ address }}; - {% endfor %} - deny all; - } -{% endmacro %} +{% set generic_workers = matrix_nginx_proxy_synapse_workers_list|selectattr('type', 'equalto', 'generic_worker')|list %} +{% set media_repository_workers = matrix_nginx_proxy_synapse_workers_list|selectattr('type', 'equalto', 'media_repository')|list %} +{% set user_dir_workers = matrix_nginx_proxy_synapse_workers_list|selectattr('type', 'equalto', 'user_dir')|list %} +{% set frontend_proxy_workers = matrix_nginx_proxy_synapse_workers_list|selectattr('type', 'equalto', 'frontend_proxy')|list %} +{% if matrix_nginx_proxy_synapse_workers_enabled %} + # Round Robin "upstream" pools for workers -{% macro render_vhost_directives() %} - gzip on; - gzip_types text/plain application/json; + {% if generic_workers %} + upstream generic_worker_upstream { + # ensures that requests from the same client will always be passed + # to the same server (except when this server is unavailable) + ip_hash; - location /.well-known/matrix { - root {{ matrix_static_files_base_path }}; - {# - A somewhat long expires value is used to prevent outages - in case this is unreachable due to network failure or - due to the base domain's server completely dying. - #} - expires 4h; - default_type application/json; - add_header Access-Control-Allow-Origin *; + {% for worker in generic_workers %} + {% if matrix_nginx_proxy_enabled %} + server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}"; + {% else %} + server "127.0.0.1:{{ worker.port }}"; + {% endif %} + {% endfor %} } - - {% if matrix_nginx_proxy_proxy_matrix_nginx_status_enabled %} - {{ render_nginx_status_location_block(matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses) }} {% endif %} - {% if matrix_nginx_proxy_proxy_matrix_corporal_api_enabled %} - location ^~ /_matrix/corporal { - {% if matrix_nginx_proxy_enabled %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "{{ matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container }}"; - proxy_pass http://$backend; - {% else %} - {# Generic configuration for use outside of our container setup #} - proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_corporal_api_addr_sans_container }}; - {% endif %} - - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $remote_addr; + {% if frontend_proxy_workers %} + upstream frontend_proxy_upstream { + {% for worker in frontend_proxy_workers %} + {% if matrix_nginx_proxy_enabled %} + server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}"; + {% else %} + server "127.0.0.1:{{ worker.port }}"; + {% endif %} + {% endfor %} } {% endif %} - {% if matrix_nginx_proxy_proxy_matrix_identity_api_enabled %} - location ^~ /_matrix/identity { - {% if matrix_nginx_proxy_enabled %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container }}"; - proxy_pass http://$backend; - {% else %} - {# Generic configuration for use outside of our container setup #} - proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container }}; - {% endif %} + {% if media_repository_workers %} + upstream media_repository_upstream { + {% for worker in media_repository_workers %} + {% if matrix_nginx_proxy_enabled %} + server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}"; + {% else %} + server "127.0.0.1:{{ worker.port }}"; + {% endif %} + {% endfor %} + } + {% endif %} - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $remote_addr; + {% if user_dir_workers %} + upstream user_dir_upstream { + {% for worker in user_dir_workers %} + {% if matrix_nginx_proxy_enabled %} + server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}"; + {% else %} + server "127.0.0.1:{{ worker.port }}"; + {% endif %} + {% endfor %} } {% endif %} +{% endif %} - {% if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled %} - location ^~ /_matrix/client/r0/user_directory/search { - {% if matrix_nginx_proxy_enabled %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "{{ matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container }}"; - proxy_pass http://$backend; - {% else %} - {# Generic configuration for use outside of our container setup #} - proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container }}; +server { + listen 12080; + server_name {{ matrix_nginx_proxy_proxy_synapse_hostname }}; + + server_tokens off; + root /dev/null; + + gzip on; + gzip_types text/plain application/json; + + {% if matrix_nginx_proxy_synapse_workers_enabled %} + {# Workers redirects BEGIN #} + + {% if generic_workers %} + # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker + {% for location in matrix_nginx_proxy_synapse_generic_worker_client_server_locations %} + location ~ {{ location }} { + proxy_pass http://generic_worker_upstream$request_uri; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endfor %} {% endif %} - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $remote_addr; - } - {% endif %} + {% if media_repository_workers %} + # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository + {% for location in matrix_nginx_proxy_synapse_media_repository_locations %} + location ~ {{ location }} { + proxy_pass http://media_repository_upstream$request_uri; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + + client_body_buffer_size 25M; + client_max_body_size {{ matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb }}M; + proxy_max_temp_file_size 0; + } + {% endfor %} + {% endif %} - {% if matrix_nginx_proxy_proxy_matrix_3pid_registration_enabled %} - location ~ ^/_matrix/client/r0/register/(email|msisdn)/requestToken$ { - {% if matrix_nginx_proxy_enabled %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "{{ matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_with_container }}"; - proxy_pass http://$backend; - {% else %} - {# Generic configuration for use outside of our container setup #} - proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_sans_container }}; + {% if user_dir_workers %} + # FIXME: obsolete if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled is set + # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappuser_dir + {% for location in matrix_nginx_proxy_synapse_user_dir_locations %} + location ~ {{ location }} { + proxy_pass http://user_dir_upstream$request_uri; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endfor %} {% endif %} - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $remote_addr; - } + {% if frontend_proxy_workers %} + # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappfrontend_proxy + {% for location in matrix_nginx_proxy_synapse_frontend_proxy_locations %} + location ~ {{ location }} { + proxy_pass http://frontend_proxy_upstream$request_uri; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endfor %} + {% if matrix_nginx_proxy_synapse_presence_disabled %} + # FIXME: keep in sync with synapse workers documentation manually + location ~ ^/_matrix/client/(api/v1|r0|unstable)/presence/[^/]+/status { + proxy_pass http://frontend_proxy_upstream$request_uri; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endif %} + {% endif %} + {# Workers redirects END #} {% endif %} - {% for configuration_block in matrix_nginx_proxy_proxy_matrix_additional_server_configuration_blocks %} + + {% for configuration_block in matrix_nginx_proxy_proxy_synapse_additional_server_configuration_blocks %} {{- configuration_block }} {% endfor %} @@ -127,19 +159,16 @@ } {% endif %} - {# - This handles the Matrix Client API only. - The Matrix Federation API is handled by a separate vhost. - #} - location ~* ^({{ matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_prefix_regexes|join('|') }}) { + {# Everything else just goes to the API server ##} + location / { {% if matrix_nginx_proxy_enabled %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; - set $backend "{{ matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container }}"; + set $backend "{{ matrix_nginx_proxy_proxy_synapse_client_api_addr_with_container }}"; proxy_pass http://$backend; {% else %} {# Generic configuration for use outside of our container setup #} - proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container }}; + proxy_pass http://{{ matrix_nginx_proxy_proxy_synapse_client_api_addr_sans_container }}; {% endif %} proxy_set_header Host $host; @@ -149,85 +178,13 @@ client_max_body_size {{ matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb }}M; proxy_max_temp_file_size 0; } - - location / { - {% if matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain %} - return 302 $scheme://{{ matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain }}$request_uri; - {% else %} - rewrite ^/$ /_matrix/static/ last; - {% endif %} - } -{% endmacro %} - -server { - listen {{ 8080 if matrix_nginx_proxy_enabled else 80 }}; - server_name {{ matrix_nginx_proxy_proxy_matrix_hostname }}; - - server_tokens off; - root /dev/null; - - {% if matrix_nginx_proxy_https_enabled %} - location /.well-known/acme-challenge { - {% if matrix_nginx_proxy_enabled %} - {# Use the embedded DNS resolver in Docker containers to discover the service #} - resolver 127.0.0.11 valid=5s; - set $backend "matrix-certbot:8080"; - proxy_pass http://$backend; - {% else %} - {# Generic configuration for use outside of our container setup #} - proxy_pass http://127.0.0.1:{{ matrix_ssl_lets_encrypt_certbot_standalone_http_port }}; - {% endif %} - } - - {% if matrix_nginx_proxy_proxy_matrix_nginx_status_enabled %} - {{ render_nginx_status_location_block(matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses) }} - {% endif %} - - location / { - return 301 https://$http_host$request_uri; - } - {% else %} - {{ render_vhost_directives() }} - {% endif %} } -{% if matrix_nginx_proxy_https_enabled %} +{% if matrix_nginx_proxy_proxy_synapse_federation_api_enabled %} server { - listen {{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; - listen [::]:{{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2; - - server_name {{ matrix_nginx_proxy_proxy_matrix_hostname }}; + listen 12088; - server_tokens off; - root /dev/null; - - ssl_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/fullchain.pem; - ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/privkey.pem; - - ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }}; - {% if matrix_nginx_proxy_ssl_ciphers != '' %} - ssl_ciphers {{ matrix_nginx_proxy_ssl_ciphers }}; - {% endif %} - ssl_prefer_server_ciphers {{ matrix_nginx_proxy_ssl_prefer_server_ciphers }}; - - {{ render_vhost_directives() }} -} -{% endif %} - -{% if matrix_nginx_proxy_proxy_matrix_federation_api_enabled %} -{# - This federation vhost is a little special. - It serves federation over HTTP or HTTPS, depending on `matrix_nginx_proxy_https_enabled`. -#} -server { - {% if matrix_nginx_proxy_https_enabled %} - listen 8448 ssl http2; - listen [::]:8448 ssl http2; - {% else %} - listen 8448; - {% endif %} - - server_name {{ matrix_nginx_proxy_proxy_matrix_hostname }}; + server_name {{ matrix_nginx_proxy_proxy_synapse_hostname }}; server_tokens off; root /dev/null; @@ -235,27 +192,42 @@ server { gzip on; gzip_types text/plain application/json; - {% if matrix_nginx_proxy_https_enabled %} - ssl_certificate {{ matrix_nginx_proxy_proxy_matrix_federation_api_ssl_certificate }}; - ssl_certificate_key {{ matrix_nginx_proxy_proxy_matrix_federation_api_ssl_certificate_key }}; - - ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }}; - {% if matrix_nginx_proxy_ssl_ciphers != '' %} - ssl_ciphers {{ matrix_nginx_proxy_ssl_ciphers }}; - {% endif %} - ssl_prefer_server_ciphers {{ matrix_nginx_proxy_ssl_prefer_server_ciphers }}; - + {% if matrix_nginx_proxy_synapse_workers_enabled %} + {% if generic_workers %} + # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker + {% for location in matrix_nginx_proxy_synapse_generic_worker_federation_locations %} + location ~ {{ location }} { + proxy_pass http://generic_worker_upstream$request_uri; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } + {% endfor %} + {% endif %} + {% if media_repository_workers %} + # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository + {% for location in matrix_nginx_proxy_synapse_media_repository_locations %} + location ~ {{ location }} { + proxy_pass http://media_repository_upstream$request_uri; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + + client_body_buffer_size 25M; + client_max_body_size {{ matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb }}M; + proxy_max_temp_file_size 0; + } + {% endfor %} + {% endif %} {% endif %} location / { {% if matrix_nginx_proxy_enabled %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; - set $backend "{{ matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container }}"; + set $backend "{{ matrix_nginx_proxy_proxy_synapse_federation_api_addr_with_container }}"; proxy_pass http://$backend; {% else %} {# Generic configuration for use outside of our container setup #} - proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container }}; + proxy_pass http://{{ matrix_nginx_proxy_proxy_synapse_federation_api_addr_sans_container }}; {% endif %} proxy_set_header Host $host; diff --git a/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 index 975c8b4f..facb0901 100644 --- a/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 +++ b/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 @@ -45,6 +45,11 @@ http { keepalive_timeout 65; #gzip on; + {# Map directive needed for proxied WebSocket upgrades #} + map $http_upgrade $connection_upgrade { + default upgrade; + '' close; + } include /etc/nginx/conf.d/*.conf; } diff --git a/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 b/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 index 2dd2619e..bd3070ac 100644 --- a/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 +++ b/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-nginx-proxy -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-nginx-proxy +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-nginx-proxy 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-nginx-proxy 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-nginx-proxy \ --log-driver=none \ @@ -47,8 +47,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-nginx-proxy \ {% endfor %} {{ matrix_nginx_proxy_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-nginx-proxy -ExecStop=-{{ matrix_host_command_docker }} rm matrix-nginx-proxy +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-nginx-proxy 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-nginx-proxy 2>/dev/null' ExecReload={{ matrix_host_command_docker }} exec matrix-nginx-proxy /usr/sbin/nginx -s reload Restart=always RestartSec=30 diff --git a/roles/matrix-nginx-proxy/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2 b/roles/matrix-nginx-proxy/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2 index 40cab22a..bc45e85e 100644 --- a/roles/matrix-nginx-proxy/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2 +++ b/roles/matrix-nginx-proxy/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2 @@ -27,4 +27,5 @@ docker run \ --standalone \ --preferred-challenges http \ --agree-tos \ - --email={{ matrix_ssl_lets_encrypt_support_email }} + --email={{ matrix_ssl_lets_encrypt_support_email }} \ + --no-random-sleep-on-renew diff --git a/roles/matrix-postgres/defaults/main.yml b/roles/matrix-postgres/defaults/main.yml index 8f1d0d78..8e1d2a8e 100644 --- a/roles/matrix-postgres/defaults/main.yml +++ b/roles/matrix-postgres/defaults/main.yml @@ -1,18 +1,27 @@ matrix_postgres_enabled: true -matrix_postgres_connection_hostname: "" -matrix_postgres_connection_username: "" +matrix_postgres_connection_hostname: "matrix-postgres" +matrix_postgres_connection_port: 5432 +matrix_postgres_connection_username: "matrix" matrix_postgres_connection_password: "" -matrix_postgres_db_name: "" +matrix_postgres_db_name: "matrix" matrix_postgres_base_path: "{{ matrix_base_data_path }}/postgres" matrix_postgres_data_path: "{{ matrix_postgres_base_path }}/data" -matrix_postgres_docker_image_v9: "docker.io/postgres:9.6.20-alpine" -matrix_postgres_docker_image_v10: "docker.io/postgres:10.15-alpine" -matrix_postgres_docker_image_v11: "docker.io/postgres:11.10-alpine" -matrix_postgres_docker_image_v12: "docker.io/postgres:12.5-alpine" -matrix_postgres_docker_image_v13: "docker.io/postgres:13.1-alpine" +matrix_postgres_architecture: amd64 + +# matrix_postgres_docker_image_suffix controls whether we use Alpine-based images (`-alpine`) or the normal Debian-based images. +# Alpine-based Postgres images are smaller and we usually prefer them, but they don't work on ARM32 (tested on a Raspberry Pi 3 running Raspbian 10.7). +# On ARM32, `-alpine` images fail with the following error: +# > LOG: startup process (PID 37) was terminated by signal 11: Segmentation fault +matrix_postgres_docker_image_suffix: "{{ '-alpine' if matrix_postgres_architecture in ['amd64', 'arm64'] else '' }}" + +matrix_postgres_docker_image_v9: "docker.io/postgres:9.6.21{{ matrix_postgres_docker_image_suffix }}" +matrix_postgres_docker_image_v10: "docker.io/postgres:10.16{{ matrix_postgres_docker_image_suffix }}" +matrix_postgres_docker_image_v11: "docker.io/postgres:11.11{{ matrix_postgres_docker_image_suffix }}" +matrix_postgres_docker_image_v12: "docker.io/postgres:12.6{{ matrix_postgres_docker_image_suffix }}" +matrix_postgres_docker_image_v13: "docker.io/postgres:13.2{{ matrix_postgres_docker_image_suffix }}" matrix_postgres_docker_image_latest: "{{ matrix_postgres_docker_image_v13 }}" # This variable is assigned at runtime. Overriding its value has no effect. @@ -23,6 +32,10 @@ matrix_postgres_docker_image_force_pull: "{{ matrix_postgres_docker_image_to_use # A list of extra arguments to pass to the container matrix_postgres_container_extra_arguments: [] +# A list of extra arguments to pass to the postgres process +# e.g. "-c 'max_connections=200'" +matrix_postgres_process_extra_arguments: [] + # Controls whether the matrix-postgres container exposes a port (tcp/5432 in the # container) that can be used to access the database from outside the container (e.g. with psql) # @@ -63,7 +76,10 @@ matrix_postgres_import_databases_ignore_regex: "^CREATE DATABASE ({{ matrix_post # and before trying to run queries for creating additional databases/users against it. # # For most (subsequent) runs, Postgres would already be running, so no waiting will be happening at all. -matrix_postgres_additional_databases_postgres_start_wait_timeout_seconds: 15 +# +# On ARM, we wait some more. ARM32 devices are especially known for being slow. +# ARM64 likely don't need such a long delay, but it doesn't hurt too much having it. +matrix_postgres_additional_databases_postgres_start_wait_timeout_seconds: "{{ 45 if matrix_postgres_architecture in ['arm32', 'arm64'] else 15 }}" matrix_postgres_pgloader_container_image_self_build: false diff --git a/roles/matrix-postgres/tasks/import_postgres.yml b/roles/matrix-postgres/tasks/import_postgres.yml index c26affbb..b8e93219 100644 --- a/roles/matrix-postgres/tasks/import_postgres.yml +++ b/roles/matrix-postgres/tasks/import_postgres.yml @@ -35,6 +35,13 @@ postgres_import_wait_time: "{{ 7 * 86400 }}" when: "postgres_import_wait_time|default('') == ''" +# By default, we connect and import into the main (`matrix`) database. +# Single-database dumps for Synapse may wish to import into `synapse` instead. +- name: Set postgres_default_import_database, if not provided + set_fact: + postgres_default_import_database: "{{ matrix_postgres_db_name }}" + when: "postgres_default_import_database|default('') == ''" + # Actual import work - name: Ensure matrix-postgres is started @@ -76,7 +83,7 @@ {{ 'gunzip |' if server_path_postgres_dump.endswith('.gz') else '' }} grep -vE '{{ matrix_postgres_import_roles_ignore_regex }}' | grep -vE '{{ matrix_postgres_import_databases_ignore_regex }}' | - psql -v ON_ERROR_STOP=1 -h matrix-postgres" + psql -v ON_ERROR_STOP=1 -h matrix-postgres --dbname={{ postgres_default_import_database }}" # This is a hack. # See: https://ansibledaily.com/print-to-standard-output-without-escaping/ diff --git a/roles/matrix-postgres/tasks/setup_postgres.yml b/roles/matrix-postgres/tasks/setup_postgres.yml index 518d1a5f..c072b2ea 100644 --- a/roles/matrix-postgres/tasks/setup_postgres.yml +++ b/roles/matrix-postgres/tasks/setup_postgres.yml @@ -120,6 +120,25 @@ - always when: "matrix_postgres_enabled|bool and matrix_postgres_additional_databases|length > 0" +- name: Check existence of matrix-postgres backup data path + stat: + path: "{{ matrix_postgres_data_path }}-auto-upgrade-backup" + register: matrix_postgres_data_backup_path_stat + when: "matrix_postgres_enabled|bool" + +- name: Inject warning if backup data remains + set_fact: + matrix_playbook_runtime_results: | + {{ + matrix_playbook_runtime_results|default([]) + + + [ + "NOTE: You have some Postgres backup data in `{{ matrix_postgres_data_path }}-auto-upgrade-backup`, which was created during the last major Postgres update you ran. If your setup works well after this upgrade, feel free to delete this whole directory." + ] + }} + when: "matrix_postgres_enabled|bool and matrix_postgres_data_backup_path_stat.stat.exists" + + # # Tasks related to getting rid of the internal postgres server (if it was previously enabled) # @@ -155,9 +174,16 @@ when: "not matrix_postgres_enabled|bool" # We just want to notify the user. Deleting data is too destructive. -- name: Notify if matrix-postgres local data remains - debug: - msg: "Note: You are not using a local PostgreSQL database, but some old data remains from before in `{{ matrix_postgres_data_path }}`. Feel free to delete it." +- name: Inject warning if matrix-postgres local data remains + set_fact: + matrix_playbook_runtime_results: | + {{ + matrix_playbook_runtime_results|default([]) + + + [ + "NOTE: You are not using a local PostgreSQL database, but some old data remains from before in `{{ matrix_postgres_data_path }}`. Feel free to delete it." + ] + }} when: "not matrix_postgres_enabled|bool and matrix_postgres_data_path_stat.stat.exists" - name: Remove Postgres scripts diff --git a/roles/matrix-postgres/tasks/validate_config.yml b/roles/matrix-postgres/tasks/validate_config.yml index 6ff5adb0..eac4dd5b 100644 --- a/roles/matrix-postgres/tasks/validate_config.yml +++ b/roles/matrix-postgres/tasks/validate_config.yml @@ -6,17 +6,29 @@ The `matrix_postgres_use_external` variable defined in your configuration is not used by this playbook anymore! You'll need to adapt to the new way of using an external Postgres server. It's a combination of `matrix_postgres_enabled: false` and specifying Postgres connection - details in a few `matrix_synapse_database_` variables. + details in a few `matrix_postgres_connection_` variables. See the "Using an external PostgreSQL server (optional)" documentation page. when: "'matrix_postgres_use_external' in vars" +# This is separate (from the other required variables below), +# because we'd like to have a friendlier message for our existing users. +- name: Fail if matrix_postgres_connection_password not defined + fail: + msg: >- + The playbook no longer has a default Postgres password defined in the `matrix_postgres_connection_password` variable, among lots of other Postgres changes. + You need to perform multiple manual steps to resolve this. + See our changelog for more details: + https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#breaking-change-postgres-changes-that-require-manual-intervention + when: "matrix_postgres_connection_password == ''" + - name: Fail if required Postgres settings not defined fail: - msg: > + msg: >- You need to define a required configuration setting (`{{ item }}`). when: "vars[item] == ''" with_items: - "matrix_postgres_connection_hostname" + - "matrix_postgres_connection_port" - "matrix_postgres_connection_username" - "matrix_postgres_connection_password" - "matrix_postgres_db_name" @@ -24,4 +36,4 @@ - name: Fail if Postgres password length exceeded fail: msg: "The maximum `matrix_postgres_connection_password` length is 99 characters" - when: "matrix_postgres_connection_hostname|length > 99" + when: "matrix_postgres_connection_password|length > 99" diff --git a/roles/matrix-postgres/templates/sql/init-additional-db-user-and-role.sql.j2 b/roles/matrix-postgres/templates/sql/init-additional-db-user-and-role.sql.j2 index 609a1344..a5a3385b 100644 --- a/roles/matrix-postgres/templates/sql/init-additional-db-user-and-role.sql.j2 +++ b/roles/matrix-postgres/templates/sql/init-additional-db-user-and-role.sql.j2 @@ -2,18 +2,18 @@ -- Seen here: https://stackoverflow.com/a/49858797 DO $$ BEGIN - CREATE USER {{ additional_db.username }}; + CREATE USER "{{ additional_db.username }}"; EXCEPTION WHEN DUPLICATE_OBJECT THEN - RAISE NOTICE 'not creating user {{ additional_db.username }}, since it already exists'; + RAISE NOTICE 'not creating user "{{ additional_db.username }}", since it already exists'; END $$; -- This is useful for initial user creation (since we don't assign a password above) and for handling subsequent password changes -- TODO - we should escape quotes in the password. -ALTER ROLE {{ additional_db.username }} PASSWORD '{{ additional_db.password }}'; +ALTER ROLE "{{ additional_db.username }}" PASSWORD '{{ additional_db.password }}'; -- This will generate an error on subsequent execution -CREATE DATABASE {{ additional_db.name }} WITH LC_CTYPE 'C' LC_COLLATE 'C' OWNER {{ additional_db.username }}; +CREATE DATABASE "{{ additional_db.name }}" WITH LC_CTYPE 'C' LC_COLLATE 'C' OWNER "{{ additional_db.username }}"; -- This is useful for changing the database owner subsequently -ALTER DATABASE {{ additional_db.name }} OWNER TO {{ additional_db.username }}; +ALTER DATABASE "{{ additional_db.name }}" OWNER TO "{{ additional_db.username }}"; diff --git a/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 b/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 index 13df99a4..6d1b1c6f 100644 --- a/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 +++ b/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 @@ -8,8 +8,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} stop matrix-postgres -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-postgres +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-postgres 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-postgres 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-postgres \ --log-driver=none \ @@ -28,10 +28,11 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-postgres \ {% for arg in matrix_postgres_container_extra_arguments %} {{ arg }} \ {% endfor %} - {{ matrix_postgres_docker_image_to_use }} + {{ matrix_postgres_docker_image_to_use }} \ + postgres {{ matrix_postgres_process_extra_arguments|join(' ') }} -ExecStop=-{{ matrix_host_command_docker }} stop matrix-postgres -ExecStop=-{{ matrix_host_command_docker }} rm matrix-postgres +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-postgres 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-postgres 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-postgres diff --git a/roles/matrix-postgres/templates/usr-local-bin/matrix-change-user-admin-status.j2 b/roles/matrix-postgres/templates/usr-local-bin/matrix-change-user-admin-status.j2 index e9c76674..6c3082ef 100644 --- a/roles/matrix-postgres/templates/usr-local-bin/matrix-change-user-admin-status.j2 +++ b/roles/matrix-postgres/templates/usr-local-bin/matrix-change-user-admin-status.j2 @@ -16,4 +16,4 @@ docker run \ --env-file={{ matrix_postgres_base_path }}/env-postgres-psql \ --network {{ matrix_docker_network }} \ {{ matrix_postgres_docker_image_to_use }} \ - psql -h {{ matrix_postgres_connection_hostname }} -c "UPDATE users set admin=$2 WHERE name like '@$1:{{ matrix_domain }}'" + psql -h {{ matrix_postgres_connection_hostname }} --dbname={{ matrix_synapse_database_database }} -c "UPDATE users set admin=$2 WHERE name like '@$1:{{ matrix_domain }}'" diff --git a/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-cli.j2 b/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-cli.j2 index 61f4cf80..de09a4eb 100644 --- a/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-cli.j2 +++ b/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-cli.j2 @@ -9,4 +9,5 @@ docker run \ --env-file={{ matrix_postgres_base_path }}/env-postgres-psql \ --network {{ matrix_docker_network }} \ {{ matrix_postgres_docker_image_to_use }} \ - psql -h {{ matrix_postgres_connection_hostname }} + psql -h {{ matrix_postgres_connection_hostname }} \ + "$@" diff --git a/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2 b/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2 index e546b2c5..0fbf4f21 100644 --- a/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2 +++ b/roles/matrix-postgres/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2 @@ -13,4 +13,4 @@ docker run \ --env-file={{ matrix_postgres_base_path }}/env-postgres-psql \ --network {{ matrix_docker_network }} \ {{ matrix_postgres_docker_image_to_use }} \ - psql -h {{ matrix_postgres_connection_hostname }} -c "UPDATE users set password_hash='$2' WHERE name = '@$1:{{ matrix_domain }}'" + psql -h {{ matrix_postgres_connection_hostname }} --dbname={{ matrix_synapse_database_database }} -c "UPDATE users set password_hash='$2' WHERE name = '@$1:{{ matrix_domain }}'" diff --git a/roles/matrix-prometheus-node-exporter/defaults/main.yml b/roles/matrix-prometheus-node-exporter/defaults/main.yml new file mode 100644 index 00000000..29dce364 --- /dev/null +++ b/roles/matrix-prometheus-node-exporter/defaults/main.yml @@ -0,0 +1,21 @@ +# matrix-prometheus-node-exporter is an Prometheus exporter for machine metrics +# See: https://prometheus.io/docs/guides/node-exporter/ + +matrix_prometheus_node_exporter_enabled: false + +matrix_prometheus_node_exporter_docker_image: "docker.io/prom/node-exporter:v1.1.0" +matrix_prometheus_node_exporter_docker_image_force_pull: "{{ matrix_prometheus_node_exporter_docker_image.endswith(':latest') }}" + +# A list of extra arguments to pass to the container +matrix_prometheus_node_exporter_container_extra_arguments: [] + +# List of systemd services that matrix-prometheus.service depends on +matrix_prometheus_node_exporter_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-prometheus.service wants +matrix_prometheus_node_exporter_systemd_wanted_services_list: [] + +# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9100 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:9100"), or empty string to not expose. +matrix_prometheus_node_exporter_container_http_host_bind_port: '' diff --git a/roles/matrix-prometheus-node-exporter/tasks/init.yml b/roles/matrix-prometheus-node-exporter/tasks/init.yml new file mode 100644 index 00000000..2894b717 --- /dev/null +++ b/roles/matrix-prometheus-node-exporter/tasks/init.yml @@ -0,0 +1,5 @@ +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-prometheus-node-exporter.service'] }}" + when: matrix_prometheus_node_exporter_enabled|bool + + diff --git a/roles/matrix-prometheus-node-exporter/tasks/main.yml b/roles/matrix-prometheus-node-exporter/tasks/main.yml new file mode 100644 index 00000000..172b5721 --- /dev/null +++ b/roles/matrix-prometheus-node-exporter/tasks/main.yml @@ -0,0 +1,8 @@ +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/setup.yml" + tags: + - setup-all + - setup-prometheus-node-exporter diff --git a/roles/matrix-prometheus-node-exporter/tasks/setup.yml b/roles/matrix-prometheus-node-exporter/tasks/setup.yml new file mode 100644 index 00000000..6f03fbaa --- /dev/null +++ b/roles/matrix-prometheus-node-exporter/tasks/setup.yml @@ -0,0 +1,60 @@ +--- + +# +# Tasks related to setting up matrix-prometheus-node-exporter +# + +- name: Ensure matrix-prometheus-node-exporter image is pulled + docker_image: + name: "{{ matrix_prometheus_node_exporter_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_prometheus_node_exporter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_node_exporter_docker_image_force_pull }}" + when: "matrix_prometheus_node_exporter_enabled|bool" + +- name: Ensure matrix-prometheus-node-exporter.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-prometheus-node-exporter.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-prometheus-node-exporter.service" + mode: 0644 + register: matrix_prometheus_node_exporter_systemd_service_result + when: matrix_prometheus_node_exporter_enabled|bool + +- name: Ensure systemd reloaded after matrix-prometheus.service installation + service: + daemon_reload: yes + when: "matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_systemd_service_result.changed" + +# +# Tasks related to getting rid of matrix-prometheus-node-exporter (if it was previously enabled) +# + +- name: Check existence of matrix-prometheus-node-exporter service + stat: + path: "{{ matrix_systemd_path }}/matrix-prometheus-node-exporter.service" + register: matrix_prometheus_node_exporter_service_stat + +- name: Ensure matrix-prometheus-node-exporter is stopped + service: + name: matrix-prometheus-node-exporter + state: stopped + daemon_reload: yes + register: stopping_result + when: "not matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_service_stat.stat.exists" + +- name: Ensure matrix-prometheus-node-exporter.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-prometheus-node-exporter.service" + state: absent + when: "not matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_service_stat.stat.exists" + +- name: Ensure systemd reloaded after matrix-prometheus-node-exporter.service removal + service: + daemon_reload: yes + when: "not matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_service_stat.stat.exists" + +- name: Ensure matrix-prometheus-node-exporter Docker image doesn't exist + docker_image: + name: "{{ matrix_prometheus_node_exporter_docker_image }}" + state: absent + when: "not matrix_prometheus_node_exporter_enabled|bool" diff --git a/roles/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 b/roles/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 new file mode 100644 index 00000000..93638c19 --- /dev/null +++ b/roles/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 @@ -0,0 +1,44 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=matrix-prometheus-node-exporter +{% for service in matrix_prometheus_node_exporter_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_prometheus_node_exporter_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus-node-exporter 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus-node-exporter 2>/dev/null' + + +ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-prometheus-node-exporter \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + {% if matrix_prometheus_node_exporter_container_http_host_bind_port %} + -p {{ matrix_prometheus_node_exporter_container_http_host_bind_port }}:9100 \ + {% endif %} + {% for arg in matrix_prometheus_node_exporter_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + --net=host \ + --pid=host \ + --mount type=bind,src=/,dst=/host,ro,bind-propagation=rslave \ + {{ matrix_prometheus_node_exporter_docker_image }} \ + --path.rootfs=/host + +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus-node-exporter 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus-node-exporter 2>/dev/null' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-prometheus-node-exporter + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-prometheus/defaults/main.yml b/roles/matrix-prometheus/defaults/main.yml new file mode 100644 index 00000000..56018ba6 --- /dev/null +++ b/roles/matrix-prometheus/defaults/main.yml @@ -0,0 +1,66 @@ +# matrix-prometheus is an open-source systems monitoring and alerting toolkit +# See: https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md + +matrix_prometheus_enabled: false + +matrix_prometheus_docker_image: "docker.io/prom/prometheus:v2.24.1" +matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}" + +matrix_prometheus_base_path: "{{ matrix_base_data_path }}/prometheus" +matrix_prometheus_config_path: "{{ matrix_prometheus_base_path }}/config" +matrix_prometheus_data_path: "{{ matrix_prometheus_base_path }}/data" + +# A list of extra arguments to pass to the container +matrix_prometheus_container_extra_arguments: [] + +# List of systemd services that matrix-prometheus.service depends on +matrix_prometheus_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-prometheus.service wants +matrix_prometheus_systemd_wanted_services_list: [] + +# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9090 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:9090"), or empty string to not expose. +matrix_prometheus_container_http_host_bind_port: '' + +# Tells whether the "synapse" scraper configuration is enabled. +matrix_prometheus_scraper_synapse_enabled: false + +# Tells whether to download and load a Synapse rules file +matrix_prometheus_scraper_synapse_rules_enabled: "{{ matrix_prometheus_scraper_synapse_enabled }}" +matrix_prometheus_scraper_synapse_rules_synapse_tag: "master" +matrix_prometheus_scraper_synapse_rules_download_url: "https://raw.githubusercontent.com/matrix-org/synapse/{{ matrix_prometheus_scraper_synapse_rules_synapse_tag }}/contrib/prometheus/synapse-v2.rules" + +matrix_prometheus_scraper_synapse_targets: [] + +# Tells whether the "node" scraper configuration is enabled. +# This configuration aims to scrape the current node (this server). +matrix_prometheus_scraper_node_enabled: false + +# Target addresses for the "node" scraper configuration. +# Unless you define this as a non-empty list, it gets populated at runtime with the IP address of `matrix-prometheus-node-exporter` and port 9100. +matrix_prometheus_scraper_node_targets: [] + +# Default prometheus configuration template which covers the generic use case. +# You can customize it by controlling the various variables inside it. +# +# For a more advanced customization, you can extend the default (see `matrix_prometheus_configuration_extension_yaml`) +# or completely replace this variable with your own template. +matrix_prometheus_configuration_yaml: "{{ lookup('template', 'templates/prometheus.yml.j2') }}" + +matrix_prometheus_configuration_extension_yaml: | + # Your custom YAML configuration goes here. + # This configuration extends the default starting configuration (`matrix_prometheus_configuration_yaml`). + # + # You can override individual variables from the default configuration, or introduce new ones. + # + # If you need something more special, you can take full control by + # completely redefining `matrix_prometheus_configuration_yaml`. + +matrix_prometheus_configuration_extension: "{{ matrix_prometheus_configuration_extension_yaml|from_yaml if matrix_prometheus_configuration_extension_yaml|from_yaml is mapping else {} }}" + +# Holds the final configuration (a combination of the default and its extension). +# You most likely don't need to touch this variable. Instead, see `matrix_prometheus_configuration_yaml`. +matrix_prometheus_configuration: "{{ matrix_prometheus_configuration_yaml|from_yaml|combine(matrix_prometheus_configuration_extension, recursive=True) }}" + diff --git a/roles/matrix-prometheus/tasks/init.yml b/roles/matrix-prometheus/tasks/init.yml new file mode 100644 index 00000000..12fae831 --- /dev/null +++ b/roles/matrix-prometheus/tasks/init.yml @@ -0,0 +1,5 @@ +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-prometheus.service'] }}" + when: matrix_prometheus_enabled|bool + + diff --git a/roles/matrix-prometheus/tasks/main.yml b/roles/matrix-prometheus/tasks/main.yml new file mode 100644 index 00000000..20f18cc3 --- /dev/null +++ b/roles/matrix-prometheus/tasks/main.yml @@ -0,0 +1,21 @@ +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/validate_config.yml" + when: "run_setup|bool and matrix_prometheus_enabled|bool" + tags: + - setup-all + - setup-prometheus + +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: "run_setup|bool and matrix_prometheus_enabled|bool" + tags: + - setup-all + - setup-prometheus + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: "run_setup|bool and not matrix_prometheus_enabled|bool" + tags: + - setup-all + - setup-prometheus diff --git a/roles/matrix-prometheus/tasks/setup_install.yml b/roles/matrix-prometheus/tasks/setup_install.yml new file mode 100644 index 00000000..8aee5178 --- /dev/null +++ b/roles/matrix-prometheus/tasks/setup_install.yml @@ -0,0 +1,66 @@ +--- + +- name: Ensure matrix-prometheus image is pulled + docker_image: + name: "{{ matrix_prometheus_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_prometheus_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_docker_image_force_pull }}" + +- name: Ensure Prometheus paths exists + file: + path: "{{ item }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - "{{ matrix_prometheus_base_path }}" + - "{{ matrix_prometheus_config_path }}" + - "{{ matrix_prometheus_data_path }}" + +- block: + # Well, this actually creates the network if it doesn't exist, but.. + # The network should have been created by `matrix-base` already. + # We don't rely on that other call and its result, because it runs + # on `--tags=setup-all`, but will get skipped during `--tags=setup-prometheus`. + - name: Fetch Matrix Docker network details + docker_network: + name: "{{ matrix_docker_network }}" + driver: bridge + register: matrix_docker_network_info + + - set_fact: + matrix_prometheus_scraper_node_targets: ["{{ matrix_docker_network_info.network.IPAM.Config[0].Gateway }}:9100"] + when: "matrix_prometheus_scraper_node_enabled|bool and matrix_prometheus_scraper_node_targets|length == 0" + + +- name: Download synapse-v2.rules + get_url: + url: "{{ matrix_prometheus_scraper_synapse_rules_download_url }}" + dest: "{{ matrix_prometheus_config_path }}/synapse-v2.rules" + force: true + mode: 0440 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: "matrix_prometheus_scraper_synapse_rules_enabled|bool" + +- name: Ensure prometheus.yml installed + copy: + content: "{{ matrix_prometheus_configuration|to_nice_yaml }}" + dest: "{{ matrix_prometheus_config_path }}/prometheus.yml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-prometheus.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-prometheus.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-prometheus.service" + mode: 0644 + register: matrix_prometheus_systemd_service_result + +- name: Ensure systemd reloaded after matrix-prometheus.service installation + service: + daemon_reload: yes + when: "matrix_prometheus_systemd_service_result.changed|bool" diff --git a/roles/matrix-prometheus/tasks/setup_uninstall.yml b/roles/matrix-prometheus/tasks/setup_uninstall.yml new file mode 100644 index 00000000..0a4a8cb6 --- /dev/null +++ b/roles/matrix-prometheus/tasks/setup_uninstall.yml @@ -0,0 +1,31 @@ +--- + +- name: Check existence of matrix-prometheus service + stat: + path: "{{ matrix_systemd_path }}/matrix-prometheus.service" + register: matrix_prometheus_service_stat + +- name: Ensure matrix-prometheus is stopped + service: + name: matrix-prometheus + state: stopped + daemon_reload: yes + register: stopping_result + when: "matrix_prometheus_service_stat.stat.exists|bool" + +- name: Ensure matrix-prometheus.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-prometheus.service" + state: absent + when: "matrix_prometheus_service_stat.stat.exists|bool" + +- name: Ensure systemd reloaded after matrix-prometheus.service removal + service: + daemon_reload: yes + when: "matrix_prometheus_service_stat.stat.exists|bool" + +- name: Ensure matrix-prometheus Docker image doesn't exist + docker_image: + name: "{{ matrix_prometheus_docker_image }}" + state: absent + when: "not matrix_prometheus_enabled|bool" diff --git a/roles/matrix-prometheus/tasks/validate_config.yml b/roles/matrix-prometheus/tasks/validate_config.yml new file mode 100644 index 00000000..9fcfe12b --- /dev/null +++ b/roles/matrix-prometheus/tasks/validate_config.yml @@ -0,0 +1,7 @@ +--- + +- name: Fail if Synapse metrics or Prometheus Node Exporter not enabled + fail: + msg: > + You need to enable `matrix_prometheus_scraper_synapse_enabled` and/or `matrix_prometheus_scraper_node_enabled` for Prometheus grab metrics. + when: "not matrix_prometheus_scraper_synapse_enabled and not matrix_prometheus_scraper_node_enabled" diff --git a/roles/matrix-prometheus/templates/prometheus.yml.j2 b/roles/matrix-prometheus/templates/prometheus.yml.j2 new file mode 100644 index 00000000..9502a08b --- /dev/null +++ b/roles/matrix-prometheus/templates/prometheus.yml.j2 @@ -0,0 +1,40 @@ +#jinja2: lstrip_blocks: "True" +global: + scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. + evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. + # scrape_timeout is set to the global default (10s). + +# Load rules once and periodically evaluate them according to the global 'evaluation_interval'. +rule_files: + {% if matrix_prometheus_scraper_synapse_rules_enabled %} + - 'synapse-v2.rules' + {% endif %} + +# A scrape configuration containing exactly one endpoint to scrape: +# Here it's Prometheus itself. +scrape_configs: + # The job name is added as a label `job=` to any timeseries scraped from this config. + - job_name: 'prometheus' + + # Override the global default and scrape targets from this job every 5 seconds. + scrape_interval: 5s + scrape_timeout: 5s + + # metrics_path defaults to '/metrics' + # scheme defaults to 'http'. + + static_configs: + - targets: ['localhost:9090'] + + {% if matrix_prometheus_scraper_synapse_enabled %} + - job_name: 'synapse' + metrics_path: '/_synapse/metrics' + static_configs: + - targets: {{ matrix_prometheus_scraper_synapse_targets|to_json }} + {% endif %} + + {% if matrix_prometheus_scraper_node_enabled %} + - job_name: node + static_configs: + - targets: {{ matrix_prometheus_scraper_node_targets|to_json }} + {% endif %} diff --git a/roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 b/roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 new file mode 100644 index 00000000..ad75d664 --- /dev/null +++ b/roles/matrix-prometheus/templates/systemd/matrix-prometheus.service.j2 @@ -0,0 +1,43 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=matrix-prometheus +{% for service in matrix_prometheus_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_prometheus_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus 2>/dev/null' + + +ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-prometheus \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --network={{ matrix_docker_network }} \ + {% if matrix_prometheus_container_http_host_bind_port %} + -p {{ matrix_prometheus_container_http_host_bind_port }}:9090 \ + {% endif %} + -v {{ matrix_prometheus_config_path }}:/etc/prometheus:z \ + -v {{ matrix_prometheus_data_path }}:/prometheus:z \ + {% for arg in matrix_prometheus_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_prometheus_docker_image }} + +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus 2>/dev/null' +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-prometheus + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-redis/defaults/main.yml b/roles/matrix-redis/defaults/main.yml new file mode 100644 index 00000000..74728d87 --- /dev/null +++ b/roles/matrix-redis/defaults/main.yml @@ -0,0 +1,21 @@ +matrix_redis_enabled: true + +matrix_redis_connection_password: "" + +matrix_redis_base_path: "{{ matrix_base_data_path }}/redis" +matrix_redis_data_path: "{{ matrix_redis_base_path }}/data" + +matrix_redis_docker_image_v6: "docker.io/redis:6.0.10-alpine" +matrix_redis_docker_image_latest: "{{ matrix_redis_docker_image_v6 }}" +matrix_redis_docker_image_to_use: '{{ matrix_redis_docker_image_latest }}' + +matrix_redis_docker_image_force_pull: "{{ matrix_redis_docker_image_to_use.endswith(':latest') }}" + +# A list of extra arguments to pass to the container +matrix_redis_container_extra_arguments: [] + +# Controls whether the matrix-redis container exposes a port (tcp/6379 in the container) +# that can be used to access redis from outside the container +# +# Takes an ":" or "" value (e.g. "127.0.0.1:6379"), or empty string to not expose. +matrix_redis_container_redis_bind_port: "" diff --git a/roles/matrix-redis/tasks/init.yml b/roles/matrix-redis/tasks/init.yml new file mode 100644 index 00000000..49068851 --- /dev/null +++ b/roles/matrix-redis/tasks/init.yml @@ -0,0 +1,3 @@ +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-redis'] }}" + when: matrix_redis_enabled|bool diff --git a/roles/matrix-redis/tasks/main.yml b/roles/matrix-redis/tasks/main.yml new file mode 100644 index 00000000..595b09f5 --- /dev/null +++ b/roles/matrix-redis/tasks/main.yml @@ -0,0 +1,9 @@ +- import_tasks: "{{ role_path }}/tasks/init.yml" + tags: + - always + +- import_tasks: "{{ role_path }}/tasks/setup_redis.yml" + when: run_setup|bool + tags: + - setup-all + - setup-redis diff --git a/roles/matrix-redis/tasks/setup_redis.yml b/roles/matrix-redis/tasks/setup_redis.yml new file mode 100644 index 00000000..6f00282b --- /dev/null +++ b/roles/matrix-redis/tasks/setup_redis.yml @@ -0,0 +1,99 @@ +--- + +# +# Tasks related to setting up an internal redis server +# + +- name: Ensure redis Docker image is pulled + docker_image: + name: "{{ matrix_redis_docker_image_to_use }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_redis_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_redis_docker_image_force_pull }}" + when: matrix_redis_enabled|bool + +- name: Ensure redis paths exist + file: + path: "{{ item }}" + state: directory + mode: 0700 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_username }}" + with_items: + - "{{ matrix_redis_base_path }}" + - "{{ matrix_redis_data_path }}" + when: matrix_redis_enabled|bool + +# We do this as a separate task, because: +# - we'd like to do it for the data path only, not for the base path (which contains root-owned environment variable files we'd like to leave as-is) +# - we need to do it without `mode`, or we risk making certain `.conf` and other files's executable bit to flip to true +- name: Ensure redis data path ownership is correct + file: + path: "{{ matrix_redis_data_path }}" + state: directory + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_username }}" + recurse: yes + when: matrix_redis_enabled|bool + +- name: Ensure redis environment variables file created + template: + src: "{{ role_path }}/templates/{{ item }}.j2" + dest: "{{ matrix_redis_base_path }}/{{ item }}" + mode: 0644 + with_items: + - "redis.conf" + when: matrix_redis_enabled|bool + +- name: Ensure matrix-redis.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-redis.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-redis.service" + mode: 0644 + register: matrix_redis_systemd_service_result + when: matrix_redis_enabled|bool + +- name: Ensure systemd reloaded after matrix-redis.service installation + service: + daemon_reload: yes + when: "matrix_redis_enabled|bool and matrix_redis_systemd_service_result.changed" + +# +# Tasks related to getting rid of the internal redis server (if it was previously enabled) +# + +- name: Check existence of matrix-redis service + stat: + path: "{{ matrix_systemd_path }}/matrix-redis.service" + register: matrix_redis_service_stat + when: "not matrix_redis_enabled|bool" + +- name: Ensure matrix-redis is stopped + service: + name: matrix-redis + state: stopped + daemon_reload: yes + when: "not matrix_redis_enabled|bool and matrix_redis_service_stat.stat.exists" + +- name: Ensure matrix-redis.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-redis.service" + state: absent + when: "not matrix_redis_enabled|bool and matrix_redis_service_stat.stat.exists" + +- name: Ensure systemd reloaded after matrix-redis.service removal + service: + daemon_reload: yes + when: "not matrix_redis_enabled|bool and matrix_redis_service_stat.stat.exists" + +- name: Check existence of matrix-redis local data path + stat: + path: "{{ matrix_redis_data_path }}" + register: matrix_redis_data_path_stat + when: "not matrix_redis_enabled|bool" + +# We just want to notify the user. Deleting data is too destructive. +- name: Notify if matrix-redis local data remains + debug: + msg: "Note: You are not using a local redis instance, but some old data remains from before in `{{ matrix_redis_data_path }}`. Feel free to delete it." + when: "not matrix_redis_enabled|bool and matrix_redis_data_path_stat.stat.exists" diff --git a/roles/matrix-redis/templates/redis.conf.j2 b/roles/matrix-redis/templates/redis.conf.j2 new file mode 100644 index 00000000..34371356 --- /dev/null +++ b/roles/matrix-redis/templates/redis.conf.j2 @@ -0,0 +1,4 @@ +#jinja2: lstrip_blocks: "True" +{% if matrix_redis_connection_password %} +requirepass {{ matrix_redis_connection_password }} +{% endif %} diff --git a/roles/matrix-redis/templates/systemd/matrix-redis.service.j2 b/roles/matrix-redis/templates/systemd/matrix-redis.service.j2 new file mode 100644 index 00000000..0752d23b --- /dev/null +++ b/roles/matrix-redis/templates/systemd/matrix-redis.service.j2 @@ -0,0 +1,36 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Matrix Redis server +After=docker.service +Requires=docker.service + +[Service] +Type=simple +ExecStartPre=-/usr/bin/docker stop matrix-redis +ExecStartPre=-/usr/bin/docker rm matrix-redis + +ExecStart=/usr/bin/docker run --rm --name matrix-redis \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --tmpfs=/tmp:rw,noexec,nosuid,size=100m \ + --network={{ matrix_docker_network }} \ + {% if matrix_redis_container_redis_bind_port %} + -p {{ matrix_redis_container_redis_bind_port }}:6379 \ + {% endif %} + -v {{ matrix_redis_base_path }}/redis.conf:/usr/local/etc/redis/redis.conf \ + {% for arg in matrix_redis_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_redis_docker_image_to_use }} \ + redis-server /usr/local/etc/redis/redis.conf + +ExecStop=-/usr/bin/docker stop matrix-redis +ExecStop=-/usr/bin/docker rm matrix-redis +Restart=always +RestartSec=30 +SyslogIdentifier=matrix-redis + +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 b/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 index f0b50030..e73e3e5f 100644 --- a/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 +++ b/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-registration -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-registration +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-registration 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-registration 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-registration \ --log-driver=none \ @@ -32,8 +32,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-registration \ {{ matrix_registration_docker_image }} \ serve -ExecStop=-{{ matrix_host_command_docker }} kill matrix-registration -ExecStop=-{{ matrix_host_command_docker }} rm matrix-registration +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-registration 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-registration 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-registration diff --git a/roles/matrix-synapse-admin/defaults/main.yml b/roles/matrix-synapse-admin/defaults/main.yml index 1dbf0ad2..ce0ccd39 100644 --- a/roles/matrix-synapse-admin/defaults/main.yml +++ b/roles/matrix-synapse-admin/defaults/main.yml @@ -8,7 +8,7 @@ matrix_synapse_admin_container_self_build_repo: "https://github.com/Awesome-Tech matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src" -matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:0.6.1" +matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:0.7.0" matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_self_build else 'docker.io/' }}" matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}" diff --git a/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 b/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 index d376238a..4823d89c 100644 --- a/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 +++ b/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 @@ -13,8 +13,8 @@ DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-synapse-admin -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-synapse-admin +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-synapse-admin 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-synapse-admin 2>/dev/null' ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-synapse-admin \ --log-driver=none \ @@ -32,8 +32,8 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-synapse-admin {% endfor %} {{ matrix_synapse_admin_docker_image }} -ExecStop=-{{ matrix_host_command_docker }} kill matrix-synapse-admin -ExecStop=-{{ matrix_host_command_docker }} rm matrix-synapse-admin +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-synapse-admin 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-synapse-admin 2>/dev/null' Restart=always RestartSec=30 SyslogIdentifier=matrix-synapse-admin diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml index 0e7cf987..5da48c08 100644 --- a/roles/matrix-synapse/defaults/main.yml +++ b/roles/matrix-synapse/defaults/main.yml @@ -11,7 +11,11 @@ matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_cont # The if statement below may look silly at times (leading to the same version being returned), # but ARM-compatible container images are only released 1-7 hours after a release, # so we may often be on different versions for different architectures when new Synapse releases come out. -matrix_synapse_docker_image_tag: "{{ 'v1.25.0' if matrix_architecture == 'amd64' else 'v1.25.0' }}" +# +# amd64 gets released first. +# arm32 relies on self-building, so the same version can be built immediately. +# arm64 users need to wait for a prebuilt image to become available. +matrix_synapse_docker_image_tag: "{{ 'v1.27.0' if matrix_architecture in ['arm32', 'amd64'] else 'v1.26.0' }}" matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}" matrix_synapse_base_path: "{{ matrix_base_data_path }}/synapse" @@ -117,6 +121,18 @@ matrix_synapse_rc_login: per_second: 0.17 burst_count: 3 +matrix_synapse_rc_admin_redaction: + per_second: 1 + burst_count: 50 + +matrix_synapse_rc_joins: + local: + per_second: 0.1 + burst_count: 3 + remote: + per_second: 0.01 + burst_count: 3 + matrix_synapse_rc_federation: window_size: 1000 sleep_limit: 10 @@ -286,6 +302,127 @@ matrix_synapse_metrics_port: 9100 # See https://github.com/matrix-org/synapse/blob/master/docs/manhole.md matrix_synapse_manhole_enabled: false +# Enable support for Synapse workers +matrix_synapse_workers_enabled: false + +# Specifies worker configuration that should be used when workers are enabled. +# +# The posible values (as seen in `matrix_synapse_workers_presets`) are: +# - "little-federation-helper" - a very minimal worker configuration to improve federation performance +# - "one-of-each" - one worker of each supported type +# +# You can override `matrix_synapse_workers_presets` to define your own presets, which is ill-advised, because it's fragile. +# To use a more custom configuration, start with one of these presets as a base and configure `matrix_synapse_workers_*_count` variables manually, to suit your liking. +matrix_synapse_workers_preset: one-of-each + +matrix_synapse_workers_presets: + little-federation-helper: + generic_workers_count: 0 + pusher_workers_count: 0 + appservice_workers_count: 0 + federation_sender_workers_count: 1 + media_repository_workers_count: 0 + user_dir_workers_count: 0 + frontend_proxy_workers_count: 0 + one-of-each: + generic_workers_count: 1 + pusher_workers_count: 1 + appservice_workers_count: 1 + federation_sender_workers_count: 1 + media_repository_workers_count: 1 + # Disabled until https://github.com/matrix-org/synapse/issues/8787 is resolved. + user_dir_workers_count: 0 + frontend_proxy_workers_count: 1 + +# Controls whether the matrix-synapse container exposes the various worker ports +# (see `port` and `metrics_port` in `matrix_synapse_workers_enabled_list`) outside of the container. +# +# Takes an "" value (e.g. "127.0.0.1", "0.0.0.0", etc), or empty string to not expose. +# It takes "*" to signify "bind on all interfaces" ("0.0.0.0" is IPv4-only). +matrix_synapse_workers_container_host_bind_address: '' + +matrix_synapse_workers_generic_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['generic_workers_count'] }}" +matrix_synapse_workers_generic_workers_port_range_start: 18111 +matrix_synapse_workers_generic_workers_metrics_range_start: 19111 + +# matrix_synapse_workers_pusher_workers_count can only be 0 or 1 for now. +# More instances are not supported due to a playbook limitation having to do with keeping `pusher_instances` in `homeserver.yaml` updated. +# See https://github.com/matrix-org/synapse/commit/ddfdf945064925eba761ae3748e38f3a1c73c328 +matrix_synapse_workers_pusher_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['pusher_workers_count'] }}" +matrix_synapse_workers_pusher_workers_metrics_range_start: 19200 + +# matrix_synapse_workers_appservice_workers_count can only be 0 or 1. More instances are not supported. +matrix_synapse_workers_appservice_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['appservice_workers_count'] }}" +matrix_synapse_workers_appservice_workers_metrics_range_start: 19300 + +# matrix_synapse_workers_federation_sender_workers_count can only be 0 or 1 for now. +# More instances are not supported due to a playbook limitation having to do with keeping `federation_sender_instances` in `homeserver.yaml` updated. +# See https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappfederation_sender +matrix_synapse_workers_federation_sender_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['federation_sender_workers_count'] }}" +matrix_synapse_workers_federation_sender_workers_metrics_range_start: 19400 + +matrix_synapse_workers_media_repository_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['media_repository_workers_count'] }}" +matrix_synapse_workers_media_repository_workers_port_range_start: 18551 +matrix_synapse_workers_media_repository_workers_metrics_range_start: 19551 + +# Disabled until https://github.com/matrix-org/synapse/issues/8787 is resolved. +matrix_synapse_workers_user_dir_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['user_dir_workers_count'] }}" +matrix_synapse_workers_user_dir_workers_port_range_start: 18661 +matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661 + +matrix_synapse_workers_frontend_proxy_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['frontend_proxy_workers_count'] }}" +matrix_synapse_workers_frontend_proxy_workers_port_range_start: 18771 +matrix_synapse_workers_frontend_proxy_workers_metrics_range_start: 19771 + +# Default list of workers to spawn. +# +# Unless you populate this manually, this list is dynamically generated +# based on other variables above: +# - `matrix_synapse_workers_*_workers_count` +# - `matrix_synapse_workers_*_workers_port_range_start` +# - `matrix_synapse_workers_*_workers_port_metrics_range_start` +# +# We advise that you use those variables and let this list be populated dynamically. +# Doing that is simpler and also protects you from shooting yourself in the foot, +# as certain workers can only be spawned just once. +# +# Each worker instance in the list defines the following fields: +# - `type` - the type of worker (`generic_worker`, etc.) +# - `instanceId` - a string that identifies the worker. The combination of (`type` + `instanceId`) represents the name of the worker and must be unique. +# - `port` - an HTTP port where the worker listens for requests (can be `0` for workers that don't do HTTP request processing) +# - `metrics_port` - an HTTP port where the worker exports Prometheus metrics +# +# Example of what this needs to look like, if you're defining it manually: +# matrix_synapse_workers_enabled_list: +# - { type: generic_worker, instanceId: '18111', port: 18111, metrics_port: 19111 } +# - { type: generic_worker, instanceId: '18112', port: 18112, metrics_port: 19112 } +# - { type: generic_worker, instanceId: '18113', port: 18113, metrics_port: 19113 } +# - { type: generic_worker, instanceId: '18114', port: 18114, metrics_port: 19114 } +# - { type: generic_worker, instanceId: '18115', port: 18115, metrics_port: 19115 } +# - { type: generic_worker, instanceId: '18116', port: 18116, metrics_port: 19116 } +# - { type: pusher, instanceId: '0', port: 0, metrics_port: 19200 } +# - { type: appservice, instanceId: '0', port: 0, metrics_port: 19300 } +# - { type: federation_sender, instanceId: '0', port: 0, metrics_port: 19400 } +# - { type: media_repository, instanceId: '18551', port: 18551, metrics_port: 19551 } +matrix_synapse_workers_enabled_list: [] + +# Redis information +matrix_synapse_redis_enabled: false +matrix_synapse_redis_host: "" +matrix_synapse_redis_port: 6379 +matrix_synapse_redis_password: "" + +# Controls whether Synapse starts a replication listener necessary for workers. +# +# If Redis is available, we prefer to use that, instead of talking over Synapse's custom replication protocol. +# +# matrix_synapse_replication_listener_enabled: "{{ matrix_synapse_workers_enabled and not matrix_redis_enabled }}" +# We force-enable this listener for now until we debug why communication via Redis fails. +matrix_synapse_replication_listener_enabled: true + +# Port used for communication between main synapse process and workers. +# Only gets used if `matrix_synapse_replication_listener_enabled: true` +matrix_synapse_replication_http_port: 9093 # Send ERROR logs to sentry.io for easier tracking # To set this up: go to sentry.io, create a python project, and set @@ -294,10 +431,10 @@ matrix_synapse_manhole_enabled: false matrix_synapse_sentry_dsn: "" # Postgres database information -matrix_synapse_database_host: "" -matrix_synapse_database_user: "" +matrix_synapse_database_host: "matrix-postgres" +matrix_synapse_database_user: "synapse" matrix_synapse_database_password: "" -matrix_synapse_database_database: "" +matrix_synapse_database_database: "synapse" matrix_synapse_turn_uris: [] matrix_synapse_turn_shared_secret: "" @@ -354,6 +491,7 @@ matrix_s3_media_store_bucket_name: "your-bucket-name" matrix_s3_media_store_aws_access_key: "your-aws-access-key" matrix_s3_media_store_aws_secret_key: "your-aws-secret-key" matrix_s3_media_store_region: "eu-central-1" +matrix_s3_media_store_path: "{{ matrix_synapse_media_store_path }}" # Controls whether the self-check feature should validate SSL certificates. matrix_synapse_self_check_validate_certificates: true diff --git a/roles/matrix-synapse/files/workers-doc-to-yaml.awk b/roles/matrix-synapse/files/workers-doc-to-yaml.awk new file mode 100755 index 00000000..d9295e32 --- /dev/null +++ b/roles/matrix-synapse/files/workers-doc-to-yaml.awk @@ -0,0 +1,146 @@ +#!/usr/bin/awk +# Hackish approach to get a machine-readable list of current matrix +# synapse REST API endpoints from the official documentation at +# https://github.com/matrix-org/synapse/raw/master/docs/workers.md +# +# invoke in shell with: +# URL=https://github.com/matrix-org/synapse/raw/master/docs/workers.md +# curl -L ${URL} | awk -f workers-doc-to-yaml.awk - + +function worker_stanza_append(string) { + worker_stanza = worker_stanza string +} + +function line_is_endpoint_url(line) { + # probably API endpoint if it starts with white-space and ^ or / + return (line ~ /^ +[\^\/].*\//) +} + +# Put YAML marker at beginning of file. +BEGIN { + print "---" + endpoint_conditional_comment = " # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually\n" +} + +# Enable further processing after the introductory text. +# Read each synapse worker section as record and its lines as fields. +/Available worker applications/ { + enable_parsing = 1 + # set record separator to markdown section header + RS = "\n### " + # set field separator to newline + FS = "\n" +} + +# Once parsing is active, this will process each section as record. +enable_parsing { + # Each worker section starts with a synapse.app.X headline + if ($1 ~ /synapse\.app\./) { + + # get rid of the backticks and extract worker type from headline + gsub("`", "", $1) + gsub("synapse.app.", "", $1) + worker_type = $1 + + # initialize empty worker stanza + worker_stanza = "" + + # track if any endpoints are mentioned in a specific section + worker_has_urls = 0 + + # some endpoint descriptions contain flag terms + endpoints_seem_conditional = 0 + + # also, collect a list of available workers + workers = (workers ? workers "\n" : "") " - " worker_type + + # loop through the lines (2 - number of fields in record) + for (i = 2; i < NF + 1; i++) { + # copy line for gsub replacements + line = $i + + # end all lines but the last with a linefeed + linefeed = (i < NF - 1) ? "\n" : "" + + # line starts with white-space and a hash: endpoint block headline + if (line ~ /^ +#/) { + + # copy to output verbatim, normalizing white-space + gsub(/^ +/, "", line) + worker_stanza_append(" " line linefeed) + + } else if (line_is_endpoint_url(line)) { + + # mark section for special output formatting + worker_has_urls = 1 + + # remove leading white-space + gsub(/^ +/, "", line) + api_endpoint_regex = line + + # FIXME: https://github.com/matrix-org/synapse/issues/new + # munge inconsistent media_repository endpoint notation + if (api_endpoint_regex == "/_matrix/media/") { + api_endpoint_regex = "^" line + } + + # FIXME: https://github.com/matrix-org/synapse/issues/7530 + # https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456#issuecomment-719015911 + if (api_endpoint_regex == "^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$") { + worker_stanza_append(" # FIXME: possible bug with SSO and multiple generic workers\n") + worker_stanza_append(" # see https://github.com/matrix-org/synapse/issues/7530\n") + worker_stanza_append(" # " api_endpoint_regex linefeed) + continue + } + + # disable endpoints which specify complications + if (endpoints_seem_conditional) { + # only add notice if previous line didn't match + if (!line_is_endpoint_url($(i - 1))) { + worker_stanza_append(endpoint_conditional_comment) + } + worker_stanza_append(" # " api_endpoint_regex linefeed) + } else { + # output endpoint regex + worker_stanza_append(" - " api_endpoint_regex linefeed) + } + + # white-space only line? + } else if (line ~ /^ *$/) { + + if (i > 3 && i < NF) { + # print white-space lines unless 1st or last line in section + worker_stanza_append(line linefeed) + } + + # nothing of the above: the line is regular documentation text + } else { + + # include this text line as comment + worker_stanza_append(" # " line linefeed) + + # and take note of words hinting at additional conditions to be met + if (line ~ /(^| )[Ii]f |(^| )[Ff]or /) { + endpoints_seem_conditional = 1 + } + } + } + + if (worker_has_urls) { + print "\nmatrix_synapse_workers_" worker_type "_endpoints:" + print worker_stanza + } else { + # include workers without endpoints as well for reference + print "\n# " worker_type " worker (no API endpoints) [" + print worker_stanza + print "# ]" + } + } +} + +END { + print "\nmatrix_synapse_workers_avail_list:" + print workers | "sort" +} + +# vim: tabstop=4 shiftwidth=4 expandtab autoindent diff --git a/roles/matrix-synapse/files/workers-doc-to-yaml.sh b/roles/matrix-synapse/files/workers-doc-to-yaml.sh new file mode 100755 index 00000000..5981523b --- /dev/null +++ b/roles/matrix-synapse/files/workers-doc-to-yaml.sh @@ -0,0 +1,6 @@ +#!/bin/sh +# Fetch the synapse worker documentation and extract endpoint URLs +# matrix-org/synapse master branch points to current stable release + +URL=https://github.com/matrix-org/synapse/raw/master/docs/workers.md +curl -L ${URL} | awk -f workers-doc-to-yaml.awk > ../vars/workers.yml diff --git a/roles/matrix-synapse/tasks/goofys/setup_install.yml b/roles/matrix-synapse/tasks/goofys/setup_install.yml index 93237986..b5e95614 100644 --- a/roles/matrix-synapse/tasks/goofys/setup_install.yml +++ b/roles/matrix-synapse/tasks/goofys/setup_install.yml @@ -8,18 +8,18 @@ # This will throw a Permission Denied error if already mounted - name: Check Matrix Goofys external storage mountpoint path stat: - path: "{{ matrix_synapse_media_store_path }}" - register: local_path_matrix_synapse_media_store_path_stat + path: "{{ matrix_s3_media_store_path }}" + register: local_path_matrix_s3_media_store_path_stat ignore_errors: yes - name: Ensure Matrix Goofys external storage mountpoint exists file: - path: "{{ matrix_synapse_media_store_path }}" + path: "{{ matrix_s3_media_store_path }}" state: directory mode: 0750 owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" - when: "not local_path_matrix_synapse_media_store_path_stat.failed and not local_path_matrix_synapse_media_store_path_stat.stat.exists" + when: "not local_path_matrix_s3_media_store_path_stat.failed and not local_path_matrix_s3_media_store_path_stat.stat.exists" - name: Ensure goofys environment variables file created template: diff --git a/roles/matrix-synapse/tasks/init.yml b/roles/matrix-synapse/tasks/init.yml index bdb62476..04b8d2b8 100644 --- a/roles/matrix-synapse/tasks/init.yml +++ b/roles/matrix-synapse/tasks/init.yml @@ -1,7 +1,19 @@ +# Unless `matrix_synapse_workers_enabled_list` is explicitly defined, +# we'll generate it dynamically. +- import_tasks: "{{ role_path }}/tasks/synapse/workers/init.yml" + when: "matrix_synapse_enabled and matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list|length == 0" + - set_fact: matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-synapse.service'] }}" when: matrix_synapse_enabled|bool +- name: Ensure systemd services for workers are injected + include_tasks: "{{ role_path }}/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml" + with_items: "{{ matrix_synapse_workers_enabled_list }}" + loop_control: + loop_var: matrix_synapse_worker_details + when: matrix_synapse_enabled|bool and matrix_synapse_workers_enabled|bool + - set_fact: matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-goofys.service'] }}" when: matrix_s3_media_store_enabled|bool diff --git a/roles/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml b/roles/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml index 8570411f..46cad808 100644 --- a/roles/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml +++ b/roles/matrix-synapse/tasks/rust-synapse-compress-state/compress_room.yml @@ -34,7 +34,7 @@ --entrypoint=/bin/sh {{ matrix_postgres_docker_image_latest }} -c "cat /work/state-compressor.sql | - psql -v ON_ERROR_STOP=1 -h matrix-postgres" + psql -v ON_ERROR_STOP=1 -h matrix-postgres -d {{ matrix_synapse_database_database }}" - name: Import compression SQL into Postgres command: "{{ matrix_synapse_rust_synapse_compress_state_psql_import_command }}" diff --git a/roles/matrix-synapse/tasks/setup_synapse.yml b/roles/matrix-synapse/tasks/setup_synapse.yml index 1ae7ade6..f8bc05a1 100644 --- a/roles/matrix-synapse/tasks/setup_synapse.yml +++ b/roles/matrix-synapse/tasks/setup_synapse.yml @@ -18,6 +18,8 @@ - import_tasks: "{{ role_path }}/tasks/ext/setup.yml" +- import_tasks: "{{ role_path }}/tasks/synapse/workers/setup.yml" + - import_tasks: "{{ role_path }}/tasks/synapse/setup.yml" - import_tasks: "{{ role_path }}/tasks/goofys/setup.yml" diff --git a/roles/matrix-synapse/tasks/synapse/workers/init.yml b/roles/matrix-synapse/tasks/synapse/workers/init.yml new file mode 100644 index 00000000..c6fc32c3 --- /dev/null +++ b/roles/matrix-synapse/tasks/synapse/workers/init.yml @@ -0,0 +1,86 @@ +# Below is a huge hack for dynamically building a list of workers and finally assigning it to `matrix_synapse_workers_enabled_list`. +# +# set_fact within a loop does not work reliably in Ansible (it only executes on the first iteration for some reason), +# so we're forced to do something much uglier. + +- name: Build generic workers + set_fact: + worker: + type: 'generic_worker' + instanceId: "{{ matrix_synapse_workers_generic_workers_port_range_start + item }}" + port: "{{ matrix_synapse_workers_generic_workers_port_range_start + item }}" + metrics_port: "{{ matrix_synapse_workers_generic_workers_metrics_range_start + item }}" + register: "matrix_synapse_workers_list_results_generic_workers" + loop: "{{ range(0, matrix_synapse_workers_generic_workers_count|int)|list }}" + +- name: Build federation sender workers + set_fact: + worker: + type: 'federation_sender' + instanceId: "{{ item }}" + port: 0 + metrics_port: "{{ matrix_synapse_workers_federation_sender_workers_metrics_range_start + item }}" + register: "matrix_synapse_workers_list_results_federation_sender_workers" + loop: "{{ range(0, matrix_synapse_workers_federation_sender_workers_count|int)|list }}" + +# This type of worker can only have a count of 1, at most +- name: Build pusher workers + set_fact: + worker: + type: 'pusher' + instanceId: "{{ item }}" + port: 0 + metrics_port: "{{ matrix_synapse_workers_pusher_workers_metrics_range_start + item }}" + register: "matrix_synapse_workers_list_results_pusher_workers" + loop: "{{ range(0, matrix_synapse_workers_pusher_workers_count|int)|list }}" + +# This type of worker can only have a count of 1, at most +- name: Build appservice workers + set_fact: + worker: + type: 'appservice' + instanceId: "{{ item }}" + port: 0 + metrics_port: "{{ matrix_synapse_workers_appservice_workers_metrics_range_start + item }}" + register: "matrix_synapse_workers_list_results_appservice_workers" + loop: "{{ range(0, matrix_synapse_workers_appservice_workers_count|int)|list }}" + +- name: Build media_repository workers + set_fact: + worker: + type: 'media_repository' + instanceId: "{{ matrix_synapse_workers_media_repository_workers_port_range_start + item }}" + port: "{{ matrix_synapse_workers_media_repository_workers_port_range_start + item }}" + metrics_port: "{{ matrix_synapse_workers_media_repository_workers_metrics_range_start + item }}" + register: "matrix_synapse_workers_list_results_media_repository_workers" + loop: "{{ range(0, matrix_synapse_workers_media_repository_workers_count|int)|list }}" + +- name: Build frontend_proxy workers + set_fact: + worker: + type: 'frontend_proxy' + instanceId: "{{ matrix_synapse_workers_frontend_proxy_workers_port_range_start + item }}" + port: "{{ matrix_synapse_workers_frontend_proxy_workers_port_range_start + item }}" + metrics_port: "{{ matrix_synapse_workers_frontend_proxy_workers_metrics_range_start + item }}" + register: "matrix_synapse_workers_list_results_frontend_proxy_workers" + loop: "{{ range(0, matrix_synapse_workers_frontend_proxy_workers_count|int)|list }}" + +- set_fact: + matrix_synapse_dynamic_workers_list: "{{ matrix_synapse_dynamic_workers_list|default([]) + [item.ansible_facts.worker] }}" + with_items: | + {{ + matrix_synapse_workers_list_results_generic_workers.results + + + matrix_synapse_workers_list_results_federation_sender_workers.results + + + matrix_synapse_workers_list_results_pusher_workers.results + + + matrix_synapse_workers_list_results_appservice_workers.results + + + matrix_synapse_workers_list_results_media_repository_workers.results + + + matrix_synapse_workers_list_results_frontend_proxy_workers.results + }} + +- set_fact: + matrix_synapse_workers_enabled_list: "{{ matrix_synapse_dynamic_workers_list }}" diff --git a/roles/matrix-synapse/tasks/synapse/workers/setup.yml b/roles/matrix-synapse/tasks/synapse/workers/setup.yml new file mode 100644 index 00000000..ce66a2e4 --- /dev/null +++ b/roles/matrix-synapse/tasks/synapse/workers/setup.yml @@ -0,0 +1,21 @@ +--- + +# A previous version of the worker setup used this. +# This is a temporary cleanup for people who ran that version. +- name: Ensure old matrix-synapse.service.wants directory is gone + file: + path: "{{ matrix_systemd_path }}/matrix-synapse.service.wants" + state: absent + +# Same. This was part of a previous version of the worker setup. +# No longer necessary. +- name: Ensure matrix-synapse-worker-write-pid script is removed + file: + path: "{{ matrix_local_bin_path }}/matrix-synapse-worker-write-pid" + state: absent + +- include_tasks: "{{ role_path }}/tasks/synapse/workers/setup_install.yml" + when: "matrix_synapse_enabled|bool and matrix_synapse_workers_enabled|bool" + +- include_tasks: "{{ role_path }}/tasks/synapse/workers/setup_uninstall.yml" + when: "not matrix_synapse_workers_enabled|bool" diff --git a/roles/matrix-synapse/tasks/synapse/workers/setup_install.yml b/roles/matrix-synapse/tasks/synapse/workers/setup_install.yml new file mode 100644 index 00000000..983f1876 --- /dev/null +++ b/roles/matrix-synapse/tasks/synapse/workers/setup_install.yml @@ -0,0 +1,42 @@ +--- + +- name: Determine current worker configs + find: + path: "{{ matrix_synapse_config_dir_path }}" + patterns: "worker.*.yaml" + use_regex: true + register: matrix_synapse_workers_current_config_files + +# This also deletes some things which we need. They will be recreated below. +- name: Ensure previous worker configs are cleaned + file: + path: "{{ item.path }}" + state: absent + with_items: "{{ matrix_synapse_workers_current_config_files.files }}" + +- name: Determine current worker systemd services + find: + path: "{{ matrix_systemd_path }}" + patterns: "matrix-synapse-worker.*.service" + use_regex: true + register: matrix_synapse_workers_current_systemd_services + +- name: Ensure unnecessary worker systemd services are stopped and disabled + service: + name: "{{ item.path|basename }}" + state: stopped + enabled: false + with_items: "{{ matrix_synapse_workers_current_systemd_services.files }}" + when: "not ansible_check_mode and item.path|basename not in matrix_systemd_services_list" + +- name: Ensure unnecessary worker systemd services are cleaned + file: + path: "{{ item.path }}" + state: absent + with_items: "{{ matrix_synapse_workers_current_systemd_services.files }}" + +- name: Ensure creation of worker systemd service files and configuration files + include_tasks: "{{ role_path }}/tasks/synapse/workers/util/setup_files_for_worker.yml" + with_items: "{{ matrix_synapse_workers_enabled_list }}" + loop_control: + loop_var: matrix_synapse_worker_details diff --git a/roles/matrix-synapse/tasks/synapse/workers/setup_uninstall.yml b/roles/matrix-synapse/tasks/synapse/workers/setup_uninstall.yml new file mode 100644 index 00000000..4a90bfa6 --- /dev/null +++ b/roles/matrix-synapse/tasks/synapse/workers/setup_uninstall.yml @@ -0,0 +1,36 @@ +--- + +- name: Populate service facts + service_facts: + +- name: Ensure any worker services are stopped + service: + name: "{{ item.key }}" + state: stopped + with_dict: "{{ ansible_facts.services|default({})|dict2items|selectattr('key', 'match', 'matrix-synapse-worker-.+\\.service')|list|items2dict }}" + +- name: Find worker configs to be cleaned + find: + path: "{{ matrix_synapse_config_dir_path }}" + patterns: "worker.*.yaml" + use_regex: true + register: matrix_synapse_workers_current_config_files + +- name: Ensure previous worker configs are cleaned + file: + path: "{{ item.path }}" + state: absent + with_items: "{{ matrix_synapse_workers_current_config_files.files }}" + +- name: Find worker systemd services to be cleaned + find: + path: "{{ matrix_systemd_path }}" + patterns: "matrix-synapse-worker.*.service" + use_regex: true + register: matrix_synapse_workers_current_systemd_services + +- name: Ensure previous worker systemd services are cleaned + file: + path: "{{ item.path }}" + state: absent + with_items: "{{ matrix_synapse_workers_current_systemd_services.files }}" diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml new file mode 100644 index 00000000..62b42625 --- /dev/null +++ b/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml @@ -0,0 +1,18 @@ +# The tasks below run before `validate_config.yml`. +# To avoid failing with a cryptic error message, we'll do validation here. +# +# This check is mostly relevant to people who explicitly define `matrix_synapse_workers_enabled_list` +# (Synapse Workers users from the earlier days of this PR - https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456). +# +# In the future, it should be possible to remove this check. +# Our own code which dynamically builds `matrix_synapse_workers_enabled_list` does things right. +- name: Fail if instanceId not defined for worker + fail: + msg: "Synapse workers (like {{ matrix_synapse_worker_details|to_json }}) need to define an instanceId property (type + instanceId must be unique)" + when: "'instanceId' not in matrix_synapse_worker_details" + +- set_fact: + matrix_synapse_worker_systemd_service_name: "matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.instanceId }}.service" + +- set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + [matrix_synapse_worker_systemd_service_name] }}" diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml new file mode 100644 index 00000000..93ed6575 --- /dev/null +++ b/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml @@ -0,0 +1,19 @@ +- set_fact: + matrix_synapse_worker_systemd_service_name: "matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.instanceId }}" + +- set_fact: + matrix_synapse_worker_container_name: "{{ matrix_synapse_worker_systemd_service_name }}" + +- set_fact: + matrix_synapse_worker_config_file_name: "worker.{{ matrix_synapse_worker_details.type }}_{{ matrix_synapse_worker_details.instanceId }}.yaml" + +- name: Ensure configuration exists for {{ matrix_synapse_worker_systemd_service_name }} + template: + src: "{{ role_path }}/templates/synapse/worker.yaml.j2" + dest: "{{ matrix_synapse_config_dir_path }}/{{ matrix_synapse_worker_config_file_name }}" + +- name: Ensure systemd service exists for {{ matrix_synapse_worker_systemd_service_name }} + template: + src: "{{ role_path }}/templates/synapse/systemd/matrix-synapse-worker.service.j2" + dest: "{{ matrix_systemd_path }}/{{ matrix_synapse_worker_systemd_service_name }}.service" + mode: 0644 diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml index fe3cb2e6..f7631111 100644 --- a/roles/matrix-synapse/tasks/validate_config.yml +++ b/roles/matrix-synapse/tasks/validate_config.yml @@ -7,6 +7,20 @@ when: "vars[item] == ''" with_items: - "matrix_synapse_macaroon_secret_key" + - "matrix_synapse_database_host" + - "matrix_synapse_database_user" + - "matrix_synapse_database_password" + - "matrix_synapse_database_database" + +- name: Fail if asking for more than 1 instance of single-instance workers + fail: + msg: >- + `{{ item }}` cannot be more than 1. This is a single-instance worker. + when: "vars[item]|int > 1" + with_items: + - "matrix_synapse_workers_appservice_workers_count" + - "matrix_synapse_workers_pusher_workers_count" + - "matrix_synapse_workers_federation_sender_workers_count" - name: (Deprecation) Catch and report renamed settings fail: diff --git a/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 b/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 index d96ab4a6..df4a4f23 100644 --- a/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 +++ b/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 @@ -16,7 +16,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name %n \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --mount type=bind,src=/etc/passwd,dst=/etc/passwd,ro \ --mount type=bind,src=/etc/group,dst=/etc/group,ro \ - --mount type=bind,src={{ matrix_synapse_media_store_path }},dst=/s3,bind-propagation=shared \ + --mount type=bind,src={{ matrix_s3_media_store_path }},dst=/s3,bind-propagation=shared \ --security-opt apparmor:unconfined \ --cap-add mknod \ --cap-add sys_admin \ @@ -30,7 +30,7 @@ TimeoutStartSec=5min ExecStop=-{{ matrix_host_command_docker }} stop %n ExecStop=-{{ matrix_host_command_docker }} kill %n ExecStop=-{{ matrix_host_command_docker }} rm %n -ExecStop=-{{ matrix_host_command_fusermount }} -u {{ matrix_synapse_media_store_path }} +ExecStop=-{{ matrix_host_command_fusermount }} -u {{ matrix_s3_media_store_path }} Restart=always RestartSec=5 SyslogIdentifier=matrix-goofys diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 index fe28779c..99169b8a 100644 --- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -43,11 +43,12 @@ pid_file: /homeserver.pid # #web_client_location: https://riot.example.com/ -# The public-facing base URL that clients use to access this HS -# (not including _matrix/...). This is the same URL a user would -# enter into the 'custom HS URL' field on their client. If you -# use synapse with a reverse proxy, this should be the URL to reach -# synapse via the proxy. +# The public-facing base URL that clients use to access this Homeserver (not +# including _matrix/...). This is the same URL a user might enter into the +# 'Custom Homeserver URL' field on their client. If you use Synapse with a +# reverse proxy, this should be the URL to reach Synapse via the proxy. +# Otherwise, it should be the URL to reach Synapse's client HTTP listener (see +# 'listeners' below). # public_baseurl: https://{{ matrix_server_fqn_matrix }}/ @@ -276,6 +277,41 @@ listeners: type: manhole {% endif %} +{% if matrix_synapse_workers_enabled %} + +{% if matrix_synapse_replication_listener_enabled %} + # c.f. https://github.com/matrix-org/synapse/tree/master/docs/workers.md + # HTTP replication: for the workers to send data to the main synapse process + - port: {{ matrix_synapse_replication_http_port }} + bind_addresses: ['0.0.0.0'] + type: http + resources: + - names: [replication] +{% endif %} + +# c.f. https://github.com/matrix-org/synapse/tree/master/contrib/systemd-with-workers/README.md +worker_app: synapse.app.homeserver + +# thx https://oznetnerd.com/2017/04/18/jinja2-selectattr-filter/ +# reduce the main worker's offerings to core homeserver business +{% if matrix_synapse_workers_enabled_list|selectattr('type', 'equalto', 'appservice')|list %} +notify_appservices: false +{% endif %} +{% if matrix_synapse_workers_enabled_list|selectattr('type', 'equalto', 'federation_sender')|list %} +send_federation: false +{% endif %} +{% if matrix_synapse_workers_enabled_list|selectattr('type', 'equalto', 'media_repository')|list %} +enable_media_repo: false +{% endif %} +{% if matrix_synapse_workers_enabled_list|selectattr('type', 'equalto', 'pusher')|list %} +start_pushers: false +{% endif %} +{% if matrix_synapse_workers_enabled_list|selectattr('type', 'equalto', 'user_dir')|list %} +update_user_directory: false +{% endif %} + +daemonize: false +{% endif %} # Forward extremities can build up in a room due to networking delays between # homeservers. Once this happens in a large room, calculation of the state of @@ -780,6 +816,9 @@ log_config: "/data/{{ matrix_server_fqn_matrix }}.log.config" # users are joining rooms the server is already in (this is cheap) vs # "remote" for when users are trying to join rooms not on the server (which # can be more expensive) +# - one for ratelimiting how often a user or IP can attempt to validate a 3PID. +# - two for ratelimiting how often invites can be sent in a room or to a +# specific user. # # The defaults are as shown below. # @@ -808,6 +847,7 @@ rc_login: {{ matrix_synapse_rc_login|to_json }} #rc_admin_redaction: # per_second: 1 # burst_count: 50 +rc_admin_redaction: {{ matrix_synapse_rc_admin_redaction|to_json }} # #rc_joins: # local: @@ -816,7 +856,19 @@ rc_login: {{ matrix_synapse_rc_login|to_json }} # remote: # per_second: 0.01 # burst_count: 3 - +rc_joins: {{ matrix_synapse_rc_joins|to_json }} +# +#rc_3pid_validation: +# per_second: 0.003 +# burst_count: 5 +# +#rc_invites: +# per_room: +# per_second: 0.3 +# burst_count: 10 +# per_user: +# per_second: 0.003 +# burst_count: 5 # Ratelimiting settings for incoming federation # @@ -1526,10 +1578,10 @@ trusted_key_servers: {{ matrix_synapse_trusted_key_servers|to_json }} # enable SAML login. # # Once SAML support is enabled, a metadata file will be exposed at -# https://:/_matrix/saml2/metadata.xml, which you may be able to +# https://:/_synapse/client/saml2/metadata.xml, which you may be able to # use to configure your SAML IdP with. Alternatively, you can manually configure # the IdP to use an ACS location of -# https://:/_matrix/saml2/authn_response. +# https://:/_synapse/client/saml2/authn_response. # saml2_config: # `sp_config` is the configuration for the pysaml2 Service Provider. @@ -1686,141 +1738,168 @@ saml2_config: #idp_entityid: 'https://our_idp/entityid' -# Enable OpenID Connect (OIDC) / OAuth 2.0 for registration and login. +# List of OpenID Connect (OIDC) / OAuth 2.0 identity providers, for registration +# and login. # -# See https://github.com/matrix-org/synapse/blob/master/docs/openid.md -# for some example configurations. +# Options for each entry include: # -oidc_config: - # Uncomment the following to enable authorization against an OpenID Connect - # server. Defaults to false. - # - #enabled: true - - # Uncomment the following to disable use of the OIDC discovery mechanism to - # discover endpoints. Defaults to true. - # - #discover: false - - # the OIDC issuer. Used to validate tokens and (if discovery is enabled) to - # discover the provider's endpoints. - # - # Required if 'enabled' is true. - # - #issuer: "https://accounts.example.com/" - - # oauth2 client id to use. - # - # Required if 'enabled' is true. - # - #client_id: "provided-by-your-issuer" - - # oauth2 client secret to use. - # - # Required if 'enabled' is true. - # - #client_secret: "provided-by-your-issuer" - - # auth method to use when exchanging the token. - # Valid values are 'client_secret_basic' (default), 'client_secret_post' and - # 'none'. - # - #client_auth_method: client_secret_post - - # list of scopes to request. This should normally include the "openid" scope. - # Defaults to ["openid"]. - # - #scopes: ["openid", "profile"] - - # the oauth2 authorization endpoint. Required if provider discovery is disabled. - # - #authorization_endpoint: "https://accounts.example.com/oauth2/auth" - - # the oauth2 token endpoint. Required if provider discovery is disabled. - # - #token_endpoint: "https://accounts.example.com/oauth2/token" - - # the OIDC userinfo endpoint. Required if discovery is disabled and the - # "openid" scope is not requested. - # - #userinfo_endpoint: "https://accounts.example.com/userinfo" - - # URI where to fetch the JWKS. Required if discovery is disabled and the - # "openid" scope is used. - # - #jwks_uri: "https://accounts.example.com/.well-known/jwks.json" - - # Uncomment to skip metadata verification. Defaults to false. - # - # Use this if you are connecting to a provider that is not OpenID Connect - # compliant. - # Avoid this in production. - # - #skip_verification: true - - # Whether to fetch the user profile from the userinfo endpoint. Valid - # values are: "auto" or "userinfo_endpoint". - # - # Defaults to "auto", which fetches the userinfo endpoint if "openid" is included - # in `scopes`. Uncomment the following to always fetch the userinfo endpoint. - # - #user_profile_method: "userinfo_endpoint" - - # Uncomment to allow a user logging in via OIDC to match a pre-existing account instead - # of failing. This could be used if switching from password logins to OIDC. Defaults to false. - # - #allow_existing_users: true - - # An external module can be provided here as a custom solution to mapping - # attributes returned from a OIDC provider onto a matrix user. - # - user_mapping_provider: - # The custom module's class. Uncomment to use a custom module. - # Default is 'synapse.handlers.oidc_handler.JinjaOidcMappingProvider'. - # - # See https://github.com/matrix-org/synapse/blob/master/docs/sso_mapping_providers.md#openid-mapping-providers - # for information on implementing a custom mapping provider. - # - #module: mapping_provider.OidcMappingProvider - - # Custom configuration values for the module. This section will be passed as - # a Python dictionary to the user mapping provider module's `parse_config` - # method. - # - # The examples below are intended for the default provider: they should be - # changed if using a custom provider. - # - config: - # name of the claim containing a unique identifier for the user. - # Defaults to `sub`, which OpenID Connect compliant providers should provide. - # - #subject_claim: "sub" - - # Jinja2 template for the localpart of the MXID. - # - # When rendering, this template is given the following variables: - # * user: The claims returned by the UserInfo Endpoint and/or in the ID - # Token - # - # If this is not set, the user will be prompted to choose their - # own username. - # - localpart_template: "{% raw %}{{ user.preferred_username }}{% endraw %}" - - # Jinja2 template for the display name to set on first login. - # - # If unset, no displayname will be set. - # - #display_name_template: "{% raw %}{{ user.given_name }} {{ user.last_name }}{% endraw %}" - - # Jinja2 templates for extra attributes to send back to the client during - # login. - # - # Note that these are non-standard and clients will ignore them without modifications. - # - #extra_attributes: - #birthdate: "{% raw %}{{ user.birthdate }}{% endraw %}" - +# idp_id: a unique identifier for this identity provider. Used internally +# by Synapse; should be a single word such as 'github'. +# +# Note that, if this is changed, users authenticating via that provider +# will no longer be recognised as the same user! +# +# idp_name: A user-facing name for this identity provider, which is used to +# offer the user a choice of login mechanisms. +# +# idp_icon: An optional icon for this identity provider, which is presented +# by identity picker pages. If given, must be an MXC URI of the format +# mxc:///. (An easy way to obtain such an MXC URI +# is to upload an image to an (unencrypted) room and then copy the "url" +# from the source of the event.) +# +# discover: set to 'false' to disable the use of the OIDC discovery mechanism +# to discover endpoints. Defaults to true. +# +# issuer: Required. The OIDC issuer. Used to validate tokens and (if discovery +# is enabled) to discover the provider's endpoints. +# +# client_id: Required. oauth2 client id to use. +# +# client_secret: Required. oauth2 client secret to use. +# +# client_auth_method: auth method to use when exchanging the token. Valid +# values are 'client_secret_basic' (default), 'client_secret_post' and +# 'none'. +# +# scopes: list of scopes to request. This should normally include the "openid" +# scope. Defaults to ["openid"]. +# +# authorization_endpoint: the oauth2 authorization endpoint. Required if +# provider discovery is disabled. +# +# token_endpoint: the oauth2 token endpoint. Required if provider discovery is +# disabled. +# +# userinfo_endpoint: the OIDC userinfo endpoint. Required if discovery is +# disabled and the 'openid' scope is not requested. +# +# jwks_uri: URI where to fetch the JWKS. Required if discovery is disabled and +# the 'openid' scope is used. +# +# skip_verification: set to 'true' to skip metadata verification. Use this if +# you are connecting to a provider that is not OpenID Connect compliant. +# Defaults to false. Avoid this in production. +# +# user_profile_method: Whether to fetch the user profile from the userinfo +# endpoint. Valid values are: 'auto' or 'userinfo_endpoint'. +# +# Defaults to 'auto', which fetches the userinfo endpoint if 'openid' is +# included in 'scopes'. Set to 'userinfo_endpoint' to always fetch the +# userinfo endpoint. +# +# allow_existing_users: set to 'true' to allow a user logging in via OIDC to +# match a pre-existing account instead of failing. This could be used if +# switching from password logins to OIDC. Defaults to false. +# +# user_mapping_provider: Configuration for how attributes returned from a OIDC +# provider are mapped onto a matrix user. This setting has the following +# sub-properties: +# +# module: The class name of a custom mapping module. Default is +# 'synapse.handlers.oidc_handler.JinjaOidcMappingProvider'. +# See https://github.com/matrix-org/synapse/blob/master/docs/sso_mapping_providers.md#openid-mapping-providers +# for information on implementing a custom mapping provider. +# +# config: Configuration for the mapping provider module. This section will +# be passed as a Python dictionary to the user mapping provider +# module's `parse_config` method. +# +# For the default provider, the following settings are available: +# +# subject_claim: name of the claim containing a unique identifier +# for the user. Defaults to 'sub', which OpenID Connect +# compliant providers should provide. +# +# localpart_template: Jinja2 template for the localpart of the MXID. +# If this is not set, the user will be prompted to choose their +# own username (see 'sso_auth_account_details.html' in the 'sso' +# section of this file). +# +# display_name_template: Jinja2 template for the display name to set +# on first login. If unset, no displayname will be set. +# +# email_template: Jinja2 template for the email address of the user. +# If unset, no email address will be added to the account. +# +# extra_attributes: a map of Jinja2 templates for extra attributes +# to send back to the client during login. +# Note that these are non-standard and clients will ignore them +# without modifications. +# +# When rendering, the Jinja2 templates are given a 'user' variable, +# which is set to the claims returned by the UserInfo Endpoint and/or +# in the ID Token. +# +# See https://github.com/matrix-org/synapse/blob/master/docs/openid.md +# for information on how to configure these options. +# +# For backwards compatibility, it is also possible to configure a single OIDC +# provider via an 'oidc_config' setting. This is now deprecated and admins are +# advised to migrate to the 'oidc_providers' format. (When doing that migration, +# use 'oidc' for the idp_id to ensure that existing users continue to be +# recognised.) +# +oidc_providers: + # Generic example + # + #- idp_id: my_idp + # idp_name: "My OpenID provider" + # idp_icon: "mxc://example.com/mediaid" + # discover: false + # issuer: "https://accounts.example.com/" + # client_id: "provided-by-your-issuer" + # client_secret: "provided-by-your-issuer" + # client_auth_method: client_secret_post + # scopes: ["openid", "profile"] + # authorization_endpoint: "https://accounts.example.com/oauth2/auth" + # token_endpoint: "https://accounts.example.com/oauth2/token" + # userinfo_endpoint: "https://accounts.example.com/userinfo" + # jwks_uri: "https://accounts.example.com/.well-known/jwks.json" + # user_mapping_provider: + # config: + # subject_claim: "id" + # localpart_template: "{ user.login }" + # display_name_template: "{ user.name }" + # email_template: "{ user.email }" + + # For use with Keycloak + # + #- idp_id: keycloak + # idp_name: Keycloak + # issuer: "https://127.0.0.1:8443/auth/realms/my_realm_name" + # client_id: "synapse" + # client_secret: "copy secret generated in Keycloak UI" + # scopes: ["openid", "profile"] + + # For use with Github + # + #- idp_id: github + # idp_name: Github + # idp_brand: org.matrix.github + # discover: false + # issuer: "https://github.com/" + # client_id: "your-client-id" # TO BE FILLED + # client_secret: "your-client-secret" # TO BE FILLED + # authorization_endpoint: "https://github.com/login/oauth/authorize" + # token_endpoint: "https://github.com/login/oauth/access_token" + # userinfo_endpoint: "https://api.github.com/user" + # scopes: ["read:user"] + # user_mapping_provider: + # config: + # subject_claim: "id" + # localpart_template: "{ user.login }" + # display_name_template: "{ user.name }" # Enable Central Authentication Service (CAS) for registration and login. @@ -1835,10 +1914,6 @@ cas_config: # #server_url: "https://cas-server.com" - # The public URL of the homeserver. - # - #service_url: "https://homeserver.domain.com:8448" - # The attribute of the CAS response to use as the display name. # # If unset, no displayname will be set. @@ -1886,33 +1961,130 @@ sso: # # Synapse will look for the following templates in this directory: # + # * HTML page to prompt the user to choose an Identity Provider during + # login: 'sso_login_idp_picker.html'. + # + # This is only used if multiple SSO Identity Providers are configured. + # + # When rendering, this template is given the following variables: + # * redirect_url: the URL that the user will be redirected to after + # login. + # + # * server_name: the homeserver's name. + # + # * providers: a list of available Identity Providers. Each element is + # an object with the following attributes: + # + # * idp_id: unique identifier for the IdP + # * idp_name: user-facing name for the IdP + # * idp_icon: if specified in the IdP config, an MXC URI for an icon + # for the IdP + # * idp_brand: if specified in the IdP config, a textual identifier + # for the brand of the IdP + # + # The rendered HTML page should contain a form which submits its results + # back as a GET request, with the following query parameters: + # + # * redirectUrl: the client redirect URI (ie, the `redirect_url` passed + # to the template) + # + # * idp: the 'idp_id' of the chosen IDP. + # + # * HTML page to prompt new users to enter a userid and confirm other + # details: 'sso_auth_account_details.html'. This is only shown if the + # SSO implementation (with any user_mapping_provider) does not return + # a localpart. + # + # When rendering, this template is given the following variables: + # + # * server_name: the homeserver's name. + # + # * idp: details of the SSO Identity Provider that the user logged in + # with: an object with the following attributes: + # + # * idp_id: unique identifier for the IdP + # * idp_name: user-facing name for the IdP + # * idp_icon: if specified in the IdP config, an MXC URI for an icon + # for the IdP + # * idp_brand: if specified in the IdP config, a textual identifier + # for the brand of the IdP + # + # * user_attributes: an object containing details about the user that + # we received from the IdP. May have the following attributes: + # + # * display_name: the user's display_name + # * emails: a list of email addresses + # + # The template should render a form which submits the following fields: + # + # * username: the localpart of the user's chosen user id + # + # * HTML page allowing the user to consent to the server's terms and + # conditions. This is only shown for new users, and only if + # `user_consent.require_at_registration` is set. + # + # When rendering, this template is given the following variables: + # + # * server_name: the homeserver's name. + # + # * user_id: the user's matrix proposed ID. + # + # * user_profile.display_name: the user's proposed display name, if any. + # + # * consent_version: the version of the terms that the user will be + # shown + # + # * terms_url: a link to the page showing the terms. + # + # The template should render a form which submits the following fields: + # + # * accepted_version: the version of the terms accepted by the user + # (ie, 'consent_version' from the input variables). + # # * HTML page for a confirmation step before redirecting back to the client # with the login token: 'sso_redirect_confirm.html'. # - # When rendering, this template is given three variables: - # * redirect_url: the URL the user is about to be redirected to. Needs - # manual escaping (see - # https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping). + # When rendering, this template is given the following variables: + # + # * redirect_url: the URL the user is about to be redirected to. # # * display_url: the same as `redirect_url`, but with the query # parameters stripped. The intention is to have a # human-readable URL to show to users, not to use it as - # the final address to redirect to. Needs manual escaping - # (see https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping). + # the final address to redirect to. # # * server_name: the homeserver's name. # + # * new_user: a boolean indicating whether this is the user's first time + # logging in. + # + # * user_id: the user's matrix ID. + # + # * user_profile.avatar_url: an MXC URI for the user's avatar, if any. + # None if the user has not set an avatar. + # + # * user_profile.display_name: the user's display name. None if the user + # has not set a display name. + # # * HTML page which notifies the user that they are authenticating to confirm # an operation on their account during the user interactive authentication # process: 'sso_auth_confirm.html'. # # When rendering, this template is given the following variables: - # * redirect_url: the URL the user is about to be redirected to. Needs - # manual escaping (see - # https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping). + # * redirect_url: the URL the user is about to be redirected to. # # * description: the operation which the user is being asked to confirm # + # * idp: details of the Identity Provider that we will use to confirm + # the user's identity: an object with the following attributes: + # + # * idp_id: unique identifier for the IdP + # * idp_name: user-facing name for the IdP + # * idp_icon: if specified in the IdP config, an MXC URI for an icon + # for the IdP + # * idp_brand: if specified in the IdP config, a textual identifier + # for the brand of the IdP + # # * HTML page shown after a successful user interactive authentication session: # 'sso_auth_success.html'. # @@ -1921,6 +2093,14 @@ sso: # # This template has no additional variables. # + # * HTML page shown after a user-interactive authentication session which + # does not map correctly onto the expected user: 'sso_auth_bad_user.html'. + # + # When rendering, this template is given the following variables: + # * server_name: the homeserver's name. + # * user_id_to_verify: the MXID of the user that we are trying to + # validate. + # # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database) # attempts to login: 'sso_account_deactivated.html'. # @@ -2667,16 +2847,16 @@ opentracing: redis: # Uncomment the below to enable Redis support. # - #enabled: true + enabled: {{ matrix_synapse_redis_enabled }} # Optional host and port to use to connect to redis. Defaults to # localhost and 6379 # - #host: localhost - #port: 6379 + host: {{ matrix_synapse_redis_host }} + port: {{ matrix_synapse_redis_port }} # Optional password if configured on the Redis instance # - #password: + password: {{ matrix_synapse_redis_password }} # vim:ft=yaml diff --git a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 new file mode 100644 index 00000000..0f5e7be2 --- /dev/null +++ b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 @@ -0,0 +1,58 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description=Synapse worker ({{ matrix_synapse_worker_container_name }}) +AssertPathExists={{ matrix_synapse_config_dir_path }}/{{ matrix_synapse_worker_config_file_name }} +After=matrix-synapse.service + +[Service] +Type=simple +Environment="HOME={{ matrix_systemd_unit_home_path }}" + +ExecStartPre=-{{ matrix_host_command_docker }} kill {{ matrix_synapse_worker_container_name }} +ExecStartPre=-{{ matrix_host_command_docker }} rm {{ matrix_synapse_worker_container_name }} + +# Intentional delay, so that the homeserver can manage to start. +ExecStartPre={{ matrix_host_command_sleep }} 5 + +ExecStart={{ matrix_host_command_docker }} run --rm --name {{ matrix_synapse_worker_container_name }} \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --entrypoint=python \ + --read-only \ + --tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_synapse_tmp_directory_size_mb }}m \ + --network={{ matrix_docker_network }} \ + {% if matrix_synapse_workers_enabled and matrix_synapse_workers_container_host_bind_address %} + {% if matrix_synapse_worker_details.port != 0 %} + -p {{ '' if matrix_synapse_workers_container_host_bind_address == '*' else (matrix_synapse_workers_container_host_bind_address + ':') }}{{ matrix_synapse_worker_details.port }}:{{ matrix_synapse_worker_details.port }} \ + {% endif %} + {% if matrix_synapse_worker_details.metrics_port != 0 %} + -p {{ '' if matrix_synapse_workers_container_host_bind_address == '*' else (matrix_synapse_workers_container_host_bind_address + ':') }}{{ matrix_synapse_worker_details.metrics_port }}:{{ matrix_synapse_worker_details.metrics_port }} \ + {% endif %} + {% endif %} + --mount type=bind,src={{ matrix_synapse_config_dir_path }},dst=/data,ro \ + --mount type=bind,src={{ matrix_synapse_storage_path }},dst=/matrix-media-store-parent,bind-propagation=slave \ + {% for volume in matrix_synapse_container_additional_volumes %} + -v {{ volume.src }}:{{ volume.dst }}:{{ volume.options }} \ + {% endfor %} + {% for arg in matrix_synapse_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_synapse_docker_image }} \ + -m synapse.app.{{ matrix_synapse_worker_details.type }} -c /data/homeserver.yaml -c /data/{{ matrix_synapse_worker_config_file_name }} + + +ExecStop=-{{ matrix_host_command_docker }} kill {{ matrix_synapse_worker_container_name }} +ExecStop=-{{ matrix_host_command_docker }} rm {{ matrix_synapse_worker_container_name }} + +ExecReload={{ matrix_host_command_docker }} exec {{ matrix_synapse_worker_container_name }} /bin/sh -c 'kill -HUP 1' +Restart=always +RestartSec=30 +SyslogIdentifier={{ matrix_synapse_worker_container_name }} + +# Intentionally not making this WantedBy=matrix-synapse.service, +# as matrix.synapse.service already has `Wants=` lines. +# Also, WantedBy will trigger the creation of some `matrix-synapse.service.wants/` directory, +# which we'd have to clean, etc. Better not. +[Install] +WantedBy=multi-user.target diff --git a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 index 88789908..5f69e315 100644 --- a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 +++ b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 @@ -4,17 +4,25 @@ Description=Synapse server {% for service in matrix_synapse_systemd_required_services_list %} Requires={{ service }} After={{ service }} + {% endfor %} {% for service in matrix_synapse_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} + +{% if matrix_synapse_workers_enabled %} +{% for matrix_synapse_worker_details in matrix_synapse_workers_enabled_list %} +Wants=matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.port }}.service +{% endfor %} +{% endif %} + DefaultDependencies=no [Service] Type=simple Environment="HOME={{ matrix_systemd_unit_home_path }}" -ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-synapse -ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-synapse +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-synapse 2>/dev/null' +ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-synapse 2>/dev/null' {% if matrix_s3_media_store_enabled %} # Allow for some time before starting, so that media store can mount. # Mounting can happen later too, but if we start writing, @@ -56,9 +64,9 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-synapse \ {{ matrix_synapse_docker_image }} \ -m synapse.app.homeserver -c /data/homeserver.yaml -ExecStop=-{{ matrix_host_command_docker }} kill matrix-synapse -ExecStop=-{{ matrix_host_command_docker }} rm matrix-synapse -ExecReload={{ matrix_host_command_docker }} exec matrix-synapse kill -HUP 1 +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-synapse 2>/dev/null' +ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-synapse 2>/dev/null' +ExecReload={{ matrix_host_command_docker }} exec matrix-synapse /bin/sh -c 'kill -HUP 1' Restart=always RestartSec=30 SyslogIdentifier=matrix-synapse diff --git a/roles/matrix-synapse/templates/synapse/worker.yaml.j2 b/roles/matrix-synapse/templates/synapse/worker.yaml.j2 new file mode 100644 index 00000000..36ae5a7e --- /dev/null +++ b/roles/matrix-synapse/templates/synapse/worker.yaml.j2 @@ -0,0 +1,45 @@ +#jinja2: lstrip_blocks: "True" +worker_app: synapse.app.{{ matrix_synapse_worker_details.type }} +worker_name: {{ matrix_synapse_worker_details.type ~ ':' ~ matrix_synapse_worker_details.port }} + +{% if matrix_synapse_replication_listener_enabled %} +worker_replication_host: matrix-synapse +worker_replication_http_port: {{ matrix_synapse_replication_http_port }} +{% endif %} + +{% set has_listeners = (matrix_synapse_worker_details.type not in [ 'appservice', 'federation_sender', 'pusher' ] or matrix_synapse_metrics_enabled) %} + +{% set http_resources = [] %} + +{% if matrix_synapse_worker_details.type in ['generic_worker', 'frontend_proxy', 'user_dir'] %} + {% set http_resources = http_resources + ['client'] %} +{% endif %} +{% if matrix_synapse_worker_details.type in ['generic_worker'] %} + {% set http_resources = http_resources+ ['federation'] %} +{% endif %} +{% if matrix_synapse_worker_details.type in ['media_repository'] %} + {% set http_resources = http_resources + ['media'] %} +{% endif %} + +{% if http_resources|length > 0 or matrix_synapse_metrics_enabled %} +worker_listeners: +{% if http_resources|length > 0 %} + - type: http + bind_addresses: ['::'] + port: {{ matrix_synapse_worker_details.port }} + resources: + - names: {{ http_resources|to_json }} +{% endif %} +{% if matrix_synapse_metrics_enabled %} + - type: metrics + bind_addresses: ['0.0.0.0'] + port: {{ matrix_synapse_worker_details.metrics_port }} +{% endif %} +{% endif %} + +{% if matrix_synapse_worker_details.type == 'frontend_proxy' %} +worker_main_http_uri: http://matrix-synapse:8008 +{% endif %} + +worker_daemonize: false +worker_log_config: /data/{{ matrix_server_fqn_matrix }}.log.config diff --git a/roles/matrix-synapse/vars/main.yml b/roles/matrix-synapse/vars/main.yml index 7c07145b..9c6d8ce4 100644 --- a/roles/matrix-synapse/vars/main.yml +++ b/roles/matrix-synapse/vars/main.yml @@ -8,3 +8,28 @@ matrix_synapse_role_executed: false matrix_synapse_media_store_parent_path: "{{ matrix_synapse_media_store_path|dirname }}" matrix_synapse_media_store_directory_name: "{{ matrix_synapse_media_store_path|basename }}" + +# A Synapse generic worker can handle both federation and client-server API endpoints. +# We wish to split these, as we normally serve federation separately and don't want them mixed up. +# +# This is some ugly Ansible/Jinja2 hack (seen here: https://stackoverflow.com/a/47831492), +# which takes a list of various strings and removes the ones NOT containing `/_matrix/client` anywhere in them. +# +# We intentionally don't do a diff between everything possible (`matrix_synapse_workers_generic_worker_endpoints`) and `matrix_synapse_workers_generic_worker_federation_endpoints`, +# because `matrix_synapse_workers_generic_worker_endpoints` also contains things like `/_synapse/client/`, etc. +# While /_synapse/client/ endpoints are somewhat client-server API-related, they're: +# - neither part of the client-server API spec (and are thus, different) +# - nor always OK to forward to a worker (we're supposed to obey `matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_client_api_enabled`) +# +# It's also not too many of these APIs (only `^/_synapse/client/password_reset/email/submit_token$` at the time of this writing / 2021-01-24), +# so it's not that important whether we forward them or not. +# +# Basically, we aim to cover most things. Skipping `/_synapse/client` or a few other minor things doesn't matter too much. +matrix_synapse_workers_generic_worker_client_server_endpoints: "{{ matrix_synapse_workers_generic_worker_endpoints|default([]) | map('regex_search', '.*/_matrix/client.*')| list | difference([none]) }}" + +# A Synapse generic worker can handle both federation and client-server API endpoints. +# We wish to split these, as we normally serve federation separately and don't want them mixed up. +# +# This is some ugly Ansible/Jinja2 hack (seen here: https://stackoverflow.com/a/47831492), +# which takes a list of various strings and removes the ones NOT containing `/_matrix/federation` or `/_matrix/key` anywhere in them. +matrix_synapse_workers_generic_worker_federation_endpoints: "{{ matrix_synapse_workers_generic_worker_endpoints|default([]) | map('regex_search', '.*(/_matrix/federation|/_matrix/key).*')| list | difference([none]) }}" diff --git a/roles/matrix-synapse/vars/workers.yml b/roles/matrix-synapse/vars/workers.yml new file mode 100644 index 00000000..3adfd9c3 --- /dev/null +++ b/roles/matrix-synapse/vars/workers.yml @@ -0,0 +1,313 @@ +--- + +matrix_synapse_workers_generic_worker_endpoints: + # This worker can handle API requests matching the following regular + # expressions: + + # Sync requests + - ^/_matrix/client/(v2_alpha|r0)/sync$ + - ^/_matrix/client/(api/v1|v2_alpha|r0)/events$ + - ^/_matrix/client/(api/v1|r0)/initialSync$ + - ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$ + + # Federation requests + - ^/_matrix/federation/v1/event/ + - ^/_matrix/federation/v1/state/ + - ^/_matrix/federation/v1/state_ids/ + - ^/_matrix/federation/v1/backfill/ + - ^/_matrix/federation/v1/get_missing_events/ + - ^/_matrix/federation/v1/publicRooms + - ^/_matrix/federation/v1/query/ + - ^/_matrix/federation/v1/make_join/ + - ^/_matrix/federation/v1/make_leave/ + - ^/_matrix/federation/v1/send_join/ + - ^/_matrix/federation/v2/send_join/ + - ^/_matrix/federation/v1/send_leave/ + - ^/_matrix/federation/v2/send_leave/ + - ^/_matrix/federation/v1/invite/ + - ^/_matrix/federation/v2/invite/ + - ^/_matrix/federation/v1/query_auth/ + - ^/_matrix/federation/v1/event_auth/ + - ^/_matrix/federation/v1/exchange_third_party_invite/ + - ^/_matrix/federation/v1/user/devices/ + - ^/_matrix/federation/v1/get_groups_publicised$ + - ^/_matrix/key/v2/query + + # Inbound federation transaction request + - ^/_matrix/federation/v1/send/ + + # Client API requests + - ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$ + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$ + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$ + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ + - ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$ + - ^/_matrix/client/(api/v1|r0|unstable)/devices$ + - ^/_matrix/client/(api/v1|r0|unstable)/keys/query$ + - ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$ + - ^/_matrix/client/versions$ + - ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$ + - ^/_matrix/client/(api/v1|r0|unstable)/joined_groups$ + - ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$ + - ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/ + + # Registration/login requests + - ^/_matrix/client/(api/v1|r0|unstable)/login$ + - ^/_matrix/client/(r0|unstable)/register$ + # FIXME: possible bug with SSO and multiple generic workers + # see https://github.com/matrix-org/synapse/issues/7530 + # ^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$ + + # Event sending requests + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/ + - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ + - ^/_matrix/client/(api/v1|r0|unstable)/join/ + - ^/_matrix/client/(api/v1|r0|unstable)/profile/ + + + # Additionally, the following REST endpoints can be handled for GET requests: + + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_matrix/federation/v1/groups/ + + # Pagination requests can also be handled, but all requests for a given + # room must be routed to the same instance. Additionally, care must be taken to + # ensure that the purge history admin API is not used while pagination requests + # for the room are in flight: + + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$ + + # Additionally, the following endpoints should be included if Synapse is configured + # to use SSO (you only need to include the ones for whichever SSO provider you're + # using): + + # for all SSO providers + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_matrix/client/(api/v1|r0|unstable)/login/sso/redirect + # ^/_synapse/client/pick_idp$ + # ^/_synapse/client/pick_username + # ^/_synapse/client/new_user_consent$ + # ^/_synapse/client/sso_register$ + + # OpenID Connect requests. + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_synapse/client/oidc/callback$ + + # SAML requests. + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_synapse/client/saml2/authn_response$ + + # CAS requests. + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_matrix/client/(api/v1|r0|unstable)/login/cas/ticket$ + + # Ensure that all SSO logins go to a single process. + # For multiple workers not handling the SSO endpoints properly, see + # [#7530](https://github.com/matrix-org/synapse/issues/7530). + + # Note that a HTTP listener with `client` and `federation` resources must be + # configured in the `worker_listeners` option in the worker config. + + # #### Load balancing + + # It is possible to run multiple instances of this worker app, with incoming requests + # being load-balanced between them by the reverse-proxy. However, different endpoints + # have different characteristics and so admins + # may wish to run multiple groups of workers handling different endpoints so that + # load balancing can be done in different ways. + + # For `/sync` and `/initialSync` requests it will be more efficient if all + # requests from a particular user are routed to a single instance. Extracting a + # user ID from the access token or `Authorization` header is currently left as an + # exercise for the reader. Admins may additionally wish to separate out `/sync` + # requests that have a `since` query parameter from those that don't (and + # `/initialSync`), as requests that don't are known as "initial sync" that happens + # when a user logs in on a new device and can be *very* resource intensive, so + # isolating these requests will stop them from interfering with other users ongoing + # syncs. + + # Federation and client requests can be balanced via simple round robin. + + # The inbound federation transaction request `^/_matrix/federation/v1/send/` + # should be balanced by source IP so that transactions from the same remote server + # go to the same process. + + # Registration/login requests can be handled separately purely to help ensure that + # unexpected load doesn't affect new logins and sign ups. + + # Finally, event sending requests can be balanced by the room ID in the URI (or + # the full URI, or even just round robin), the room ID is the path component after + # `/rooms/`. If there is a large bridge connected that is sending or may send lots + # of events, then a dedicated set of workers can be provisioned to limit the + # effects of bursts of events from that bridge on events sent by normal users. + + # #### Stream writers + + # Additionally, there is *experimental* support for moving writing of specific + # streams (such as events) off of the main process to a particular worker. (This + # is only supported with Redis-based replication.) + + # Currently supported streams are `events` and `typing`. + + # To enable this, the worker must have a HTTP replication listener configured, + # have a `worker_name` and be listed in the `instance_map` config. For example to + # move event persistence off to a dedicated worker, the shared configuration would + # include: + + # ```yaml + # instance_map: + # event_persister1: + # host: localhost + # port: 8034 + + # stream_writers: + # events: event_persister1 + # ``` + + # The `events` stream also experimentally supports having multiple writers, where + # work is sharded between them by room ID. Note that you *must* restart all worker + # instances when adding or removing event persisters. An example `stream_writers` + # configuration with multiple writers: + + # ```yaml + # stream_writers: + # events: + # - event_persister1 + # - event_persister2 + # ``` + + # #### Background tasks + + # There is also *experimental* support for moving background tasks to a separate + # worker. Background tasks are run periodically or started via replication. Exactly + # which tasks are configured to run depends on your Synapse configuration (e.g. if + # stats is enabled). + + # To enable this, the worker must have a `worker_name` and can be configured to run + # background tasks. For example, to move background tasks to a dedicated worker, + # the shared configuration would include: + + # ```yaml + # run_background_tasks_on: background_worker + # ``` + + # You might also wish to investigate the `update_user_directory` and + # `media_instance_running_background_jobs` settings. + +# pusher worker (no API endpoints) [ + # Handles sending push notifications to sygnal and email. Doesn't handle any + # REST endpoints itself, but you should set `start_pushers: False` in the + # shared configuration file to stop the main synapse sending push notifications. + + # Note this worker cannot be load-balanced: only one instance should be active. +# ] + +# appservice worker (no API endpoints) [ + # Handles sending output traffic to Application Services. Doesn't handle any + # REST endpoints itself, but you should set `notify_appservices: False` in the + # shared configuration file to stop the main synapse sending appservice notifications. + + # Note this worker cannot be load-balanced: only one instance should be active. + +# ] + +# federation_sender worker (no API endpoints) [ + # Handles sending federation traffic to other servers. Doesn't handle any + # REST endpoints itself, but you should set `send_federation: False` in the + # shared configuration file to stop the main synapse sending this traffic. + + # If running multiple federation senders then you must list each + # instance in the `federation_sender_instances` option by their `worker_name`. + # All instances must be stopped and started when adding or removing instances. + # For example: + + # ```yaml + # federation_sender_instances: + # - federation_sender1 + # - federation_sender2 + # ``` +# ] + +matrix_synapse_workers_media_repository_endpoints: + # Handles the media repository. It can handle all endpoints starting with: + + - ^/_matrix/media/ + + # ... and the following regular expressions matching media-specific administration APIs: + + - ^/_synapse/admin/v1/purge_media_cache$ + - ^/_synapse/admin/v1/room/.*/media.*$ + - ^/_synapse/admin/v1/user/.*/media.*$ + - ^/_synapse/admin/v1/media/.*$ + - ^/_synapse/admin/v1/quarantine_media/.*$ + + # You should also set `enable_media_repo: False` in the shared configuration + # file to stop the main synapse running background jobs related to managing the + # media repository. + + # In the `media_repository` worker configuration file, configure the http listener to + # expose the `media` resource. For example: + + # ```yaml + # worker_listeners: + # - type: http + # port: 8085 + # resources: + # - names: + # - media + # ``` + + # Note that if running multiple media repositories they must be on the same server + # and you must configure a single instance to run the background tasks, e.g.: + + # ```yaml + # media_instance_running_background_jobs: "media-repository-1" + # ``` + + # Note that if a reverse proxy is used , then `/_matrix/media/` must be routed for both inbound client and federation requests (if they are handled separately). + +matrix_synapse_workers_user_dir_endpoints: + # Handles searches in the user directory. It can handle REST endpoints matching + # the following regular expressions: + + - ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$ + + # When using this worker you must also set `update_user_directory: False` in the + # shared configuration file to stop the main synapse running background + # jobs related to updating the user directory. + +matrix_synapse_workers_frontend_proxy_endpoints: + # Proxies some frequently-requested client endpoints to add caching and remove + # load from the main synapse. It can handle REST endpoints matching the following + # regular expressions: + + - ^/_matrix/client/(api/v1|r0|unstable)/keys/upload + + # If `use_presence` is False in the homeserver config, it can also handle REST + # endpoints matching the following regular expressions: + + # FIXME: ADDITIONAL CONDITIONS REQUIRED: to be enabled manually + # ^/_matrix/client/(api/v1|r0|unstable)/presence/[^/]+/status + + # This "stub" presence handler will pass through `GET` request but make the + # `PUT` effectively a no-op. + + # It will proxy any requests it cannot handle to the main synapse instance. It + # must therefore be configured with the location of the main instance, via + # the `worker_main_http_uri` setting in the `frontend_proxy` worker configuration + # file. For example: + + # worker_main_http_uri: http://127.0.0.1:8008 + +matrix_synapse_workers_avail_list: + - appservice + - federation_sender + - frontend_proxy + - generic_worker + - media_repository + - pusher + - user_dir diff --git a/setup.yml b/setup.yml index d070bcae..d9f25ecf 100755 --- a/setup.yml +++ b/setup.yml @@ -3,11 +3,15 @@ hosts: "{{ target if target is defined else 'matrix_servers' }}" become: true + vars_files: + - roles/matrix-synapse/vars/workers.yml + roles: - matrix-base - matrix-dynamic-dns - matrix-mailer - matrix-postgres + - matrix-redis - matrix-corporal - matrix-bridge-appservice-discord - matrix-bridge-appservice-slack @@ -15,10 +19,12 @@ - matrix-bridge-appservice-irc - matrix-bridge-mautrix-facebook - matrix-bridge-mautrix-hangouts + - matrix-bridge-mautrix-instagram - matrix-bridge-mautrix-signal - matrix-bridge-mautrix-telegram - matrix-bridge-mautrix-whatsapp - matrix-bridge-mx-puppet-discord + - matrix-bridge-mx-puppet-groupme - matrix-bridge-mx-puppet-steam - matrix-bridge-mx-puppet-skype - matrix-bridge-mx-puppet-slack @@ -28,11 +34,15 @@ - matrix-bot-matrix-reminder-bot - matrix-synapse - matrix-synapse-admin + - matrix-prometheus-node-exporter + - matrix-prometheus + - matrix-grafana - matrix-registration - matrix-client-element - matrix-jitsi - matrix-ma1sd - matrix-dimension + - matrix-etherpad - matrix-email2matrix - matrix-nginx-proxy - matrix-coturn