Merge branch 'master' into postgres-per-default

master
Slavi Pantaleev 3 years ago
commit 77a5c7cf3c

@ -1,3 +1,22 @@
# 2020-12-11
## synapse-janitor support removed
We've removed support for the unmaintained [synapse-janitor](https://github.com/xwiki-labs/synapse_scripts) script. There's been past reports of it corrupting the Synapse database. Since there hasn't been any new development on it and it doesn't seem too useful nowadays, there's no point in including it in the playbook.
If you need to clean up or compact your database, consider using the Synapse Admin APIs directly. See our [Synapse maintenance](docs/maintenance-synapse.md) and [Postgres maintenance](docs/maintenance-postgres.md) documentation pages for more details.
## Docker 20.10 is here
(No need to do anything special in relation to this. Just something to keep in mind)
Docker 20.10 got released recently and your server will likely get it the next time you update.
This is the first major Docker update in a long time and it packs a lot of changes.
Some of them introduced some breakage for us initially (see [here](https://github.com/spantaleev/matrix-docker-ansible-deploy/commit/d08b27784f222effcbce2abf924bf07bbe0893be) and [here](https://github.com/spantaleev/matrix-docker-ansible-deploy/commit/7593d969e316cc0144bce378a5be58c76c2c37ee)), but it should be all good now.
# 2020-12-08
## openid APIs exposed by default on the federation port when federation disabled

@ -4,14 +4,11 @@ This document shows you how to perform various maintenance tasks related to the
Table of contents:
- [Purging unused data with synapse-janitor](#purging-unused-data-with-synapse-janitor), for when you wish to delete unused data from the Synapse database
- [Purging old data with the Purge History API](#purging-old-data-with-the-purge-history-api), for when you wish to delete in-use (but old) data from the Synapse database
- [Synapse maintenance](#synapse-maintenance)
- [Purging old data with the Purge History API](#purging-old-data-with-the-purge-history-api)
- [Compressing state with rust-synapse-compress-state](#compressing-state-with-rust-synapse-compress-state)
- [Purging unused data with synapse-janitor](#purging-unused-data-with-synapse-janitor)
- [Browse and manipulate the database](#browse-and-manipulate-the-database)
- [Browse and manipulate the database](#browse-and-manipulate-the-database), for when you really need to take matters into your own hands
@ -57,27 +54,6 @@ If you need to adjust this, pass: `--extra-vars='matrix_synapse_rust_synapse_com
After state compression, you may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql).
## Purging unused data with synapse-janitor
**NOTE**: There are [reports](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/465) that **synapse-janitor is dangerous to use and causes database corruption**. You may wish to refrain from using it.
When you **leave** and **forget** a room, Synapse can clean up its data, but currently doesn't.
This **unused and unreachable data** remains in your database forever.
There are external tools (like [synapse-janitor](https://github.com/xwiki-labs/synapse_scripts)), which are meant to solve this problem.
To ask the playbook to run synapse-janitor, execute:
```bash
ansible-playbook -i inventory/hosts setup.yml --tags=run-postgres-synapse-janitor,start
```
**Note**: this will automatically stop Synapse temporarily and restart it later.
Running synapse-janitor potentially deletes a lot of data from the Postgres database.
You may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql) after that.
## Browse and manipulate the database
When the [matrix admin API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api) and the other tools do not provide a more convenient way, having a look at synapse's postgresql database can satisfy a lot of admins' needs.

@ -106,7 +106,6 @@ matrix_docker_package_name: docker-ce
run_postgres_import: true
run_postgres_upgrade: true
run_postgres_import_sqlite_db: true
run_postgres_synapse_janitor: true
run_postgres_vacuum: true
run_synapse_register_user: true
run_synapse_update_user_password: true

@ -19,7 +19,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mailer \
--tmpfs=/var/spool/exim:rw,noexec,nosuid,size=100m \
--network={{ matrix_docker_network }} \
--env-file={{ matrix_mailer_base_path }}/env-mailer \
--hostname={{ matrix_server_fqn_matrix }} \
--hostname=matrix-mailer \
{% for arg in matrix_mailer_container_extra_arguments %}
{{ arg }} \
{% endfor %}

@ -30,5 +30,3 @@ matrix_postgres_container_extra_arguments: []
#
# Takes an "<ip>:<port>" or "<port>" value (e.g. "127.0.0.1:5432"), or empty string to not expose.
matrix_postgres_container_postgres_bind_port: ""
matrix_postgres_tool_synapse_janitor: "https://raw.githubusercontent.com/xwiki-labs/synapse_scripts/a9188ff175ae581610f92d58ea6eac9a114d854b/synapse_janitor.sql"

@ -29,11 +29,6 @@
tags:
- upgrade-postgres
- import_tasks: "{{ role_path }}/tasks/run_synapse_janitor.yml"
when: run_postgres_synapse_janitor|bool
tags:
- run-postgres-synapse-janitor
- import_tasks: "{{ role_path }}/tasks/run_vacuum.yml"
when: run_postgres_vacuum|bool
tags:

@ -1,117 +0,0 @@
---
# Pre-checks
- name: Fail if Postgres not enabled
fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot run synapse-janitor."
when: "not matrix_postgres_enabled|bool"
- name: Fail if not aware of the risks
fail:
msg: >-
Using Synapse Janitor is considered dangerous and may break your database.
See https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/465.
If you'd like to run it anyway, add `--extra-vars='i_know_synapse_janitor_is_dangerous=1'` to your command.
when: "i_know_synapse_janitor_is_dangerous|default('') == ''"
# Defaults
- name: Set postgres_start_wait_time, if not provided
set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time|default('') == ''"
- name: Set postgres_synapse_janitor_wait_time, if not provided
set_fact:
postgres_synapse_janitor_wait_time: "{{ 7 * 86400 }}"
when: "postgres_synapse_janitor_wait_time|default('') == ''"
- name: Set postgres_synapse_janitor_tool_path, if not provided
set_fact:
postgres_synapse_janitor_tool_path: "{{ matrix_postgres_base_path }}/synapse_janitor.sql"
when: "postgres_synapse_janitor_tool_path|default('') == ''"
# Actual janitor work
- name: Download synapse-janitor tool
get_url:
url: "{{ matrix_postgres_tool_synapse_janitor }}"
dest: "{{ postgres_synapse_janitor_tool_path }}"
force: true
mode: 0550
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
- name: Ensure matrix-postgres is started
service:
name: matrix-postgres
state: started
daemon_reload: yes
- name: Wait a bit, so that Postgres can start
wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
- import_tasks: tasks/util/detect_existing_postgres_version.yml
- name: Abort, if no existing Postgres version detected
fail:
msg: "Could not find existing Postgres installation"
when: "not matrix_postgres_detected_existing|bool"
- name: Generate Postgres database synapse-janitor command
set_fact:
matrix_postgres_synapse_janitor_command: >-
{{ matrix_host_command_docker }} run --rm --name matrix-postgres-synapse-janitor
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
--mount type=bind,src={{ postgres_synapse_janitor_tool_path }},dst=/synapse_janitor.sql,ro=true
{{ matrix_postgres_docker_image_latest }}
psql -v ON_ERROR_STOP=1 -h matrix-postgres {{ matrix_synapse_database_database }} -f /synapse_janitor.sql
- name: Note about Postgres purging alternative
debug:
msg: >-
Running synapse-janitor with the following Postgres command: `{{ matrix_postgres_synapse_janitor_command }}`.
If this crashes, you can stop all processes (`systemctl stop matrix-*`),
start Postgres only (`systemctl start matrix-postgres`)
and manually run the above command directly on the server.
- name: Populate service facts
service_facts:
- set_fact:
matrix_postgres_synapse_was_running: "{{ ansible_facts.services['matrix-synapse.service']|default(none) is not none and ansible_facts.services['matrix-synapse.service'].state == 'running' }}"
- name: Ensure matrix-synapse is stopped
service:
name: matrix-synapse
state: stopped
daemon_reload: yes
- name: Run synapse-janitor
command: "{{ matrix_postgres_synapse_janitor_command }}"
async: "{{ postgres_synapse_janitor_wait_time }}"
poll: 10
register: matrix_postgres_synapse_janitor_result
# Intentionally show the results
- debug: var="matrix_postgres_synapse_janitor_result"
- name: Ensure matrix-synapse is started, if it previously was
service:
name: matrix-synapse
state: started
daemon_reload: yes
when: "matrix_postgres_synapse_was_running|bool"
- name: Delete synapse-janitor tool
file:
path: "{{ postgres_synapse_janitor_tool_path }}"
state: absent

@ -5,13 +5,14 @@ matrix_registration_enabled: true
matrix_registration_container_image_self_build: false
matrix_registration_container_image_self_build_repo: "https://github.com/ZerataX/matrix-registration"
matrix_registration_container_image_self_build_branch: "{{ 'master' if matrix_registration_version == 'latest' else matrix_registration_version }}"
matrix_registration_base_path: "{{ matrix_base_data_path }}/matrix-registration"
matrix_registration_config_path: "{{ matrix_registration_base_path }}/config"
matrix_registration_data_path: "{{ matrix_registration_base_path }}/data"
matrix_registration_docker_src_files_path: "{{ matrix_registration_base_path }}/docker-src"
matrix_registration_version: "v0.7.0"
matrix_registration_version: "v0.7.1"
matrix_registration_docker_image: "{{ matrix_registration_docker_image_name_prefix }}devture/zeratax-matrix-registration:{{ matrix_registration_version }}"
matrix_registration_docker_image_name_prefix: "{{ 'localhost/' if matrix_registration_container_image_self_build else 'docker.io/' }}"

@ -30,7 +30,7 @@
git:
repo: "{{ matrix_registration_container_image_self_build_repo }}"
dest: "{{ matrix_registration_docker_src_files_path }}"
version: "{{ matrix_registration_version }}"
version: "{{ matrix_registration_container_image_self_build_branch }}"
force: "yes"
register: matrix_registration_git_pull_results
when: "matrix_registration_enabled|bool and matrix_registration_container_image_self_build|bool"

@ -8,7 +8,7 @@ matrix_synapse_admin_container_self_build_repo: "https://github.com/Awesome-Tech
matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src"
matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:0.5.0"
matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:0.6.1"
matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_self_build else 'docker.io/' }}"
matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}"

Loading…
Cancel
Save