From 96e6111aa62fe29b251c8fe9c1dd2d4c473375da Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Tue, 9 Feb 2021 22:09:08 +0200 Subject: [PATCH] Improve S3 docs around data migration --- docs/configuring-playbook-s3.md | 135 ++++++++++++++++++++++++++++++++ 1 file changed, 135 insertions(+) diff --git a/docs/configuring-playbook-s3.md b/docs/configuring-playbook-s3.md index 643edb5b2..9132ff718 100644 --- a/docs/configuring-playbook-s3.md +++ b/docs/configuring-playbook-s3.md @@ -6,6 +6,11 @@ If that's alright, you can skip this. If you'd like to store Synapse's content repository (`media_store`) files on Amazon S3 (or other S3-compatible service), you can let this playbook configure [Goofys](https://github.com/kahing/goofys) for you. +Using a Goofys-backed media store works, but performance may not be ideal. If possible, try to use a region which is close to your Matrix server. + +If you'd like to move your locally-stored media store data to Amazon S3 (or another S3-compatible object store), we also provide some migration instructions below. + + ## Amazon S3 You'll need an Amazon S3 bucket and some IAM user credentials (access key + secret key) with full write access to the bucket. Example security policy: @@ -50,3 +55,133 @@ matrix_s3_media_store_custom_endpoint_enabled: true # Example: "https://storage.googleapis.com" matrix_s3_media_store_custom_endpoint: "your-custom-endpoint" ``` + +### Backblaze B2 + +To use [Backblaze B2](https://www.backblaze.com/b2/cloud-storage.html): + +- create a new **private** bucket through its user interface (you can call it something like `matrix-DOMAIN-media-store`) +- note the **Endpoint** for your bucket (something like `s3.us-west-002.backblazeb2.com`) +- adjust its lifecycle rules to use the following **custom** rules: + - File Path: *empty value* + - Days Till Hide: *empty value* + - Days Till Delete: `1` +- go to [App Keys](https://secure.backblaze.com/app_keys.htm) and use the **Add a New Application Key** to create a new one + - restrict it to the previously created bucket (e.g. `matrix-DOMAIN-media-store`) + - give it *Read & Write* access + +Copy the `keyID` and `applicationKey`. + +You need the following *additional* playbook configuration (on top of what you see above): + +```yaml +matrix_s3_media_store_bucket_name: "YOUR_BUCKET_NAME_GOES_HERE" +matrix_s3_media_store_aws_access_key: "YOUR_keyID_GOES_HERE" +matrix_s3_media_store_aws_secret_key: "YOUR_applicationKey_GOES_HERE" +matrix_s3_media_store_custom_endpoint_enabled: true +matrix_s3_media_store_custom_endpoint: "https://s3.us-west-002.backblazeb2.com" # this may be different for your bucket +``` + +If you have local media store files and wish to migrate to Backblaze B2 subsequently, follow our [migration guide to Backblaze B2](#migrating-to-backblaze-b2) below instead of applying this configuration as-is. + + +## Migrating from local filesystem storage to S3 + +It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before migrating your local media store to an S3-backed one. + +Follow one of the guides below for a migration path from a locally-stored media store to one stored on S3-compatible storage: + +- [Migrating to any S3-compatible storage (universal, but likely slow)](#migrating-to-any-s3-compatible-storage-universal-but-likely-slow) +- [Migrating to Backblaze B2](#migrating-to-backblaze-b2) + +### Migrating to any S3-compatible storage (universal, but likely slow) + +It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before doing this. + +1. Proceed with the steps below without stopping Matrix services + +2. Start by adding the base S3 configuration in your `vars.yml` file (seen above, may be different depending on the S3 provider of your choice) + +3. In addition to the base configuration you see above, add this to your `vars.yml` file: + +```yaml +matrix_s3_media_store_path: /matrix/s3-media-store +``` + +This enables S3 support, but mounts the S3 storage bucket to `/matrix/s3-media-store` without hooking it to your homeserver yet. Your homeserver will still continue using your local filesystem for its media store. + +5. Run the playbook to apply the changes: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +6. Do an **initial sync of your files** by running this **on the server** (it may take a very long time): + +```sh +sudo -u matrix -- rsync --size-only --ignore-existing -avr /matrix/synapse/storage/media-store/. /matrix/s3-media-store/. +``` + +You may need to install `rsync` manually. + +7. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`) + +8. Start the S3 service by running this **on the server**: `systemctl start matrix-goofys` + +9. Sync the files again by re-running the `rsync` command you see in step #6 + +10. Stop the S3 service by running this **on the server**: `systemctl stop matrix-goofys` + +11. Get the old media store out of the way by running this command on the server: + +```sh +mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup +``` + +12. Remove the `matrix_s3_media_store_path` configuration from your `vars.yml` file (undoing step #3 above) + +13. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +14. You're done! Verify that loading existing (old) media files works and that you can upload new ones. + +15. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup` + + +### Migrating to Backblaze B2 + +It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before doing this. + +1. While all Matrix services are running, run the following command on the server: + +(you need to adjust the 3 `--env` line below with your own data) + +```sh +docker run -it --rm -w /work \ +--env='B2_KEY_ID=YOUR_KEY_GOES_HERE' \ +--env='B2_KEY_SECRET=YOUR_SECRET_GOES_HERE' \ +--env='B2_BUCKET_NAME=YOUR_BUCKET_NAME_GOES_HERE' \ +-v /matrix/synapse/storage/media-store/:/work \ +--entrypoint=/bin/sh \ +docker.io/tianon/backblaze-b2:2.1.0 \ +-c 'b2 authorize-account $B2_KEY_ID $B2_KEY_SECRET > /dev/null && b2 sync /work/ b2://$B2_BUCKET_NAME' +``` + +This is some initial file sync, which may take a very long time. + +2. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`) + +3. Run the command from step #1 again. + +Doing this will sync any new files that may have been created locally in the meantime. + +Now that Matrix services aren't running, we're sure to get Backblaze B2 and your local media store fully in sync. + +4. Get the old media store out of the way by running this command on the server: + +```sh +mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup +``` + +5. Put the [Backblaze B2 settings seen above](#backblaze-b2) in your `vars.yml` file + +6. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start` + +7. You're done! Verify that loading existing (old) media files works and that you can upload new ones. + +8. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup`