diff --git a/knowledge base/cloud computing/aws/ebs.md b/knowledge base/cloud computing/aws/ebs.md index 1c01c07..8819a36 100644 --- a/knowledge base/cloud computing/aws/ebs.md +++ b/knowledge base/cloud computing/aws/ebs.md @@ -44,6 +44,7 @@ take **up to 72h**. - AWS' [CLI] - [Archive Amazon EBS snapshots] - [Automate snapshot lifecycles] +- [Choose the best Amazon EBS volume type for your self-managed database deployment] ### Sources @@ -65,6 +66,7 @@ take **up to 72h**. [archive amazon ebs snapshots]: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-archive.html [automate snapshot lifecycles]: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-ami-policy.html +[choose the best amazon ebs volume type for your self-managed database deployment]: https://aws.amazon.com/blogs/storage/how-to-choose-the-best-amazon-ebs-volume-type-for-your-self-managed-database-deployment/ [delete-volume]: https://docs.aws.amazon.com/cli/latest/reference/ec2/delete-volume.html [describe-volumes]: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-volumes.html [documentation]: https://docs.aws.amazon.com/ebs/ diff --git a/knowledge base/cloud computing/aws/rds.md b/knowledge base/cloud computing/aws/rds.md index e6db136..2a64a87 100644 --- a/knowledge base/cloud computing/aws/rds.md +++ b/knowledge base/cloud computing/aws/rds.md @@ -15,6 +15,7 @@ 1. [Reduce allocated storage by migrating using transportable databases](#reduce-allocated-storage-by-migrating-using-transportable-databases) 1. [Troubleshooting](#troubleshooting) 1. [ERROR: extension must be loaded via shared\_preload\_libraries](#error-extension-must-be-loaded-via-shared_preload_libraries) + 1. [ERROR: must be superuser to alter superuser roles or change superuser attribute](#error-must-be-superuser-to-alter-superuser-roles-or-change-superuser-attribute) 1. [Further readings](#further-readings) 1. [Sources](#sources) @@ -375,7 +376,11 @@ a database from a source DB instance. > When the transport begins, all current sessions on the **source** database are ended and the DB is put in ReadOnly > mode.
> Only the specific source database that is being transported is affected. Others are **not** affected. -> + +Primary instances with replicas **can** be used as source instances.
+TODO: test using a RO replica **as** the source instance. I expect this will **not** work due to the transport extension +putting the source DB in RO mode. + > The in-transit database will be **inaccessible** on the **target** DB instance for the duration of the transport.
> During transport, the target DB instance **cannot be restored** to a point in time, as the transport is **not** > transactional and does **not** use the PostgreSQL write-ahead log to record changes. @@ -390,7 +395,8 @@ a database from a source DB instance. > needs to be transported.
> Should the target contain the DB already, it **will** need to be dropped beforehand. -- Both DB instances **must** run the same major version of PostgreSQL. +- Both DB instances **must** run the same **major** version of PostgreSQL.
+ Differences in **minor** versions seem to be fine. - Should the source DB have the `pgaudit` extension _loaded_, that extension will **need** to be _installed_ on the target instance so that it can be ported. - The target instance **must** be able to connect to the source instance. @@ -445,6 +451,31 @@ as the middleman to operate on both DBs. 1. Create a new _target_ instance with the required allocated storage. 1. Make sure the middleman can connect to both DBs. 1. Make sure the _target_ DB instance can connect to the _source_. +1. RDS does **not** grant _full_ SuperUser permissions even to instances' master users. This makes impossible to use + `pg_dumpall -r` to _fully_ dump rules and permissions from the source.
+ One **_can_** export them by **excluding the passwords** from the dump: + + ```sh + pg_dumpall -h 'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -U 'admin' -l 'postgres' -W \ + -rf 'roles.sql' --no-role-passwords + ``` + + but statements involving protected roles (like `rdsadmin` and any other matching `rds_*`) and change in 'superuser' + or 'replication' attributes will fail on restore.
+ Clean them up from the dump: + + ```sh + # Ignore *everything* that has to do with 'rdsadmin' + # Ignore the creation or alteration of AWS-managed RDS roles + # Ignore changes involving protected attributes + sed -Ei'.backup' \ + -e '/rdsadmin/d' \ + -e '/(CREATE|ALTER) ROLE rds_/d' \ + -e 's/(NO)(SUPERUSER|REPLICATION)\s?//g' \ + 'roles.sql' + ``` + + Just make sure one has a way to reinstate existing roles and permissions onto the target. 1. Prepare the **source** DB for transport: 1. Connect to the DB: @@ -504,14 +535,24 @@ as the middleman to operate on both DBs. 1. Run the transport by running the `transport.import_from_server` function on the **target** DB instance: ```sql - SELECT transport.import_from_server( …, …, …, …, …, …, false); + SELECT transport.import_from_server( …, …, …, …, …, …, false ); ``` 1. Validate the data in the target. -1. Add all the needed roles and permissions to the target. 1. Restore uninstalled extensions in the public schema of **both** DB instances.
- `pg_transport` _can_ be uninstalled. -1. Revert the value of the max_worker_processes parameter. + `pg_transport` _can_ be dropped now. +1. Restore all the needed roles and permissions onto the target: + + ```sh + psql -h 'target-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -p '5432' -U 'admin' -d 'postgres' --password \ + -f 'roles.sql' + ``` + + > Restoring roles from raw dumps **will** throw a lot of errors about altering superuser attributes or protected + > roles. Check the list item about dumping data above. + +1. Revert the value of the `max_worker_processes` parameter if necessary.
+ This will require a restart of the instance.
@@ -534,18 +575,42 @@ ALTER DATABASE db-name SET default_transaction_read_only = false;
db.t4g.medium to db.t4g.medium, gp3 storage, ~ 350 GB database -| | 1 | 2 | 3 | 4 | 5 | -| -------------------------- | --------------- | --------------- | --------------- | --------------- | --------------- | -| `pg_transport.num_workers` | 2 | 4 | 8 | 8 | 12 | -| `max_worker_processes` | 15 | 21 | 33 | 33 | 45 | -| `pg_transport.work_mem` | 131072 (128 MB) | 131072 (128 MB) | 131072 (128 MB) | 262144 (256 MB) | 131072 (128 MB) | -| Minimum transfer rate | ~ 19 MB/s | ~ 19 MB/s | ~ 50 MB/s | ~ 4 MB/s | ~ 25 MB/s | -| Maximum transfer rate | ~ 58 MB/s | ~ 95 MB/s | ~ 255 MB/s | ~ 255 MB/s | ~ 165 MB/s | -| Average transfer rate | ~ 31 MB/s | ~ 66 MB/s | ~ 138 MB/s | ~ 101 MB/s | ~ 85 MB/s | -| Time estimated | ~ 3h 13m | ~ 1h 36m | ~ 48m | ~ 1h | ~ 1h 11m | -| Time taken | - (interrupted) | - (interrupted) | - (interrupted) | - (interrupted) | - (interrupted) | -| Source CPU usage | ~ 10% | ~ 15% | ~ 45% | ~ 39% | ~ 37% | -| Target CPU usage | ~ 12% | ~ 18% | ~ 38% | ~ 28% | ~ 25% | +Interruptions are due to the exhaustion of I/O burst credits, which tainted the benchmark. + +| | 1st run | 2nd run | 3rd and 6th run | 4 | 5 | +| -------------------------- | ------------------- | ------------------- | ----------------- | ------------------- | ------------------- | +| `pg_transport.num_workers` | 2 | 4 | 8 | 8 | 12 | +| `max_worker_processes` | 15 | 21 | 33 | 33 | 45 | +| `pg_transport.work_mem` | 131072 (128 MB) | 131072 (128 MB) | 131072 (128 MB) | 262144 (256 MB) | 131072 (128 MB) | +| Minimum transfer rate | ~ 19 MB/s | ~ 19 MB/s | ~ 50 MB/s | ~ 4 MB/s | ~ 25 MB/s | +| Maximum transfer rate | ~ 58 MB/s | ~ 95 MB/s | ~ 255 MB/s | ~ 255 MB/s | ~ 165 MB/s | +| Average transfer rate | ~ 31 MB/s | ~ 66 MB/s | ~ 138 MB/s | ~ 101 MB/s | ~ 85 MB/s | +| Time estimated after 10m | ~ 3h 13m | ~ 1h 36m | ~ 52m | ~ 1h | ~ 1h 11m | +| Time taken | N/A (interrupted) | N/A (interrupted) | N/A (interrupted) | N/A (interrupted) | N/A (interrupted) | +| Source CPU usage | ~ 10% | ~ 15% | ~ 40% | ~ 39% | ~ 37% | +| Source RAM usage delta | N/A (did not check) | N/A (did not check) | + ~ 1.5 GB | N/A (did not check) | N/A (did not check) | +| Target CPU usage | ~ 12% | ~ 18% | ~ 34% | ~ 28% | ~ 25% | +| Target RAM usage delta | N/A (did not check) | N/A (did not check) | + ~ 1.5 GB | N/A (did not check) | N/A (did not check) | + +
+ +
+ db.m6i.xlarge to db.m6i.xlarge, gp3 storage, ~ 390 GB database + +| | 1st run | 2nd to 5th run | +| -------------------------- | --------------- | --------------- | +| `pg_transport.num_workers` | 8 | 16 | +| `max_worker_processes` | 33 | 57 | +| `pg_transport.work_mem` | 131072 (128 MB) | 131072 (128 MB) | +| Minimum transfer rate | ~ 97 MB/s | ~ 248 MB/s | +| Maximum transfer rate | ~ 155 MB/s | ~ 545 MB/s | +| Average transfer rate | ~ 135 MB/s | ~ 490 MB/s | +| Time estimated after 10m | ~ 46m | ~ 14m | +| Time taken | ~ 48m | ~ 14m | +| Source CPU usage | ~ 12% | ~ 42% | +| Source RAM usage delta | + ~ 940 MB | + ~ 1.5 GB | +| Target CPU usage | ~ 17% | ~ 65% | +| Target RAM usage delta | + ~ 1.3 GB | + ~ 3.3 GB |
@@ -560,6 +625,16 @@ Refer [How can I resolve the "ERROR: must be loaded via share 1. Reboot the instance to apply the change. 1. Try reloading it again. +### ERROR: must be superuser to alter superuser roles or change superuser attribute + +Error message examples: + +> ERROR: must be superuser to alter superuser roles or change superuser attribute
+> ERROR: must be superuser to alter replication roles or change replication attribute + +RDS does **not** grant _full_ SuperUser permissions even to instances' master users.
+Actions involving altering protected roles or changing protected attributes are practically blocked on RDS. + ## Further readings - [Working with DB instance read replicas] @@ -580,6 +655,7 @@ Refer [How can I resolve the "ERROR: must be loaded via share - [Migrating databases using RDS PostgreSQL Transportable Databases] - [Importing data into PostgreSQL on Amazon RDS] - [Working with parameters on your RDS for PostgreSQL DB instance] +- [Backing up login roles aka users and group roles] +[backing up login roles aka users and group roles]: https://www.postgresonline.com/article_pfriendly/81.html diff --git a/knowledge base/postgresql.md b/knowledge base/postgresql.md index 9db0870..a4b372b 100644 --- a/knowledge base/postgresql.md +++ b/knowledge base/postgresql.md @@ -22,7 +22,8 @@ sudo zypper install 'postgresql15' 'postgresql15-server' psql 'my-db' psql 'my-db' 'user' psql 'postgresql://host:5433/my-db?sslmode=require' -psql -U 'username' -d 'my-db' -h 'hostname' -p 'port' --password +psql -U 'username' -d 'my-db' -h 'hostname' -p 'port' -W +psql --host 'host.fqnd' --port '5432' --username 'postgres' --database 'postgres' --password # List available databases. psql … --list @@ -38,11 +39,20 @@ pgbench -i 'test-db' -h 'hostname' -p '5555' -U 'user' # Create full backups of databases. pg_dump -U 'postgres' -d 'sales' -F 'custom' -f 'sales.bak' +pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password --schema-only pg_dump … -T 'customers,orders' -t 'salespeople,performances' -pg_dump … -s +pg_dump … -s --format 'custom' + +# Dump users and groups to file +pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -W --roles-only --file 'roles.sql' +pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -Wrf 'roles.sql' --no-role-passwords # Restore backups. pg_restore -U 'postgres' -d 'sales' 'sales.bak' + +# Execute commands from file +# E.g., restore from dump +psql -h 'host.fqnd' -U 'postgres' -d 'postgres' -W -f 'dump.sql' -e ``` ## Further readings diff --git a/snippets/postgresql.sh b/snippets/postgresql.sh new file mode 100644 index 0000000..672f95f --- /dev/null +++ b/snippets/postgresql.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env sh + +# Connect to DBs +psql 'postgres' +psql 'postgres' 'admin' +psql --host 'prod.db.lan' --port '5432' --username 'postgres' --database 'postgres' --password +psql -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -W +psql 'postgresql://localhost:5433/games?sslmode=require' + +# List available databases. +psql … --list + +# Execute commands. +psql … -c 'select * from tableName;' -o 'out.file' +psql … -c 'select * from tableName;' -H +psql … -f 'commands.sql' +psql … -f 'dump.sql' -e + +# Dump DBs +pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password +pg_dump -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -W +pg_dump -U 'postgres' -d 'sales' -F 'custom' -f 'sales.bak' --schema-only +pg_dump … -T 'customers,orders' -t 'salespeople,performances' +pg_dump … -s --format 'custom' + +# Dump DBs' schema only +pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password --schema-only +pg_dump -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -Ws + +# Dump users and groups to file +pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -W --roles-only --file 'roles.sql' +pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -Wrf 'roles.sql' --no-role-passwords + +# Restore backups. +pg_restore -U 'postgres' -d 'sales' 'sales.bak' + +# Initialize a test DB. +pgbench -i 'test-db' +pgbench -i 'test-db' -h 'hostname' -p '5555' -U 'user'