diff --git a/knowledge base/cloud computing/aws/ebs.md b/knowledge base/cloud computing/aws/ebs.md
index 1c01c07..8819a36 100644
--- a/knowledge base/cloud computing/aws/ebs.md
+++ b/knowledge base/cloud computing/aws/ebs.md
@@ -44,6 +44,7 @@ take **up to 72h**.
- AWS' [CLI]
- [Archive Amazon EBS snapshots]
- [Automate snapshot lifecycles]
+- [Choose the best Amazon EBS volume type for your self-managed database deployment]
### Sources
@@ -65,6 +66,7 @@ take **up to 72h**.
[archive amazon ebs snapshots]: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-archive.html
[automate snapshot lifecycles]: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-ami-policy.html
+[choose the best amazon ebs volume type for your self-managed database deployment]: https://aws.amazon.com/blogs/storage/how-to-choose-the-best-amazon-ebs-volume-type-for-your-self-managed-database-deployment/
[delete-volume]: https://docs.aws.amazon.com/cli/latest/reference/ec2/delete-volume.html
[describe-volumes]: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-volumes.html
[documentation]: https://docs.aws.amazon.com/ebs/
diff --git a/knowledge base/cloud computing/aws/rds.md b/knowledge base/cloud computing/aws/rds.md
index e6db136..2a64a87 100644
--- a/knowledge base/cloud computing/aws/rds.md
+++ b/knowledge base/cloud computing/aws/rds.md
@@ -15,6 +15,7 @@
1. [Reduce allocated storage by migrating using transportable databases](#reduce-allocated-storage-by-migrating-using-transportable-databases)
1. [Troubleshooting](#troubleshooting)
1. [ERROR: extension must be loaded via shared\_preload\_libraries](#error-extension-must-be-loaded-via-shared_preload_libraries)
+ 1. [ERROR: must be superuser to alter superuser roles or change superuser attribute](#error-must-be-superuser-to-alter-superuser-roles-or-change-superuser-attribute)
1. [Further readings](#further-readings)
1. [Sources](#sources)
@@ -375,7 +376,11 @@ a database from a source DB instance.
> When the transport begins, all current sessions on the **source** database are ended and the DB is put in ReadOnly
> mode.
> Only the specific source database that is being transported is affected. Others are **not** affected.
->
+
+Primary instances with replicas **can** be used as source instances.
+TODO: test using a RO replica **as** the source instance. I expect this will **not** work due to the transport extension
+putting the source DB in RO mode.
+
> The in-transit database will be **inaccessible** on the **target** DB instance for the duration of the transport.
> During transport, the target DB instance **cannot be restored** to a point in time, as the transport is **not**
> transactional and does **not** use the PostgreSQL write-ahead log to record changes.
@@ -390,7 +395,8 @@ a database from a source DB instance.
> needs to be transported.
> Should the target contain the DB already, it **will** need to be dropped beforehand.
-- Both DB instances **must** run the same major version of PostgreSQL.
+- Both DB instances **must** run the same **major** version of PostgreSQL.
+ Differences in **minor** versions seem to be fine.
- Should the source DB have the `pgaudit` extension _loaded_, that extension will **need** to be _installed_ on the
target instance so that it can be ported.
- The target instance **must** be able to connect to the source instance.
@@ -445,6 +451,31 @@ as the middleman to operate on both DBs.
1. Create a new _target_ instance with the required allocated storage.
1. Make sure the middleman can connect to both DBs.
1. Make sure the _target_ DB instance can connect to the _source_.
+1. RDS does **not** grant _full_ SuperUser permissions even to instances' master users. This makes impossible to use
+ `pg_dumpall -r` to _fully_ dump rules and permissions from the source.
+ One **_can_** export them by **excluding the passwords** from the dump:
+
+ ```sh
+ pg_dumpall -h 'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -U 'admin' -l 'postgres' -W \
+ -rf 'roles.sql' --no-role-passwords
+ ```
+
+ but statements involving protected roles (like `rdsadmin` and any other matching `rds_*`) and change in 'superuser'
+ or 'replication' attributes will fail on restore.
+ Clean them up from the dump:
+
+ ```sh
+ # Ignore *everything* that has to do with 'rdsadmin'
+ # Ignore the creation or alteration of AWS-managed RDS roles
+ # Ignore changes involving protected attributes
+ sed -Ei'.backup' \
+ -e '/rdsadmin/d' \
+ -e '/(CREATE|ALTER) ROLE rds_/d' \
+ -e 's/(NO)(SUPERUSER|REPLICATION)\s?//g' \
+ 'roles.sql'
+ ```
+
+ Just make sure one has a way to reinstate existing roles and permissions onto the target.
1. Prepare the **source** DB for transport:
1. Connect to the DB:
@@ -504,14 +535,24 @@ as the middleman to operate on both DBs.
1. Run the transport by running the `transport.import_from_server` function on the **target** DB instance:
```sql
- SELECT transport.import_from_server( …, …, …, …, …, …, false);
+ SELECT transport.import_from_server( …, …, …, …, …, …, false );
```
1. Validate the data in the target.
-1. Add all the needed roles and permissions to the target.
1. Restore uninstalled extensions in the public schema of **both** DB instances.
- `pg_transport` _can_ be uninstalled.
-1. Revert the value of the max_worker_processes parameter.
+ `pg_transport` _can_ be dropped now.
+1. Restore all the needed roles and permissions onto the target:
+
+ ```sh
+ psql -h 'target-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -p '5432' -U 'admin' -d 'postgres' --password \
+ -f 'roles.sql'
+ ```
+
+ > Restoring roles from raw dumps **will** throw a lot of errors about altering superuser attributes or protected
+ > roles. Check the list item about dumping data above.
+
+1. Revert the value of the `max_worker_processes` parameter if necessary.
+ This will require a restart of the instance.
@@ -534,18 +575,42 @@ ALTER DATABASE db-name SET default_transaction_read_only = false;
db.t4g.medium to db.t4g.medium, gp3 storage, ~ 350 GB database
-| | 1 | 2 | 3 | 4 | 5 |
-| -------------------------- | --------------- | --------------- | --------------- | --------------- | --------------- |
-| `pg_transport.num_workers` | 2 | 4 | 8 | 8 | 12 |
-| `max_worker_processes` | 15 | 21 | 33 | 33 | 45 |
-| `pg_transport.work_mem` | 131072 (128 MB) | 131072 (128 MB) | 131072 (128 MB) | 262144 (256 MB) | 131072 (128 MB) |
-| Minimum transfer rate | ~ 19 MB/s | ~ 19 MB/s | ~ 50 MB/s | ~ 4 MB/s | ~ 25 MB/s |
-| Maximum transfer rate | ~ 58 MB/s | ~ 95 MB/s | ~ 255 MB/s | ~ 255 MB/s | ~ 165 MB/s |
-| Average transfer rate | ~ 31 MB/s | ~ 66 MB/s | ~ 138 MB/s | ~ 101 MB/s | ~ 85 MB/s |
-| Time estimated | ~ 3h 13m | ~ 1h 36m | ~ 48m | ~ 1h | ~ 1h 11m |
-| Time taken | - (interrupted) | - (interrupted) | - (interrupted) | - (interrupted) | - (interrupted) |
-| Source CPU usage | ~ 10% | ~ 15% | ~ 45% | ~ 39% | ~ 37% |
-| Target CPU usage | ~ 12% | ~ 18% | ~ 38% | ~ 28% | ~ 25% |
+Interruptions are due to the exhaustion of I/O burst credits, which tainted the benchmark.
+
+| | 1st run | 2nd run | 3rd and 6th run | 4 | 5 |
+| -------------------------- | ------------------- | ------------------- | ----------------- | ------------------- | ------------------- |
+| `pg_transport.num_workers` | 2 | 4 | 8 | 8 | 12 |
+| `max_worker_processes` | 15 | 21 | 33 | 33 | 45 |
+| `pg_transport.work_mem` | 131072 (128 MB) | 131072 (128 MB) | 131072 (128 MB) | 262144 (256 MB) | 131072 (128 MB) |
+| Minimum transfer rate | ~ 19 MB/s | ~ 19 MB/s | ~ 50 MB/s | ~ 4 MB/s | ~ 25 MB/s |
+| Maximum transfer rate | ~ 58 MB/s | ~ 95 MB/s | ~ 255 MB/s | ~ 255 MB/s | ~ 165 MB/s |
+| Average transfer rate | ~ 31 MB/s | ~ 66 MB/s | ~ 138 MB/s | ~ 101 MB/s | ~ 85 MB/s |
+| Time estimated after 10m | ~ 3h 13m | ~ 1h 36m | ~ 52m | ~ 1h | ~ 1h 11m |
+| Time taken | N/A (interrupted) | N/A (interrupted) | N/A (interrupted) | N/A (interrupted) | N/A (interrupted) |
+| Source CPU usage | ~ 10% | ~ 15% | ~ 40% | ~ 39% | ~ 37% |
+| Source RAM usage delta | N/A (did not check) | N/A (did not check) | + ~ 1.5 GB | N/A (did not check) | N/A (did not check) |
+| Target CPU usage | ~ 12% | ~ 18% | ~ 34% | ~ 28% | ~ 25% |
+| Target RAM usage delta | N/A (did not check) | N/A (did not check) | + ~ 1.5 GB | N/A (did not check) | N/A (did not check) |
+
+
+
+
+ db.m6i.xlarge to db.m6i.xlarge, gp3 storage, ~ 390 GB database
+
+| | 1st run | 2nd to 5th run |
+| -------------------------- | --------------- | --------------- |
+| `pg_transport.num_workers` | 8 | 16 |
+| `max_worker_processes` | 33 | 57 |
+| `pg_transport.work_mem` | 131072 (128 MB) | 131072 (128 MB) |
+| Minimum transfer rate | ~ 97 MB/s | ~ 248 MB/s |
+| Maximum transfer rate | ~ 155 MB/s | ~ 545 MB/s |
+| Average transfer rate | ~ 135 MB/s | ~ 490 MB/s |
+| Time estimated after 10m | ~ 46m | ~ 14m |
+| Time taken | ~ 48m | ~ 14m |
+| Source CPU usage | ~ 12% | ~ 42% |
+| Source RAM usage delta | + ~ 940 MB | + ~ 1.5 GB |
+| Target CPU usage | ~ 17% | ~ 65% |
+| Target RAM usage delta | + ~ 1.3 GB | + ~ 3.3 GB |
@@ -560,6 +625,16 @@ Refer [How can I resolve the "ERROR: must be loaded via share
1. Reboot the instance to apply the change.
1. Try reloading it again.
+### ERROR: must be superuser to alter superuser roles or change superuser attribute
+
+Error message examples:
+
+> ERROR: must be superuser to alter superuser roles or change superuser attribute
+> ERROR: must be superuser to alter replication roles or change replication attribute
+
+RDS does **not** grant _full_ SuperUser permissions even to instances' master users.
+Actions involving altering protected roles or changing protected attributes are practically blocked on RDS.
+
## Further readings
- [Working with DB instance read replicas]
@@ -580,6 +655,7 @@ Refer [How can I resolve the "ERROR: must be loaded via share
- [Migrating databases using RDS PostgreSQL Transportable Databases]
- [Importing data into PostgreSQL on Amazon RDS]
- [Working with parameters on your RDS for PostgreSQL DB instance]
+- [Backing up login roles aka users and group roles]
+[backing up login roles aka users and group roles]: https://www.postgresonline.com/article_pfriendly/81.html
diff --git a/knowledge base/postgresql.md b/knowledge base/postgresql.md
index 9db0870..a4b372b 100644
--- a/knowledge base/postgresql.md
+++ b/knowledge base/postgresql.md
@@ -22,7 +22,8 @@ sudo zypper install 'postgresql15' 'postgresql15-server'
psql 'my-db'
psql 'my-db' 'user'
psql 'postgresql://host:5433/my-db?sslmode=require'
-psql -U 'username' -d 'my-db' -h 'hostname' -p 'port' --password
+psql -U 'username' -d 'my-db' -h 'hostname' -p 'port' -W
+psql --host 'host.fqnd' --port '5432' --username 'postgres' --database 'postgres' --password
# List available databases.
psql … --list
@@ -38,11 +39,20 @@ pgbench -i 'test-db' -h 'hostname' -p '5555' -U 'user'
# Create full backups of databases.
pg_dump -U 'postgres' -d 'sales' -F 'custom' -f 'sales.bak'
+pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password --schema-only
pg_dump … -T 'customers,orders' -t 'salespeople,performances'
-pg_dump … -s
+pg_dump … -s --format 'custom'
+
+# Dump users and groups to file
+pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -W --roles-only --file 'roles.sql'
+pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -Wrf 'roles.sql' --no-role-passwords
# Restore backups.
pg_restore -U 'postgres' -d 'sales' 'sales.bak'
+
+# Execute commands from file
+# E.g., restore from dump
+psql -h 'host.fqnd' -U 'postgres' -d 'postgres' -W -f 'dump.sql' -e
```
## Further readings
diff --git a/snippets/postgresql.sh b/snippets/postgresql.sh
new file mode 100644
index 0000000..672f95f
--- /dev/null
+++ b/snippets/postgresql.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env sh
+
+# Connect to DBs
+psql 'postgres'
+psql 'postgres' 'admin'
+psql --host 'prod.db.lan' --port '5432' --username 'postgres' --database 'postgres' --password
+psql -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -W
+psql 'postgresql://localhost:5433/games?sslmode=require'
+
+# List available databases.
+psql … --list
+
+# Execute commands.
+psql … -c 'select * from tableName;' -o 'out.file'
+psql … -c 'select * from tableName;' -H
+psql … -f 'commands.sql'
+psql … -f 'dump.sql' -e
+
+# Dump DBs
+pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password
+pg_dump -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -W
+pg_dump -U 'postgres' -d 'sales' -F 'custom' -f 'sales.bak' --schema-only
+pg_dump … -T 'customers,orders' -t 'salespeople,performances'
+pg_dump … -s --format 'custom'
+
+# Dump DBs' schema only
+pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password --schema-only
+pg_dump -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -Ws
+
+# Dump users and groups to file
+pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -W --roles-only --file 'roles.sql'
+pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -Wrf 'roles.sql' --no-role-passwords
+
+# Restore backups.
+pg_restore -U 'postgres' -d 'sales' 'sales.bak'
+
+# Initialize a test DB.
+pgbench -i 'test-db'
+pgbench -i 'test-db' -h 'hostname' -p '5555' -U 'user'