brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(postgresql): dump findings after failed migration

Browse Source
This commit is contained in:
Michele Cereda
2024-07-13 00:40:27 +02:00
parent 4f83d70b3e
commit 16722fb743
3 changed files with 77 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
knowledge base/cloud computing/aws/rds.md
View File

@@ -17,6 +17,7 @@
1. [Troubleshooting](#troubleshooting)
1. [ERROR: extension must be loaded via shared\_preload\_libraries](#error-extension-must-be-loaded-via-shared_preload_libraries)
1. [ERROR: must be superuser to alter _X_ roles or change _X_ attribute](#error-must-be-superuser-to-alter-x-roles-or-change-x-attribute)
1. [Transport fails asking for the remote user must have superuser, but it already does](#transport-fails-asking-for-the-remote-user-must-have-superuser-but-it-already-does)
1. [Further readings](#further-readings)
1. [Sources](#sources)
@@ -70,9 +71,23 @@ aws rds cancel-export-task --export-task-identifier 'my_export'
</details>
<br/>
RDS _Instances_ are managed database environments.<br/>
Instances _can_ be part of a _cluster_, or _standalone_ deployments.<br/>
RDS _Clusters_ are collections of RDS Instances built on the Aurora engine.<br/>
Cluster-specific resources (snapshots, etc) are prefixed by _Cluster_ in the APIs, e.g. `create-db-cluster-snapshot`,
`DBClusterIdentifier` and `DBClusterSnapshotIdentifier`.
Instances [**can** be renamed][renaming a db instance].<br/>
Renaming them has some effects and requirements. Check the reference.
> Try and keep the DBs identifiers under 22 characters when using PostgreSQL.<br/>
> The `pg_transport` extension will try and truncate any `host` argument to 63 characters.
RDS creates FQDNs for the Instances by suffixing the instance identifier with
`.{{12-char-internal-id}}.{{region}}.rds.amazonaws.com`.<br/>
That internal ID is generated by RDS and is based on the combination of the AWS Region and Account the instance is in.
Read replicas **can** be promoted to standalone DB instances.<br/>
See [Working with DB instance read replicas].
@@ -186,8 +201,9 @@ Automatic backups are storage volume snapshots of **entire** DB instances.
Automatic backups are **enabled** by default.<br/>
Setting the backup retention period to 0 disables them, setting it to a nonzero value (re)enables them.
> Enabling automatic backups takes the affected instances offline to have a backup created immediately.<br/>
> It **will** cause outages.
> Enabling automatic backups takes the affected instances offline to have a backup created **immediately**.<br/>
> While the backup is created, the instance is kept in the _Modifying_ state. This **will** block actions on the
> instance and _could_ cause outages.
Automatic backups occur **daily** during the instances' backup window, configured in 30 minute periods. Should backups
require more time than allotted to the backup window, they will continue after the window ends and until they finish.
@@ -396,13 +412,20 @@ putting the source DB in RO mode.
<details>
<summary>Limitations</summary>
- The access privileges and ownership from the source database are **not** transferred to the target database.<br/>
Dump them from the source or recreate them in other ways.
- Databases **cannot** be transported onto read replicas or parent instances of read replicas.
- The access privileges (including the _default_ ones) and ownership from the source database are **not** transferred to
the target instance.<br/>
Dump them from the source, or (preferred) keep sql files with their definitions close to recreate them in other ways.
- Databases **cannot** be transported onto read replicas or parent instances of read replicas.<br/>
They _can_ be read _from_ instances with replicas, though.
- `reg` data types **cannot** be used in any source database's table that are about to be transported.
- There can be **up to 32** total transports (including both imports and exports) active at the same time on any DB
instance.
- All the DB's data is migrated **as is**.
- Triggers and functions are apparently not transported either.<br/>
Noticed after a production DB migration.
- All extensions must be dropped from the source database.<br/>
> This means that, for some extensions, the data they manage is also dropped.
</details>
<details>
@@ -429,9 +452,9 @@ putting the source DB in RO mode.
To avoid locking the operator's machine for the time needed by the transport, it is suggested the use of an EC2 instance
as the middleman to operate on both DBs.
> Try and keep the DBs identifiers under 22 characters.<br/>
> PostgreSQL will try and truncate the identifier after 63 characters, and AWS will add something like
> `.{{12-char-id}}.{{region}}.rds.amazonaws.com` to it.
> Keep the DBs identifiers under 22 characters.<br/>
> The `pg_transport` extension will try and truncate any `host` argument to 63 characters, and RDS FQDNs are something
> like `{{instance-id}}.{{12-char-internal-id}}.{{region}}.rds.amazonaws.com`.
</details>
<details>
@@ -456,12 +479,12 @@ as the middleman to operate on both DBs.
Specifies whether to report timing information during the transport. Defaults to 1 (true), meaning that timing
information is reported.
1. Reboot the instances equipped with the Parameter Group to apply static changes.
1. Assign the Parameter Group to the source instance and reboot it to apply static changes.
1. Create a new _target_ instance with the required allocated storage.<br/>
Check the requirements again.
1. Make sure the middleman can connect to both DBs.
1. Make sure the _target_ DB instance can connect to the _source_.
1. make sure one has a way to reinstate existing roles and permissions onto the target.<br/>
1. Make sure one has a way to reinstate existing roles and permissions onto the target.<br/>
Dump existing roles and permissions from the source if required on the target.
RDS does **not** grant _full_ SuperUser permissions even to instances' master users. This makes impossible to use
@@ -478,9 +501,9 @@ as the middleman to operate on both DBs.
Clean them up from the dump:
```sh
# Ignore *everything* that has to do with 'rdsadmin'
# Ignore the creation or alteration of AWS-managed RDS roles
# Ignore changes involving protected attributes
# Ignore *everything* involving the 'rdsadmin' user.
# Ignore the creation or alteration of AWS-managed RDS roles.
# Ignore changes involving protected attributes.
sed -Ei'.backup' \
-e '/rdsadmin/d' \
-e '/(CREATE|ALTER) ROLE rds_/d' \
@@ -493,7 +516,7 @@ as the middleman to operate on both DBs.
1. Connect to the DB:
```sh
psql -h 'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -p '5432' -d 'source-db' -U 'admin' --password
psql -h 'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -p '5432' -d 'source_db' -U 'admin' --password
```
1. Only the `pg_transport` extension is allowed in the source DB during the actual transport operation.<br/>
@@ -523,7 +546,7 @@ as the middleman to operate on both DBs.
1. Make sure no DB exists with the same name of the source DB:
```sql
DROP DATABASE IF EXISTS "source-db";
DROP DATABASE IF EXISTS "source_db";
```
1. Load the `pg_transport` extension if missing:
@@ -538,7 +561,7 @@ as the middleman to operate on both DBs.
-- Keep arguments in *single* quotes here
SELECT transport.import_from_server(
'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com', 5432,
'admin', 'source-user-password', 'source-db',
'admin', 'source-user-password', 'source_db',
'target-user-password',
true
);
@@ -647,6 +670,24 @@ Error message examples:
RDS does **not** grant _full_ SuperUser permissions even to instances' master users.<br/>
Actions involving altering protected roles or changing protected attributes are practically blocked on RDS.
### Transport fails asking for the remote user must have superuser, but it already does
Error message example:
> Cannot execute SQL 'SELECT transport.import_from_server(
> 'source.ab0123456789.eu-west-1.rds.amazonaws.com',
> 5432,
> 'masta',
> '********',
> 'sales',
> '********',
> true
> );' None: remote user must have superuser (or rds_superuser if on RDS)
_Speculative_ root cause: RDS did not finish to properly apply the settings.
Solution: reboot the source and target instance and retry.
## Further readings
- [Working with DB instance read replicas]
@@ -670,6 +711,7 @@ Actions involving altering protected roles or changing protected attributes are
- [Working with parameters on your RDS for PostgreSQL DB instance]
- [Backing up login roles aka users and group roles]
- [Renaming a DB instance]
- [Amazon RDS DB instances]
<!--
Reference
@@ -683,6 +725,7 @@ Actions involving altering protected roles or changing protected attributes are
<!-- Files -->
<!-- Upstream -->
[amazon rds db instance storage]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html
[amazon rds db instances]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.DBInstance.html
[aws kms key management]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.Keys.html
[how can i decrease the total provisioned storage size of my amazon rds db instance?]: https://repost.aws/knowledge-center/rds-db-storage-size
[how can i resolve the "error: <module/extension> must be loaded via shared_preload_libraries" error?]: https://repost.aws/knowledge-center/rds-postgresql-resolve-preload-error

2
knowledge base/postgresql.md
View File

@@ -104,6 +104,7 @@ scram-sha-256 'mySecretPassword'
- [Connect to a PostgreSQL database]
- [The password file]
- [How to Generate SCRAM-SHA-256 to Create Postgres 13 User]
- [PostgreSQL: Get member roles and permissions]
<!--
Reference
@@ -121,4 +122,5 @@ scram-sha-256 'mySecretPassword'
[connect to a postgresql database]: https://www.postgresqltutorial.com/connect-to-postgresql-database/
[how to generate scram-sha-256 to create postgres 13 user]: https://stackoverflow.com/questions/68400120/how-to-generate-scram-sha-256-to-create-postgres-13-user
[how to scram in postgres with pgbouncer]: https://www.crunchydata.com/blog/pgbouncer-scram-authentication-postgresql
[postgresql: get member roles and permissions]: https://www.cybertec-postgresql.com/en/postgresql-get-member-roles-and-permissions/
[what is the pg_dump command for backing up a postgresql database?]: https://www.linkedin.com/advice/3/what-pgdump-command-backing-up-postgresql-ke2ef

27
snippets/postgresql.sql
View File

@@ -28,13 +28,15 @@ SELECT * FROM pg_settings;
SELECT "name", "setting" FROM pg_settings WHERE NAME LIKE '%log%';
SHOW "wal_keep_size";
SHOW "password_encryption";
SHOW "pgaudit.log";
-- Change database settings for the current session
SET pgaudit.log = 'none';
SET password_encryption = 'scram-sha-256';
-- Change database settings *for the current session* only
SET pgaudit.log = none;
SET password_encryption = scram-sha-256;
-- Change database settings permanently
ALTER DATABASE reviser SET pgaudit.log TO 'none';
-- Will *not* be active for the current session, logout and login again to see the change
ALTER DATABASE reviser SET pgaudit.log TO none;
-- Switch between databases
\c sales
@@ -71,22 +73,20 @@ ALTER DEFAULT PRIVILEGES FOR ROLE juan IN SCHEMA cache REVOKE all ON TABLES FROM
-- List users only
select usename FROM pg_catalog.pg_user;
-- Check the current user has SuperUser permissions
-- Check the current user has SuperUser privileges
SHOW is_superuser
-- Create roles
-- Roles *are* users and groups since PostgreSQL vFIXME
-- Does *not* support IF NOT EXISTS
CREATE ROLE miriam;
CREATE ROLE miriam WITH LOGIN PASSWORD 'jw8s0F4' VALID UNTIL '2005-01-01';
-- Create users
-- Does *not* support IF NOT EXISTS
CREATE USER mike;
-- Grant users SuperUser permissions
-- Executing user must be already SuperUser
-- Grant roles SuperUser privileges
-- The role granting privileges must be already SuperUser
ALTER USER joel WITH SUPERUSER;
-- Revoke SuperUser permissions
-- Revoke SuperUser privileges
ALTER USER joel WITH NOSUPERUSER;
-- Grant privileges to users
ALTER USER mark CREATEDB;
@@ -96,6 +96,11 @@ ALTER USER jonathan WITH PASSWORD 'seagull5-pantomime-Resting';
ALTER ROLE samantha WITH PASSWORD 'Wing5+Trunks3+Relic2' VALID UNTIL 'August 4 12:00:00 2024 +1';
-- Change password's validity
ALTER ROLE fred VALID UNTIL 'infinity';
-- Rename
ALTER ROLE manager RENAME TO boss
-- Assign roles to users
GRANT rds_superuser TO mike;
-- Close the connection to the current DB