brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(postgresql): dump findings after failed migration

Browse Source
This commit is contained in:
Michele Cereda
2024-07-13 00:40:27 +02:00
parent 4f83d70b3e
commit 16722fb743
3 changed files with 77 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
knowledge base/cloud computing/aws/rds.md
View File

@@ -17,6 +17,7 @@
1. [Troubleshooting](#troubleshooting)
1. [ERROR: extension must be loaded via shared\_preload\_libraries](#error-extension-must-be-loaded-via-shared_preload_libraries)
1. [ERROR: must be superuser to alter _X_ roles or change _X_ attribute](#error-must-be-superuser-to-alter-x-roles-or-change-x-attribute)
1. [Transport fails asking for the remote user must have superuser, but it already does](#transport-fails-asking-for-the-remote-user-must-have-superuser-but-it-already-does)
1. [Further readings](#further-readings)
1. [Sources](#sources)
@@ -70,9 +71,23 @@ aws rds cancel-export-task --export-task-identifier 'my_export'
</details>
<br/>
RDS _Instances_ are managed database environments.<br/>
Instances _can_ be part of a _cluster_, or _standalone_ deployments.<br/>
RDS _Clusters_ are collections of RDS Instances built on the Aurora engine.<br/>
Cluster-specific resources (snapshots, etc) are prefixed by _Cluster_ in the APIs, e.g. `create-db-cluster-snapshot`,
`DBClusterIdentifier` and `DBClusterSnapshotIdentifier`.
Instances [**can** be renamed][renaming a db instance].<br/>
Renaming them has some effects and requirements. Check the reference.
> Try and keep the DBs identifiers under 22 characters when using PostgreSQL.<br/>
> The `pg_transport` extension will try and truncate any `host` argument to 63 characters.
RDS creates FQDNs for the Instances by suffixing the instance identifier with
`.{{12-char-internal-id}}.{{region}}.rds.amazonaws.com`.<br/>
That internal ID is generated by RDS and is based on the combination of the AWS Region and Account the instance is in.
Read replicas **can** be promoted to standalone DB instances.<br/>
See [Working with DB instance read replicas].
@@ -186,8 +201,9 @@ Automatic backups are storage volume snapshots of **entire** DB instances.
Automatic backups are **enabled** by default.<br/>
Setting the backup retention period to 0 disables them, setting it to a nonzero value (re)enables them.
> Enabling automatic backups takes the affected instances offline to have a backup created immediately.<br/>
> It **will** cause outages.
> Enabling automatic backups takes the affected instances offline to have a backup created **immediately**.<br/>
> While the backup is created, the instance is kept in the _Modifying_ state. This **will** block actions on the
> instance and _could_ cause outages.
Automatic backups occur **daily** during the instances' backup window, configured in 30 minute periods. Should backups
require more time than allotted to the backup window, they will continue after the window ends and until they finish.
@@ -396,13 +412,20 @@ putting the source DB in RO mode.
<details>
<summary>Limitations</summary>
- The access privileges and ownership from the source database are **not** transferred to the target database.<br/>
Dump them from the source or recreate them in other ways.
- Databases **cannot** be transported onto read replicas or parent instances of read replicas.
- The access privileges (including the _default_ ones) and ownership from the source database are **not** transferred to
the target instance.<br/>
Dump them from the source, or (preferred) keep sql files with their definitions close to recreate them in other ways.
- Databases **cannot** be transported onto read replicas or parent instances of read replicas.<br/>
They _can_ be read _from_ instances with replicas, though.
- `reg` data types **cannot** be used in any source database's table that are about to be transported.
- There can be **up to 32** total transports (including both imports and exports) active at the same time on any DB
instance.
- All the DB's data is migrated **as is**.
- Triggers and functions are apparently not transported either.<br/>
Noticed after a production DB migration.
- All extensions must be dropped from the source database.<br/>
> This means that, for some extensions, the data they manage is also dropped.
</details>
<details>
@@ -429,9 +452,9 @@ putting the source DB in RO mode.
To avoid locking the operator's machine for the time needed by the transport, it is suggested the use of an EC2 instance
as the middleman to operate on both DBs.
> Try and keep the DBs identifiers under 22 characters.<br/>
> PostgreSQL will try and truncate the identifier after 63 characters, and AWS will add something like
> `.{{12-char-id}}.{{region}}.rds.amazonaws.com` to it.
> Keep the DBs identifiers under 22 characters.<br/>
> The `pg_transport` extension will try and truncate any `host` argument to 63 characters, and RDS FQDNs are something
> like `{{instance-id}}.{{12-char-internal-id}}.{{region}}.rds.amazonaws.com`.
</details>
<details>
@@ -456,12 +479,12 @@ as the middleman to operate on both DBs.
Specifies whether to report timing information during the transport. Defaults to 1 (true), meaning that timing
information is reported.
1. Reboot the instances equipped with the Parameter Group to apply static changes.
1. Assign the Parameter Group to the source instance and reboot it to apply static changes.
1. Create a new _target_ instance with the required allocated storage.<br/>
Check the requirements again.
1. Make sure the middleman can connect to both DBs.
1. Make sure the _target_ DB instance can connect to the _source_.
1. make sure one has a way to reinstate existing roles and permissions onto the target.<br/>
1. Make sure one has a way to reinstate existing roles and permissions onto the target.<br/>
Dump existing roles and permissions from the source if required on the target.
RDS does **not** grant _full_ SuperUser permissions even to instances' master users. This makes impossible to use
@@ -478,9 +501,9 @@ as the middleman to operate on both DBs.
Clean them up from the dump:
```sh
# Ignore *everything* that has to do with 'rdsadmin'
# Ignore the creation or alteration of AWS-managed RDS roles
# Ignore changes involving protected attributes
# Ignore *everything* involving the 'rdsadmin' user.
# Ignore the creation or alteration of AWS-managed RDS roles.
# Ignore changes involving protected attributes.
sed -Ei'.backup' \
-e '/rdsadmin/d' \
-e '/(CREATE|ALTER) ROLE rds_/d' \
@@ -493,7 +516,7 @@ as the middleman to operate on both DBs.
1. Connect to the DB:
```sh
psql -h 'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -p '5432' -d 'source-db' -U 'admin' --password
psql -h 'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com' -p '5432' -d 'source_db' -U 'admin' --password
```
1. Only the `pg_transport` extension is allowed in the source DB during the actual transport operation.<br/>
@@ -523,7 +546,7 @@ as the middleman to operate on both DBs.
1. Make sure no DB exists with the same name of the source DB:
```sql
DROP DATABASE IF EXISTS "source-db";
DROP DATABASE IF EXISTS "source_db";
```
1. Load the `pg_transport` extension if missing:
@@ -538,7 +561,7 @@ as the middleman to operate on both DBs.
-- Keep arguments in *single* quotes here
SELECT transport.import_from_server(
'source-instance.5f7mp3pt3n6e.eu-west-1.rds.amazonaws.com', 5432,
'admin', 'source-user-password', 'source-db',
'admin', 'source-user-password', 'source_db',
'target-user-password',
true
);
@@ -647,6 +670,24 @@ Error message examples:
RDS does **not** grant _full_ SuperUser permissions even to instances' master users.<br/>
Actions involving altering protected roles or changing protected attributes are practically blocked on RDS.
### Transport fails asking for the remote user must have superuser, but it already does
Error message example:
> Cannot execute SQL 'SELECT transport.import_from_server(
> 'source.ab0123456789.eu-west-1.rds.amazonaws.com',
> 5432,
> 'masta',
> '********',
> 'sales',
> '********',
> true
> );' None: remote user must have superuser (or rds_superuser if on RDS)
_Speculative_ root cause: RDS did not finish to properly apply the settings.
Solution: reboot the source and target instance and retry.
## Further readings
- [Working with DB instance read replicas]
@@ -670,6 +711,7 @@ Actions involving altering protected roles or changing protected attributes are
- [Working with parameters on your RDS for PostgreSQL DB instance]
- [Backing up login roles aka users and group roles]
- [Renaming a DB instance]
- [Amazon RDS DB instances]
<!--
Reference
@@ -683,6 +725,7 @@ Actions involving altering protected roles or changing protected attributes are
<!-- Files -->
<!-- Upstream -->
[amazon rds db instance storage]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html
[amazon rds db instances]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.DBInstance.html
[aws kms key management]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.Keys.html
[how can i decrease the total provisioned storage size of my amazon rds db instance?]: https://repost.aws/knowledge-center/rds-db-storage-size
[how can i resolve the "error: <module/extension> must be loaded via shared_preload_libraries" error?]: https://repost.aws/knowledge-center/rds-postgresql-resolve-preload-error