brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(postgresql): try backing up an rds instance

Browse Source
This commit is contained in:
Michele Cereda
2025-07-10 20:15:43 +02:00
parent 9f123b4305
commit 7554d355b6
8 changed files with 498 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
knowledge base/less.md
View File

@@ -2,24 +2,34 @@
## TL;DR
```sh
```plaintext
# Search words *forwards* in the current document.
:/keyword <ENTER>
:/keyword ↵
# Search words *backwards* in the current document.
:?keyword <ENTER>
:?keyword ↵
# Toggle case insensitivity in searches.
:-i <ENTER>
# Toggle case sensitivity in searches.
:-I ↵
```
## Sources
```sh
# Start with case sensitivity *disabled* in searches
less -I
```
## Further readings
### Sources
- [Less searches are always case-insensitive]
- [How to Search in Less Command]
<!--
References
Reference
═╬═Time══
-->
<!-- Others -->
[less searches are always case-insensitive]: https://unix.stackexchange.com/questions/116395/less-searches-are-always-case-insensitive#577376
[How to Search in Less Command]: https://linuxhandbook.com/search-less-command/

46
knowledge base/sed.md
View File

@@ -5,16 +5,22 @@
1. [TL;DR](#tldr)
1. [Character classes and bracket expressions](#character-classes-and-bracket-expressions)
1. [Further readings](#further-readings)
1. [Sources](#sources)
## TL;DR
Use [character classes](#character-classes-and-bracket-expressions) instead of regex shorthands.
```sh
# Quote any set of characters that is not a space.
sed -E 's|([[:graph:]]+)|"\1"|g'
sed -E 's|([[:graph:]]+)|"\1"|g' 'file.txt'
# Delete lines matching "OAM" from a file.
# Delete lines matching 'OAM' from a file.
# Overwrite the source file with the changes.
sed '/OAM/d' -i .bash_history
sed -i '/OAM/d' '.bash_history'
# Delete lines matching 'pattern' plus the next 5 ones.
sed '/pattern/,+5d' 'file.txt'
# Show changed fstab entries.
# Don't save the changes.
@@ -26,28 +32,33 @@ sed /etc/fstab \
## Character classes and bracket expressions
| Class | Description |
| ----- | ----------- |
| `[[:alnum:]]` | alphanumeric characters `[[:alpha:]]` and `[[:digit:]]`; this is the same as `[0-9A-Za-z]` in the `C` locale and ASCII character |
| `[[:alpha:]]` | alphabetic characters `[[:lower:]]` and `[[:upper:]]`; this is the same as `[A-Za-z]` in the `C` locale and ASCII character encoding |
| `[[:blank:]]` | blank characters `space` and `tab` |
| `[[:cntrl:]]` | control characters; in ASCII these characters have octal codes 000 through 037 and 177 (DEL), in other character sets these are the equivalent characters, if any |
| `[[:digit:]]` | digits `0` to `9` |
| `[[:graph:]]` | graphical characters `[[:alnum:]]` and `[[:punct:]]` |
| `[[:lower:]]` | lower-case letters `a` to `z` in the `C` locale and ASCII character encoding |
| `[[:print:]]` | printable characters `[[:alnum:]]`, `[[:punct:]]` and `space` |
| Class | Description |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `[[:alnum:]]` | alphanumeric characters `[[:alpha:]]` and `[[:digit:]]`; this is the same as `[0-9A-Za-z]` in the `C` locale and ASCII character |
| `[[:alpha:]]` | alphabetic characters `[[:lower:]]` and `[[:upper:]]`; this is the same as `[A-Za-z]` in the `C` locale and ASCII character encoding |
| `[[:blank:]]` | blank characters `space` and `tab` |
| `[[:cntrl:]]` | control characters; in ASCII these characters have octal codes 000 through 037 and 177 (DEL), in other character sets these are the equivalent characters, if any |
| `[[:digit:]]` | digits `0` to `9` |
| `[[:graph:]]` | graphical characters `[[:alnum:]]` and `[[:punct:]]` |
| `[[:lower:]]` | lower-case letters `a` to `z` in the `C` locale and ASCII character encoding |
| `[[:print:]]` | printable characters `[[:alnum:]]`, `[[:punct:]]` and `space` |
| `[[:punct:]]` | punctuation characters `!`, `"`, `#`, `$`, `%`, `&`, `'`, `(`, `)`, `*`, `+`, `,`, `-`, `.`, `/`, `:`, `;`, `<`, `=`, `>`, `?`, `@`, `[`, `\`, `]`, `^`, `_`, `` ` ``, `{`, `\|`, `}` and `~` in the `C` locale and ASCII character encoding |
| `[[:space:]]` | space characters `tab`, `newline`, `vertical tab`, `form feed`, `carriage return` and `space` in the `C` locale |
| `[[:upper:]]` | upper-case letters `A` to `Z` in the `C` locale and ASCII character encoding |
| `[[:xdigit:]]` | hexadecimal digits `0` to `9`, `A` to `F` and `a` to `f` |
| `[[:space:]]` | space characters `tab`, `newline`, `vertical tab`, `form feed`, `carriage return` and `space` in the `C` locale |
| `[[:upper:]]` | upper-case letters `A` to `Z` in the `C` locale and ASCII character encoding |
| `[[:xdigit:]]` | hexadecimal digits `0` to `9`, `A` to `F` and `a` to `f` |
## Further readings
- [GNU SED Online Tester]
- [Character Classes and Bracket Expressions]
### Sources
- [sed or awk: delete n lines following a pattern]
<!--
References
Reference
═╬═Time══
-->
<!-- Upstream -->
@@ -55,3 +66,4 @@ sed /etc/fstab \
<!-- Others -->
[gnu sed online tester]: https://sed.js.org/
[sed or awk: delete n lines following a pattern]: https://stackoverflow.com/questions/4396974/sed-or-awk-delete-n-lines-following-a-pattern

62
snippets/ansible/tasks.yml
View File

@@ -756,30 +756,54 @@
# 'amazon.aws.rds_instance' will *not* have the 'endpoint' key defined if not waiting
ansible.builtin.set_fact:
pitr_restored_instance: "{{ pitr_restored_instance_ready_check.instances[0] }}"
- name: Dump roles' privileges
- name: Dump roles and their permissions
environment:
PGHOST: instance-id.0123456789ab.eu-west-1.rds.amazonaws.com
PGPORT: 5432
PGDATABASE: postgres
PGUSER: postgres
PGPASSWORD: someSecurePassword
block:
- name: Dump to file
environment:
PGPASSWORD: someRandomString
vars:
out_file: /tmp/instance-id_roles.sql
ansible.builtin.command: >-
pg_dumpall
--host 'instance-id.0123456789ab.eu-west-1.rds.amazonaws.com' --port '5432'
--user 'postgres' --database 'postgres' --no-password
--roles-only --no-role-passwords
--file '{{ out_file }}'
ansible.builtin.shell:
cmd: >-
pg_dumpall --no-password
--roles-only --no-role-passwords
--file '{{ out_file }}'
| grep -v -e 'rds_superuser'
creates: "{{ out_file }}"
changed_when: false
- name: Dump to variable for later use through 'dump_execution.stdout_lines'
environment:
PGPASSWORD: someRandomString
ansible.builtin.command: >-
pg_dumpall
-h 'instance-id.0123456789ab.eu-west-1.rds.amazonaws.com' -p '5432'
-U 'postgres' -l 'postgres' -w
-r --no-role-passwords
changed_when: false
register: dump_execution
block:
- name: FIXME
ansible.builtin.command: >-
pg_dumpall -w
-r --no-role-passwords
changed_when: false
register: dump_execution
- name: FIXME
# remove empty lines
# remove comments
# remove creation of the master user
# remove anything involving 'rdsadmin'
# remove changes to protected RDS users
# remove protected 'superuser' and 'replication' assignments
vars:
dump_content_as_lines: "{{ dump_execution.content | ansible.builtin.b64decode | split('\n') }}"
master_username: postgres
ansible.builtin.set_fact:
permissions_commands: >-
{{
dump_content_as_lines
| reject('match', '^$')
| reject('match', '^--')
| reject('match', '^CREATE ROLE ' + master_username)
| reject('match', '.*rdsadmin.*')
| reject('match', '^(CREATE|ALTER) ROLE rds_')
| map('regex_replace', '(NO)(SUPERUSER|REPLICATION)\s?', '')
}}
- name: Wait for pending changes to be applied
amazon.aws.rds_instance_info:
db_instance_identifier: identifier-for-db-instance
@@ -796,7 +820,7 @@
- name: Download objects from S3
# The 'amazon.aws.s3_object' module might be *not* suitable for files bigger than the executor's currently
# available memory. See <https://github.com/ansible-collections/amazon.aws/issues/2395>.
# TL:DR: at the time of writing, the module keeps downloaded data in memory before flushing it to disk,
# TL;DR: at the time of writing, the module keeps downloaded data in memory before flushing it to disk,
# filling up the host's memory when downloading big files and causing it to stall or crash.
amazon.aws.s3_object:
bucket: my-bucket

191
snippets/ansible/tasks/aws/rds/clone db instance.yml Normal file
View File

@@ -0,0 +1,191 @@
---
###
# Clone an RDS instance
# ------------------
# Usage examples:
# - ansible-navigator run 'clone db instance.yml' \
# --pass-environment-variable='ANSIBLE_VAULT_PASSWORD' \
# --pass-environment-variable='ANSIBLE_VAULT_PASSWORD_FILE' \
# --pass-environment-variable='AWS_ACCESS_KEY_ID' \
# --pass-environment-variable='AWS_DEFAULT_REGION' \
# --pass-environment-variable='AWS_PROFILE' \
# --pass-environment-variable='AWS_REGION' \
# --pass-environment-variable='AWS_SECRET_ACCESS_KEY' \
# --log-file='/dev/null'
# -- \
# --inventory 'localhost,' --diff -Cvvv \
# -e 'db_instance_identifier=some-db-identifier'
# TODO:
# - improve input checks?
# - increase db creation parameters?
###
- name: Clone RDS instance
hosts: localhost
connection: local
gather_facts: false
vars_prompt:
- name: db_instance_identifier
prompt: Identifier of the RDS DB instance to clone
private: false
vars:
clone_db_instance_identifier: "{{ db_instance_identifier }}-clone"
pre_tasks:
- name: PRE DEBUG Print run's variables
tags:
- pre_flight
- debug
ansible.builtin.debug:
verbosity: 3
var: vars
- name: PRE DEBUG Print shell environment
tags:
- pre_flight
- debug
check_mode: false
ansible.builtin.shell: set
- name: PRE CHECK Check input is usable
tags:
- pre_flight
- check_input
ansible.builtin.assert:
that:
- db_instance_identifier not in [None, '']
- clone_db_instance_identifier | length < 64
tasks:
- name: Get source DB instance information
tags: get_source_instance_information
block:
- name: Get information about source DB instance '{{ db_instance_identifier }}'
amazon.aws.rds_instance_info:
db_instance_identifier: "{{ db_instance_identifier }}"
register: source_instance_information_gathering
- name: Check source DB instance '{{ db_instance_identifier }}' has been found
ansible.builtin.assert:
that: source_instance_information_gathering.instances | length > 0
fail_msg: No RDS DB instances found with identifier '{{ db_instance_identifier }}'
success_msg: At least one RDS DB instance found with identifier '{{ db_instance_identifier }}'
- name: Register information about source DB instance '{{ db_instance_identifier }}' for later use
ansible.builtin.set_fact:
source_db_instance: "{{ source_instance_information_gathering.instances | first }}"
- name: >-
Create clone DB instance '{{ clone_db_instance_identifier }}' from
'{{ source_db_instance.db_instance_identifier }}'
tags: create_clone_instance
when: source_db_instance.db_parameter_groups is defined
amazon.aws.rds_instance:
creation_source: instance
source_db_instance_identifier: "{{ source_db_instance.db_instance_identifier }}"
db_instance_identifier: "{{ clone_db_instance_identifier }}"
tags: >-
{{
source_db_instance.tags
| combine({
'Description': 'Clone of ' + source_db_instance.db_instance_identifier,
'ManagedByAnsible': 'true',
})
}}
db_subnet_group_name: >-
{{
[
clone_db_subnet_group_name | default(None),
source_db_instance.db_subnet_group.db_subnet_group_name,
] | select | first
}}
publicly_accessible: >-
{{
[
clone_publicly_accessible | default(None),
source_db_instance.publicly_accessible,
] | select | first
}}
vpc_security_group_ids: >-
{{
[
clone_vpc_security_group_ids | default(None),
source_db_instance.db_security_groups,
] | reject("none") | first
}}
port: >-
{{
[
clone_port | default(None),
source_db_instance.endpoint.port,
] | select | first
}}
db_name: >-
{{
[
clone_db_name | default(None),
source_db_instance.db_name,
] | select | first
}}
master_username: >-
{{
[
clone_master_username | default(None),
source_db_instance.master_username,
] | select | first
}}
master_user_password: >-
{{
clone_master_user_password
| default(lookup('ansible.builtin.password', '/dev/null', seed=db_instance_identifier, length=16))
}}
engine: "{{ source_db_instance.engine }}"
engine_version: "{{ source_db_instance.engine_version }}"
db_instance_class: >-
{{
[
clone_db_instance_class | default(None),
source_db_instance.db_instance_class,
] | select | first
}}
storage_type: >-
{{
[
clone_storage_type | default(None),
source_db_instance.storage_type,
] | select | first
}}
iops: "{{ clone_iops | default(omit) }}"
storage_throughput: "{{ clone_storage_throughput | default(omit) }}"
storage_encrypted: >-
{{
[
clone_storage_encrypted | default(None),
source_db_instance.storage_encrypted,
] | select | first
}}
kms_key_id: >-
{{
[
clone_kms_key_id | default(None),
source_db_instance.kms_key_id | default(None),
'aws/rds',
] | select | first
}}
option_group_name: >-
{{
[
clone_option_group_name | default(None),
source_db_instance.option_group_memberships[0].option_group_name,
] | select | first
}}
db_parameter_group_name: >-
{{
[
clone_db_parameter_group_name | default(None),
source_db_instance.db_parameter_groups[0].db_parameter_group_name,
] | select | first
}}
auto_minor_version_upgrade: >-
{{
[
clone_auto_minor_version_upgrade | default(None),
source_db_instance.auto_minor_version_upgrade,
] | reject("none") | first
}}
apply_immediately: true
register: clone_db_instance

181
snippets/ansible/tasks/aws/rds/restore db instance from snapshot.yml Normal file
View File

@@ -0,0 +1,181 @@
---
###
# Restore an RDS instance from snapshot
# ------------------
# Creates an RDS instance from a specified snapshot.
# Usage examples:
# - ansible-navigator run 'restore db instance from snapshot.yml' \
# --pass-environment-variable='ANSIBLE_VAULT_PASSWORD' \
# --pass-environment-variable='ANSIBLE_VAULT_PASSWORD_FILE' \
# --pass-environment-variable='AWS_ACCESS_KEY_ID' \
# --pass-environment-variable='AWS_DEFAULT_REGION' \
# --pass-environment-variable='AWS_PROFILE' \
# --pass-environment-variable='AWS_REGION' \
# --pass-environment-variable='AWS_SECRET_ACCESS_KEY' \
# --log-file='/dev/null'
# -- \
# --inventory 'localhost,' --diff -Cvvv \
# -e 'db_snapshot_identifier=some-snapshot-identifier' \
# -e 'db_instance_identifier=some-restored-db-identifier'
# TODO:
# - improve input checks?
# - increase db creation parameters?
###
- name: Restore RDS DB instance from snapshot
hosts: localhost
connection: local
gather_facts: false
vars_prompt:
- name: db_instance_identifier
prompt: Identifier for the restored RDS DB instance
private: false
- name: db_snapshot_identifier
prompt: Identifier for the restored RDS DB instance
private: false
pre_tasks:
- name: PRE DEBUG Print run's variables
tags:
- pre_flight
- debug
ansible.builtin.debug:
verbosity: 3
var: vars
- name: PRE DEBUG Print shell environment
tags:
- pre_flight
- debug
check_mode: false
ansible.builtin.shell: set
- name: PRE CHECK Check input is usable
tags:
- pre_flight
- check_input
ansible.builtin.assert:
that:
- db_snapshot_identifier not in [None, '']
- db_instance_identifier | length < 64
tasks:
- name: Get information for snapshot '{{ db_snapshot_identifier }}'
tags: get_snapshot_information
amazon.aws.rds_snapshot_info:
db_snapshot_identifier: "{{ db_snapshot_identifier }}"
register: get_snapshot_information
- name: Check at least one snapshot with identifier '{{ db_snapshot_identifier }}' is in the 'available' state
ansible.builtin.assert:
that: get_snapshot_information.snapshots | selectattr("status", "equalto", "available") | length > 0
fail_msg: No snapshots found in the 'available' state for identifier '{{ db_snapshot_identifier }}'
success_msg: >-
At least one snapshot found in the 'available' state for identifier '{{ db_snapshot_identifier }}'
- name: Save latest available snapshot's information for identifier '{{ db_snapshot_identifier }}' for later use
ansible.builtin.set_fact:
snapshot: >-
{{
get_snapshot_information.snapshots
| selectattr("status", "equalto", "available")
| sort(attribute='snapshot_create_time')
| last
}}
- name: >-
Create new RDS DB instance '{{ db_instance_identifier }}' from snapshot '{{ snapshot.db_snapshot_identifier }}'
tags: create_instance
amazon.aws.rds_instance:
creation_source: snapshot
db_snapshot_identifier: "{{ snapshot.db_snapshot_identifier }}"
db_instance_identifier: "{{ db_instance_identifier }}"
tags: >-
{{
snapshot.tags
| combine({
'Description': [
'Restore of', snapshot.db_instance_identifier, 'from snapshot', snapshot.db_snapshot_identifier,
] | join(" "),
'ManagedByAnsible': 'true',
})
}}
db_subnet_group_name: >-
{{
[
db_subnet_group_name | default(None),
'default-private',
] | select | first
}}
publicly_accessible: >-
{{
[
publicly_accessible | default(None),
false,
] | reject("none") | first
}}
vpc_security_group_ids: >-
{{
[
vpc_security_group_ids | default(None),
[],
] | reject("none") | first
}}
port: >-
{{
[
port | default(None),
snapshot.port,
] | reject("none") | first
}}
master_user_password: "{{ master_user_password | default(omit) }}"
force_update_password: "{{ (master_user_password is truthy) | ternary(true, false, omit) }}"
engine: "{{ snapshot.engine }}"
engine_version: "{{ snapshot.engine_version }}"
db_instance_class: >-
{{
[
db_instance_class | default(None),
'db.t4g.micro',
] | select | first
}}
storage_type: >-
{{
[
storage_type | default(None),
snapshot.storage_type,
] | select | first
}}
iops: >-
{{
[
iops | default(None),
snapshot.iops,
] | select | first
}}
storage_throughput: >-
{{
[
storage_throughput | default(None),
snapshot.storage_throughput,
] | select | first
}}
storage_encrypted: >-
{{
[
storage_encrypted | default(None),
snapshot.encrypted,
] | reject("none") | first
}}
kms_key_id: >-
{{
[
kms_key_id | default(None),
snapshot.kms_key_id,
] | select | first
}}
option_group_name: >-
{{
[
option_group_name | default(None),
snapshot.option_group_name,
] | select | first
}}
db_parameter_group_name: "{{ db_parameter_group_name | default(omit) }}"
auto_minor_version_upgrade: "{{ auto_minor_version_upgrade | default(omit) }}"
apply_immediately: true
register: db_instance

6
snippets/grep.fish
View File

@@ -3,3 +3,9 @@
# Check unwanted data
# Show file name and line number
! grep -EHinr -e 'some' -e 'regexp' * || ( echo 'unwanted data found' >&2 && false )
# Only print matching lines
grep -Eo 'CONFIG_[A-Z0-9_]+' 'kernel_config'
# Print matching lines with context
grep -E --after-context=1 '[[:digit:]]+ VIEW'

48
snippets/postgres/commands.sh
View File

@@ -42,21 +42,23 @@ psql 'postgres' 'admin'
psql --host 'prod.db.lan' --port '5432' --username 'postgres' --database 'postgres' --password
psql -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -W
psql 'postgresql://localhost:5433/games?sslmode=require'
PGPASSWORD='password' psql 'host=host.fqdn port=5467 user=admin dbname=postgres'
psql 'host=host.fqdn port=5467 user=admin dbname=postgres'
psql "service=prod sslmode=require"
PGHOST='host.fqdn' PGPORT=5432 PGDATABASE='postgres' PGUSER='postgres' PGPASSWORD='somePassword' …
# List available databases
psql … --list
psql --list
# Change passwords
psql … -U 'jonathan' -c '\password'
psql … -U 'admin' -c '\password jonathan'
# Execute SQL commands
psql … -c 'select * from tableName;' -o 'out.file'
psql … -c 'select * from tableName;' -H
psql … -f 'commands.sql'
psql … -f 'dump.sql' -e
# The action is done in a single transaction
psql -c 'select * from tableName;' -o 'out.file'
psql -c 'select * from tableName;' -H
psql -f 'commands.sql'
psql -f 'dump.sql' -e
# Dump DBs
pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password
@@ -67,28 +69,32 @@ pg_dump … -s --format 'custom'
pg_dump … -F'd' --jobs '3'
# Dump DBs' schema only
pg_dump --host 'host.fqnd' --port '5432' --username 'postgres' --dbname 'postgres' --password --schema-only
pg_dump -h 'host.fqnd' -p '5432' -U 'admin' -d 'postgres' -Ws
pg_dump … --schema-only
# Dump users and groups to file
pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -W --roles-only --file 'roles.sql'
pg_dumpall -h 'host.fqnd' -p '5432' -U 'postgres' -l 'postgres' -Wrf 'roles.sql' --no-role-passwords
# Dump only users and groups to file
pg_dumpall … --roles-only --file 'roles.sql'
pg_dumpall … -rf 'roles.sql' --no-role-passwords
# Restore backups
pg_restore -U 'postgres' -d 'sales' 'sales.dump'
pg_restore -h 'host.fqdn' -U 'master' -d 'sales' -Oxj '8' 'sales.dump'
pg_restore … --dbname 'sales' 'sales.dump'
pg_restore … -d 'sales' -Oxj '8' 'sales.dump'
pg_restore … -d 'sales' --clean --if-exists 'sales.dump'
# Initialize a test DB
pgbench -i 'test-db'
pgbench -i 'test-db' -h 'hostname' -p '5555' -U 'user'
pgbench … -i 'test-db'
# Check a DB is ready for use
pg_isready -U 'denis' -d 'sales'
pg_isready
# Skip materialized views during a restore
pg_dump 'database' -Fc 'backup.dump'
pg_restore -l 'backup.dump' | sed '/MATERIALIZED VIEW DATA/d' > 'restore.lst'
pg_restore -L 'restore.lst' -d 'database' 'backup.dump'
# Only then, refresh with them
pg_restore -l 'backup.dump' | grep 'MATERIALIZED VIEW DATA' > 'refresh.lst'
pg_restore -L 'refresh.lst' -d 'database' 'backup.dump'
pg_restore --list 'backup.dump' | sed -E '/[[:digit:]]+ VIEW/,+1d' > 'no-views.lst'
pg_restore -d 'database' --use-list 'no-views.lst' 'backup.dump'
# Only then, if needed, refresh the dump with the views
pg_restore --list 'backup.dump' | grep -E --after-context=1 '[[:digit:]]+ VIEW' | sed '/--/d' > 'only-views.lst'
pg_restore -d 'database' --use-list 'only-views.lst' 'backup.dump'
# Recreate databases
# Cannot be done in a single transaction
psql -c 'DROP DATABASE IF EXISTS sales;' && psql -c 'CREATE DATABASE sales;'
dropdb --if-exists 'sales' && createdb 'sales'

5
snippets/sed.sh
View File

@@ -1,4 +1,7 @@
#!/usr/bin/env sh
# Quote whatever is not a space.
# Quote whatever is not a space
sed -E 's|([[:graph:]]+)|"\1"|g'
# Delete 5 lines after a pattern (including the line with the pattern)
sed '/pattern/,+5d' 'file.txt'