brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-08 21:34:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: record knowledge from pg_flo testing

Browse Source
This commit is contained in:
Michele Cereda
2024-11-09 12:46:09 +01:00
parent d9221145dd
commit 42c126875f
4 changed files with 341 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.vscode/settings.json vendored
View File

@@ -231,6 +231,7 @@
"mountpoint",
"mpiexec",
"multiarch",
"nats",
"netcat",
"nfsmount",
"nindent",

175
knowledge base/nats.md Normal file
View File

@@ -0,0 +1,175 @@
# NATS
Messaging system.
1. [TL;DR](#tldr)
1. [Troubleshooting](#troubleshooting)
1. [Context deadline exceeded](#context-deadline-exceeded)
1. [No responders available for request](#no-responders-available-for-request)
1. [Further readings](#further-readings)
1. [Sources](#sources)
## TL;DR
<details>
<summary>Setup</summary>
<details style="padding: 0 0 0 1em">
<summary>Server</summary>
```sh
# Install.
brew install 'nats-server'
choco install 'nats-server'
docker pull 'nats'
go install 'github.com/nats-io/nats-server/v2@latest'
yay 'nats-server'
# Validate the configuration file.
nats-server -c '/etc/nats/nats-server.conf' -t
docker run --rm --name 'pg_flo_nats' -v "$PWD/config/nats-server.conf:/etc/nats/nats-server.conf" 'nats' \
-c '/etc/nats/nats-server.conf' -t
```
</details>
<details style="padding: 0 0 0 1em">
<summary>Client</summary>
```sh
# Install.
brew install 'nats-io/nats-tools/nats'
```
</details>
</details>
<details>
<summary>Usage</summary>
<details style="padding: 0 0 0 1em">
<summary>Server</summary>
```sh
# Get help.
docker run --rm --name 'pg_flo_nats' 'nats' --help
# Run.
nats-server -V
nats-server -config 'nats-server.conf'
docker run --name 'nats' -p '4222:4222' -ti 'nats:latest' -js
# Run as cluster.
docker run --name 'nats-0' --network 'nats' -p '4222:4222' -p '8222:8222' \
'nats' --http_port '8222' --cluster_name 'NATS' --cluster 'nats://0.0.0.0:6222' \
&& docker run --name 'nats-1' --network 'nats' \
'nats' --cluster_name 'NATS' --cluster 'nats://0.0.0.0:6222' --routes='nats://ruser:T0pS3cr3t@nats:6222' \
&& curl -fs 'http://localhost:8222/routez'
# Reload the configuration.
nats-server --signal 'reload'
```
</details>
<details style="padding: 0 0 0 1em">
<summary>Client</summary>
```sh
# Get help.
nats cheat server
# Check connection to the server.
nats server check connection --server 'nats://0.0.0.0:4222'
nats server check connection -s 'nats://localhost:4222'
# Request a configuration reload.
nats --user 'sys' --password 'sys' request '$SYS.REQ.SERVER.<server-id>.RELOAD' ""
# Start subscribers.
nats subscribe '>' -s '0.0.0.0:4222'
nats subscribe -s 'nats://demo.nats.io' '>'
# Publish messages.
nats pub 'hello' 'world' -s '0.0.0.0:4222'
# Start listeners for Request-Reply patterns.
nats reply 'subject' 'message'
# Send requests for Request-Reply patterns.
nats request 'help.please' 'I need help!'
# List configuration contexts
nats context ls
nats context ls --all
# List available JetStream streams.
nats stream ls
nats stream ls --all
# List available JetStream consumer.
nats consumer ls
nats consumer ls --all
```
</details>
</details>
<details>
<summary>Real world use cases</summary>
```sh
# Try out the Request-Reply pattern.
# The listener will hang waiting, run the second in another shell session.
nats reply 'help.please' 'OK, I CAN HELP!!!'
nats request 'help.please' 'I need help!'
```
</details>
## Troubleshooting
### Context deadline exceeded
Error message example:
```plaintext
FTL Failed to create NATS client error="failed to create main stream: context deadline exceeded"
```
Root cause: the client cannot reach the server.
### No responders available for request
Error message example:
```plaintext
FTL Failed to create NATS client error="failed to create main stream: nats: no responders available for request"
```
Root cause: a request is sent to a subject that has no subscribers.
## Further readings
- [Website]
- [Codebase]
- [Documentation]
### Sources
<!--
Reference
═╬═Time══
-->
<!-- In-article sections -->
<!-- Knowledge base -->
<!-- Files -->
<!-- Upstream -->
[codebase]: https://github.com/nats-io
[documentation]: https://docs.nats.io
[website]: https://nats.io/
<!-- Others -->

125
knowledge base/pg_flo.md
View File

@@ -3,11 +3,22 @@
Move and transform data between PostgreSQL databases using Logical Replication.
1. [TL;DR](#tldr)
1. [How this works](#how-this-works)
1. [State Management](#state-management)
1. [Further readings](#further-readings)
1. [Sources](#sources)
## TL;DR
Leverages PostgreSQL's logical replication system to capture changes and apply transformations and filtrations to the
data before streaming it to the destination.
Decouples the _replicator_ and _worker_ processes using [NATS] as message broker.<br/>
The NATS server **must have JetStream** enabled (`nats-server -js`).
The _replicator_ component captures PostgreSQL changes via logical replication.<br/>
The _worker_ component processes and routes changes through NATS.
<details>
<summary>Setup</summary>
@@ -63,41 +74,130 @@ nats-url: "nats://localhost:4222"
<summary>Usage</summary>
```sh
# Start NATS server
docker run -d --name 'pg_flo_nats' --network 'host' -v "$PWD/config/nats-server.conf:/etc/nats/nats-server.conf" \
'nats' -c '/etc/nats/nats-server.conf'
# Open a shell
# For debugging purposes, mostly
docker run --rm --name 'pg_flo' --network 'host' --entrypoint 'sh' -ti 'shayonj/pg_flo'
# Start replicator (using config file)
docker run -d --name 'pg_flo_replicator' --network 'host' -v "$PWD/config/pg_flo.yaml:/etc/pg_flo/config.yaml" \
'shayonj/pg_flo' replicator --config '/etc/pg_flo/config.yaml'
# Start the replicator
# Using the config file failed for some reason at the time of writing
docker run --rm --name 'replicator' --network 'host' 'shayonj/pg_flo' \
replicator \
--host 'source-db.fqdn' --dbname 'sales' --user 'pgflo' --password '1q2w3e4r' \
--group 'whatevah' --nats-url 'nats://localhost:4222'
# Start worker
docker run -d --name 'pg_flo_worker' --network 'host' -v "$PWD/config/pg_flo.yaml:/etc/pg_flo/config.yaml" \
'shayonj/pg_flo' worker postgres --config '/etc/pg_flo/config.yaml'
# Start the worker
docker run --rm --name 'pg_flo_worker' --network 'host' 'shayonj/pg_flo' \
worker stdout --group 'whatevah' --nats-url 'nats://localhost:4222'
docker run … \
worker postgres --group 'whatevah' --nats-url 'nats://localhost:4222' \
--target-host 'dest-db.fqdn' --target-dbname 'sales' --target-user 'pgflo' --target-password '1q2w3e4r'
```
</details>
<!-- Uncomment if used
<details>
<summary>Real world use cases</summary>
```sh
# Start a basic replication to stdout as example.
docker run --rm --name 'pg_flo_nats' -p '4222:4222' 'nats' -js \
&& docker run -d --name 'pg_flo_replicator' --network 'host' 'shayonj/pg_flo' \
replicator \
--host 'source-db.fqdn' --port '6001' --dbname 'sales' --user 'pgflo' --password '1q2w3e4r' \
--copy-and-stream --group 'whatevah' --nats-url 'nats://localhost:4222' \
&& docker run -d --name 'pg_flo_worker' --network 'host' 'shayonj/pg_flo' \
worker stdout --group 'whatevah' --nats-url 'nats://localhost:4222'
```
</details>
-->
## How this works
Refer [How it Works].
1. The _replicator_ creates a PostgreSQL **publication** in the source DB for the replicated tables.
1. The _replicator_ creates a **replication slot** in the source DB.<br/>
This ensures no data is lost between streaming sessions.
1. The _replicator_ starts streaming changes from the source DB and publishes them to NATS:
- **After** performing an initial bulk copy, if in _Copy-and-Stream_ mode.
<details style="margin-top: -1em; padding: 0 0 1em 0">
If no valid LSN is found in NATS, `pg_flo` performs an initial bulk copy of existing data.
The process is parallelized for fast data sync:
1. A snapshot is taken to ensure consistency.
1. Each table is divided into page ranges.
1. Multiple workers copy different ranges concurrently.
</details>
- **Immediately**, from the last known position, if in _Stream-Only_ mode.
It also stores the last processed LSN in NATS, allowing the _worker_ to resume operations from where it left off in
case of interruptions.
1. The _worker_ processes messages from NATS.
<details style="margin-top: -1em; padding: 0 0 1em 0">
| Message type | Summary |
| ------------------------------------------------ | ------------------------------------ |
| Relation | Allow understanding table structures |
| `Insert`, `Update`, `Delete` | Contain actual data changes |
| `Begin`, `Commit` | Enable transaction boundaries |
| DDL changes (e.g. `ALTER TABLE`, `CREATE INDEX`) | Contain actual structural changes |
</details>
1. The _worker_ converts received data into a structured format with type-aware conversions for different PostgreSQL
data types.
1. \[If any rule is configured] The _worker_ applies transformation and filtering rules to the data.
<details style="margin-top: -1em; padding: 0 0 1em 0">
Transform Rules:
- Regex: apply regular expression transformations to string values.
- Mask: hide sensitive data, keeping the first and last characters visible.
Filter Rules:
- Comparison: filter based on equality, inequality, greater than, less than, etc.
- Contains: filter string values based on whether they contain a specific substring.
Rules _can_ be applied selectively to `insert`, `update`, or `delete` operations.
</details>
1. The _worker_ buffers processed data.
1. The _worker_ flushes data periodically from the buffer to the configured _sinks_.<br/>
Currently, _sinks_ can be `stdout`, files, PostgreSQL DBs or webhooks.<br/>
Flushed data is written to DB sinks in batches to optimize write operations.
### State Management
The _replicator_ keeps track of its progress by updating the _Last LSN_ in NATS.
The _worker_ maintains its progress to ensure data consistency.<br/>
This allows for resumable operations across multiple runs.
Periodic status updates are sent to the source DB to maintain the replication's connection.
## Further readings
- [Website]
- [Main repository]
- [Transformation rules]
- [NATS]
### Sources
- [How to set the wal_level in AWS RDS Postgresql?]
- [Configuration file reference]
- [How it Works]
<!--
Reference
@@ -106,9 +206,12 @@ docker run -d --name 'pg_flo_worker' --network 'host' -v "$PWD/config/pg_flo.yam
<!-- In-article sections -->
<!-- Knowledge base -->
[nats]: nats.md
<!-- Files -->
<!-- Upstream -->
[configuration file reference]: https://github.com/shayonj/pg_flo/blob/main/internal/pg-flo.yaml
[how it works]: https://github.com/shayonj/pg_flo/blob/main/internal/how-it-works.md
[main repository]: https://github.com/shayonj/pg_flo
[transformation rules]: https://github.com/shayonj/pg_flo/blob/main/pkg/rules/README.md
[website]: https://www.pgflo.io/

51
snippets/nats.fish Normal file
View File

@@ -0,0 +1,51 @@
#!/usr/bin/env fish
###
# Server
# ------------------
###
# Install
brew install 'nats-server'
choco install 'nats-server'
docker pull 'nats'
go install 'github.com/nats-io/nats-server/v2@latest'
yay 'nats-server'
# Validate the configuration file
nats-server -c '/etc/nats/nats-server.conf' -t
docker run --rm --name 'pg_flo_nats' -v "$PWD/config/nats-server.conf:/etc/nats/nats-server.conf" 'nats' \
-c '/etc/nats/nats-server.conf' -t
# Get help
docker run --rm --name 'pg_flo_nats' 'nats' --help
# Run
nats-server -V
docker run --name 'nats' -p '4222:4222' -ti 'nats:latest'
# Run as cluster
docker run --name 'nats-0' --network 'nats' -p '4222:4222' -p '8222:8222' \
'nats' --http_port '8222' --cluster_name 'NATS' --cluster 'nats://0.0.0.0:6222' \
&& docker run --name 'nats-1' --network 'nats' \
'nats' --cluster_name 'NATS' --cluster 'nats://0.0.0.0:6222' --routes='nats://ruser:T0pS3cr3t@nats:6222' \
&& curl -fs 'http://localhost:8222/routez'
###
# Client
# ------------------
###
# Install
brew install 'nats-io/nats-tools/nats'
# Check connection to the server
nats server check connection --server 'nats://0.0.0.0:4222'
nats server check connection -s 'nats://localhost:4222'
# Start subscribers
nats subscribe '>' -s '0.0.0.0:4222'
nats subscribe -s 'nats://demo.nats.io' '>'
# Publish messages
nats pub 'hello' 'world' -s '0.0.0.0:4222'