brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
885ebd11d1c7cfc7fc1ad0a9e77f3e6b6cd0f3a5
oam/knowledge base/database lab.md
Michele Cereda 782ba801bd chore(database-lab): add reference to clone settings
2024-11-06 21:18:41 +01:00

3.3 KiB
Raw Blame History

Database Lab

Database Lab Engine is an open-source platform developed by Postgres.ai to create instant, full-size clones of production databases.
Use cases of the clones are to test database migrations, optimize SQL, or deploy full-size staging apps.

The website https://Postgres.ai/ hosts the SaaS version of the Database Lab Engine.

Configuration file examples are available at https://gitlab.com/postgres-ai/database-lab/-/tree/v3.0.0/configs.

  1. Engine
  2. Clones
  3. Further readings
    1. Sources

Engine

Config file in YAML format, at ~/.dblab/engine/configs/server.yml by default.

Metadata files at ~/.dblab/engine/meta by default. The metadata folder must be writable.

# Reload the configuration without downtime.
docker exec -it 'dblab_server' kill -SIGHUP 1

# Follow logs.
docker logs --since '1m' -f 'dblab_server'
docker logs --since '2024-05-01' -f 'dblab_server'
docker logs --since '2024-08-01T23:11:35' -f 'dblab_server'

Images for the Standard and Enterprise editions are available at https://gitlab.com/postgres-ai/se-images/container_registry/.
Images for the Community edition are available at https://gitlab.com/postgres-ai/custom-images.

Clones

Database clones comes in two flavours:

  • Thick cloning: the regular way to copy data.
    It is also how data is copied to Database Lab the first time a source is added.

    Thick clones can be:

    • Logical: do a regular dump and restore using pg_dump and pg_restore.
    • Physical: done using pg_basebackup or restoring data from physical archives created by backup tools such as WAL-E/WAL-G, Barman, pgBackRest, or pg_probackup.

    Managed PostgreSQL databases in cloud environments (e.g.: AWS RDS) support only the logical clone type.

    The Engine supports continuous synchronization with the source databases.
    Achieved by repeating the thick cloning method one initially used for the source.

  • Thin cloning: local containerized database clones based on CoW (Copy-on-Write) spin up in few seconds.
    They share most of the data blocks, but logically they look fully independent.
    The speed of thin cloning does not depend on the database size.

    As of 2024-06, Database Lab Engine supports ZFS and LVM for thin cloning.
    With ZFS, the Engine periodically creates a new snapshot of the data directory and maintains a set of snapshots. When requesting a new clone, users choose which snapshot to use as base.

Clone DBs configuration starting point is at ~/.dblab/postgres_conf/postgresql.conf.

Further readings

Sources