pgEdge Posts from Matthew Mols https://www.pgedge.com/blog The latest pgEdge Posts from Matthew Mols en-us pgEdge,Distributed Postgres,PostgreSQL <![CDATA[pgEdge + CloudNativePG: Simplifying Distributed Postgres on Kubernetes]]> https://www.pgedge.com/blog/pgedge-cloudnativepg-simplifying-distributed-postgres-on-kubernetes Wed, 05 Nov 2025 09:27:00 GMT With pgEdge now fully open source, we’re continuing our mission to make distributed Postgres accessible to developers, operators, and the broader open-source community. A key part of that story is how we can make it easier to run pgEdge using tools that have broad adoption in the community.Today, we’re excited to introduce two key releases that make it even easier to deploy and operate pgEdge Distributed Postgres on Kubernetes:
  • New pgEdge Postgres Container images built for compatibility with CloudNativePG
  • An updated pgEdge Helm chart that simplifies deploying pgEdge on Kubernetes by leveraging CloudNativePG
CloudNativePG is an open-source Kubernetes operator that automates the lifecycle of PostgreSQL clusters using native Kubernetes resources. Its adoption has skyrocketed in recent years, and its recent acceptance as a CNCF Sandbox project has cemented it as the community standard for running Postgres natively on Kubernetes.

New pgEdge Postgres Container Images

In order to make it easier to operate pgEdge in CloudNativePG, and to support other integrations, we’re releasing a new container image built from our pgEdge Enterprise Postgres packages with support for Postgres 16 through 18.These images are published on the Github Container Registry as https://github.com/pgEdge/postgres-images/pkgs/container/pgedge-postgresWe’re releasing two image flavors initially.
  • The
  • minimal
  • image comes bundled by default with the required pgEdge extensions to support distributed deployments: spock, snowflake and lolor
  • The
  • standard
  • image includes popular extensions pgVector, PostGIS and pgAudit
This approach means that you can utilize a single set of images across your Postgres deployments, whether they be in a single region, or distributed with spock’s multi-master replication.Over time, we’ll make additions to the image flavors we publish to support additional extensions and improvements. You can also extend these images to add other extensions to your deployment.This image is designed to be compatible with CloudNativePG, but also includes support for the official Postgres entrypoint, as well as a Patroni entrypoint. This adds more integration opportunities for popular open source tools.You can learn more about the new images here: https://github.com/pgEdge/postgres-images

pgEdge Distributed Postgres in CloudNativePG with pgedge-helm

We also want to make it easier to operate distributed architectures in Kubernetes so that more users can leverage spock’s powerful multi-master capabilities.In order to do this, we’ve released an updated version of our pgEdge Helm chart which supports deploying both pgEdge Enterprise Postgres and pgEdge Distributed Postgres in Kubernetes.This new version leverages CloudNativePG to manage Postgres, providing flexible options for single-region and multi-region deployments.The new chart supports the following features:
  • Postgres 16, 17, and 18 via
  • pgEdge Enterprise Postgres Images
  • Flexible deployment options for both single-region and multi-region deployments
  • Configuring Spock replication configuration across all nodes during helm install and upgrade processes.
  • Best practice configuration defaults for deploying pgEdge Distributed Postgres in Kubernetes.
  • Extending / overriding configuration for CloudNativePG across all nodes, or on specific nodes.
  • Configuring standby instances with automatic failover, leveraging Spock's delayed feedback and failover slots worker to maintain active-active replication across failovers and promotions.
  • Adding pgEdge nodes using Spock or CloudNativePG's bootstrap capabilities to synchronize
  • data from existing nodes or backups.
  • Performing Postgres major and minor version upgrades.
  • Client certificate authentication for managed users, including the
  • pgedge
  • replication user.
  • Configuration options to support deployments across multiple Kubernetes clusters.
The chart includes a simple example which demonstrates deploying a pgEdge Distributed Postgres deployment with 3 nodes.You can install this example by first downloading the latest release package and setting up the required dependencies:1. Download the latest pgedge-helm release package from pgEdge Helm Releases.After downloading and extracting the package on your machine, navigate into the pgedge-helm directory.2. Install pre-requisites (CloudNativePG and cert-manager)3. Install the chartThe chart includes a Kubernetes job which ensures spock’s configuration is kept up to date across chart upgrades.Once the chart is deployed, you can utilize the CloudNativePG kubectl plugin to connect to the app database on the primary for each pgEdge nodeAutomatic DDL replication is enabled by default, so inserting a new table with data will be replicated to all other nodes:4. Create a table and insert data on n15. Query the data on n2For more details on using chart features, see the pgEdge documentation.

Deploying across multiple Kubernetes clusters

A single Kubernetes cluster is most commonly deployed in one region, with support for running workloads across multiple availability zones. Most customers who are taking advantage of pgEdge Distributed Postgres operate nodes in different regions for performance or availability reasons, sometimes across multiple Cloud providers.Deploying across multiple Kubernetes clusters with pgEdge Distributed requires addressing two aspects:
  • Network Connectivity
  • Certificate Management
These domains are well known in the Kubernetes community as part of operating other multi-cluster workloads, and customers often have solutions in place to manage them, so building a single approach into pgedge-helm doesn’t make sense.Instead, the new chart includes a few configuration mechanisms to support multi-cluster deployments:
        In order to apply these to a multi-cluster scenario, you can utilize these configuration elements across deployments in multiple clusters.For example, let’s assume you want to deploy 2 pgEdge nodes across 2 Kubernetes clusters, with a single helm install run against each cluster. These values files highlight how to leverage these options, ensuring that:
        • Certificates are only issued during deployment to the first Kubernetes cluster
        • Spock configuration is applied across nodes in both clusters by the initialization job run in the second Kubernetes cluster

        Cluster A: cluster-a.yaml

        Cluster B: cluster-b.yaml

        This example assumes you have a cross-cluster DNS solution in place. If you want to simulate this type of deployment in a single Kubernetes cluster, deploying into two separate namespaces should provide a similar experience without needing to handle this aspect.We’ll be working to produce more blog content for multi-cluster approaches using different Kubernetes networking / certificate management solutions as we move ahead.

        Conclusion

        These updates mark an important step toward making pgEdge simpler, more flexible, and easier to integrate into Kubernetes environments.You can explore the new images and Helm chart today on GitHub:Whether you’re running in a single region or operating a multi-cluster deployment across clouds, pgEdge now provides the open-source foundation and tools to achieve your requirements in Kubernetes.Our team is here to help with your journey, including 24×7×365 global support from seasoned Postgres experts with decades of experience and direct contributions to the PostgreSQL community, with optional Forward Deployed Engineer services for dedicated assistance.Learn more and try pgEdge Enterprise Postgres for free - www.pgedge.com/download

        ]]>         https://www.pgedge.com/blog/pgedge-cloudnativepg-simplifying-distributed-postgres-on-kubernetes Matthew Mols         PostgreSQL <![CDATA[Managing DDL Migrations in a Multi-master Database]]> https://www.pgedge.com/blog/managing-ddl-migrations-in-a-multi-master-database Wed, 02 Jul 2025 05:50:00 GMT A DDL Migration is a set of DDL changes applied in a consistent manner, often via version-controlled scripts or tools, which enable a schema to evolve alongside applications that use it. In an active-active multi-master database, properly managing DDL migration is key to maintaining replication health across your database nodes.Generally, DDL changes in a multi-master database must be coordinated across all nodes to prevent divergence and to ensure that replication can appropriately apply ongoing changes to each node.pgEdge provides Automatic DDL Replication (Auto-DDL) to make it easier to manage changes to your database schema. This feature allows you to make your DDL changes against a single node and have them replicated to other active nodes. With Spock’s powerful capabilities, you can also take full control of schema updates and their associated replication configuration using Replication Sets.As you consider moving your applications to use a multi-master database, you should consider employing the following strategies to improve your application’s write availability and to limit unexpected downtime due to DDL changes that can be made more safely and efficiently.

        Make Backward-compatible Changes

        To ensure the safe evolution of your schema, it’s best to make safe, backward-compatible changes to your schema that ensure writes can continue to be replicated to other nodes without needing  downtime.These guidelines can help you craft backward-compatible changes:
        • Ensure new columns are nullable or use a default value, to ensure that write transactions that are currently replicating can safely land on other nodes, regardless of the migration state.
        • Avoid renaming existing objects (tables, columns, indexes, etc).
        • Use a phased deployment approach for non-backward compatible changes, such as column, table, or index renames:
        • ○
        • On the first deployment, add the new column and begin dual writes to the old and new columns to populate the new column
        • ○
        • On the second deployment, adjust your application to apply read and write transactions only to the new column.
        • Avoid reusing object names across multiple dependent migrations:
        • ○
        • For example, if you are dropping a constraint and introducing a newer version of that constraint, use a different name so it’s easier to recognize the migration status, and perform corrective actions.
        • Avoid changing types for existing columns:
        • ○
        • Instead, introduce a new column with the new type and populate it through application logic or triggers to keep it in sync with the original column.
        • ○
        • Once your application has migrated write transactions to the new column, you can then safely remove the old column in a separate migration.
        • Avoid removing an object (table, column, index) until it is no longer used by application code.
        • Utilize pre-migration validations when applying UNIQUE or NOT NULL constraints, which safely skip these migrations (and any that depend on it) if the underlying data is not valid.
        • Use
        • CREATE INDEX CONCURRENTLY
        •  separately on each node if you want to avoid blocking other queries in the database.
        • Use
        • IF EXISTS / IF NOT EXISTS 
        • where possible in your DDL to defend against migrations being run multiple times:
        • ○
        • This also gives you an escape hatch to make manual corrective changes on specific nodes when queued DDL statements are not completing, as they will become a No op.
        • For more complex migrations that include longer operations, it may be best to apply your changes individually on each node using
        • spock.repair_mode
        •  rather than making them with Spock's AutoDDL functionality:
        • ○
        • This may require coordination / scripting to run across all nodes, with verification performed to ensure it is complete.
        • ○
        • If necessary, you can leverage
        • pg_advisory_lock
        •  in coordination with your application to better control the behavior of other write statements to the database.
        Regardless of the approach you take to your DDL changes, it’s best to always test and verify the changes on a staging environment before rolling them out to production.

        Run Migrations Once when Using AutoDDL

        When you are relying on AutoDDL to propagate DDL changes to other nodes, it’s generally best to run your migrations only once against a single, dedicated node.If you are leveraging DNS routing to your nodes or using a load balancer that routes writes to multiple nodes, you run the risk of creating a situation where DDL changes with dependencies are applied to different nodes out of order, causing changes to not apply cleanly due to missing dependencies, resulting in schema mismatch.This issue can be made worse when using migration tools, including ones bundled with Object-Relational Mappers (ORMs). Migration tools typically rely on a migration state that is stored in a database table and updated as migrations are successfully applied in their desired order to a single node.If your changes are applied on application startup, multiple deployments of your application may attempt to apply the same DDL changes multiple times. This can generate errors and break replication across your subscriptions if the replicated statements do not prevent this scenario.To counteract this, you should configure your applications to only run DDL changes from a single migration or application instance. If you're using a standalone migration tool, such as the Liquibase CLI, you should only run that tool against a single pgEdge Postgres instance.If you are leveraging a library that runs as part of your application startup, you may need to introduce an environment variable like  and wrap any library code involved in the migration so that it only executes on one application instance.Migration documentation for various libraries and tools is linked below. If you are using a home-grown solution, the same approach should work to ensure your changes are only run once.

        Python

        • SQLAlchemy + Alembic
        • ○
        • https://alembic.sqlalchemy.org/en/latest/tutorial.html
        • Django ORM
        • ○
        • https://docs.djangoproject.com/en/stable/topics/migrations/

        Go

        • GORM
        • ○
        • https://gorm.io/docs/migration.html
        • Golang-migrate
        • ○
        • https://github.com/golang-migrate/migrate/blob/master/README.md#usage
        • Ent
        • ○
        • https://entgo.io/docs/migrate/

        Node.js

        • Prisma
        • ○
        • https://www.prisma.io/docs/orm/migrate
        • Sequelize
        • ○
        • https://sequelize.org/docs/v6/other-topics/migrations/
        • TypeORM
        • ○
        • https://typeorm.io/migrations

        Java

        • Hibernate
        • ○
        • https://docs.jboss.org/hibernate/orm/current/userguide/html_single/Hibernate_User_Guide.html#schema-generation
        • Flyway
        • ○
        • https://flywaydb.org/documentation/usage/
        • Liquibase
        • ○
        • https://docs.liquibase.com/

        Make DDL Changes in Advance

        With the popularity of migration tooling integrations with applications, it’s common to perform migrations during startup and then immediately begin leveraging new tables and columns in newly deployed application code.This can cause issues in a multi-master system, especially when deployments may contain many DDL updates that may take time to apply across all nodes, even with AutoDDL managing those changes. DDL changes are queued by Spock, and applied in order, so if one change takes longer than expected, other database nodes may not be updated immediately.In that event, your application may begin to exhibit errors if connecting to those other nodes, because it expects new tables and columns to be available for write transactions. In general, making DDL changes well in advance of when they are actually needed will help to avoid this problem.One strategy to avoid this problem is to perform migrations as their own deployments. DDL changes typically occur early in the development process for new features, and if you are using a continuous delivery approach, you can make smaller deployments that are only meant to migrate the database. When you combine this approach with backward-compatible DDL changes, your application should not be impacted by new columns and tables that are not required for functionality.Another strategy is to run data migrations prior to application deployment. Within your deployment process, add a step prior to application deployment that performs and verifies migrations against your database. If you are deploying your application in Kubernetes, an init container or pre-deployment job can achieve the same outcome.Data migrations should be verified using a separate verification script, ensuring they are applied on all nodes before moving on to deploying the application. If you integrate this separate step into your CI/CD process, it should be easy to set up alarms and notifications to know when something may be wrong

        Monitoring Migrations and Resolving Problems

        It’s important to be able to monitor the status of schema migrations across all nodes in the system to know when they have succeeded.In general, you should monitor the PostgreSQL logs to ensure that there were no issues applying changes to other nodes within the system. Look for spock apply errors to see if there might be issues with updates taking too long, or not being applied due to a previous mismatch.A good way to understand where differences may exist is to utilize the schema-diff capability in the Active Consistency Engine (ACE) to identify differences in your schemas across nodes. You can then understand if a previous mismatch is preventing a DDL statement from being applied, or where your schema is out of sync. ACE could be run after your migration process to ensure the schema is stable before proceeding to deploy applications.If you identify any differences across nodes, you can fix up individual node(s) using a manual DDL update with spocks’ repair_mode feature enabled. This can help to unblock a queued DDL change that may have relied on the existence of a specific column or table without also applying that DDL to other nodes.If you make manual updates to resolve inconsistencies using , you may need to delete a queued DDL statement from  to allow other DDL changes and data replication to be unblocked. This is uncommon if you follow the best practices highlighted here, but may be necessary if a troublesome DDL migration is applied that is causing timeouts within Spock’s apply worker.

        ]]>         https://www.pgedge.com/blog/managing-ddl-migrations-in-a-multi-master-database Matthew Mols