Scaling Without Stopping: Inside pgEdge Distributed Postgres Zero-Downtime and Exception-Resilient Replication
pgEdge Distributed Postgres v25 introduces several new features that make managing and scaling distributed PostgreSQL clusters dramatically easier and more resilient. Among these, zero-downtime node addition and a new Apply-Replay mechanism for replication exception handling stand out for their ability to improve operational efficiency and system stability, allowing teams to scale and more effectively handle a wide range of runtime scenarios with minimal disruption.
What Is the Spock Extension?
Spock is pgEdge’s advanced logical replication extension for PostgreSQL. It powers active-active, multi-master clusters with support for row filtering, column projection, conflict handling, and more. Spock is a core component underpinning both the self-hosted pgEdge Distributed postreSQL: VM Edition and the managed pgEdge Distributed postreSQL: Cloud Edition from pgEdge.
While Spock is descended from earlier projects like pgLogical and BDR 1, it has evolved far beyond its roots. Backed by a dedicated team of PostgreSQL experts, Spock has undergone continuous innovation and improvement to become a high-performance, enterprise-grade replication system built for distributed environments.
Feature Spotlight: Zero-Downtime Node Addition
Adding a new node to a live cluster has historically meant a tradeoff between downtime and complexity. In Spock 5.0, zero-downtime node addition eliminates this tradeoff entirely.
This feature allows you to add a new PostgreSQL node to an existing Spock cluster without requiring any downtime on the origin or existing subscriber nodes. The feature works by creating a temporary replication slot and subscription, and allowing the new node to clone the origin’s state in parallel. Once synchronization is complete, the temporary slot is retired, and the new node is promoted to a fully active peer.
Benefits of Zero-Downtime Node Addition:
No service interruption or replication pause
Safe scaling of production clusters
Minimal manual intervention
Works with the standard Spock CLI or via scripted workflows
This workflow is based on a coordinated process that ensures the new node joins the cluster cleanly and consistently:
Initialize Subscriptions (excluding the source): The new node is configured with a disabled subscription to each existing node except the node designated as the source. Each existing node begins buffering new transactions for the new node via a dedicated replication slot.
Ensure Transaction Synchronization: Before initiating the data copy, the source node is fully synchronized with all in-flight transactions from the other nodes. The spock.sync_event() and spock.wait_for_sync_event() functions ensure this precondition is met.
Copy Data from Source to New Node: The source node establishes a subscription to the new node that synchronizes the data and structure, and initiates a snapshot-based copy to bring the new node current to the point of the copy.
Enable Remaining Subscriptions: Once the copy completes, the new node activates its subscriptions to the other existing nodes. Spock calculates the last commit timestamp for each node (as seen via the source) and advances each node’s replication slot to that timestamp. This prevents duplicate transactions and allows replication to begin at the appropriate point, receiving only the new transactions that occurred after the initial copy.
This approach ensures a seamless and accurate integration of the new node without interrupting activity in the existing cluster.
This same workflow can be used to perform seamless, in-place major PostgreSQL version upgrades across your entire cluster. By introducing a new node running the desired higher version of PostgreSQL, and following the coordinated steps for data synchronization and slot management, you can bring an updated node into the cluster without interrupting read or write traffic. Once the new node is in place, older-version nodes can be removed or replaced one at a time, performing a rolling upgrade with zero downtime. This approach provides a safe, flexible method for upgrading infrastructure while maintaining full application availability.
For users ready to implement zero-downtime node addition in their own environment, the Spock documentation offers a step-by-step guide to the full process.
To further support implementation, the Spock GitHub repository provides several working examples:
A Python-based orchestration script, designed to run outside the database and coordinate the node addition via external automation tools.
A stored procedure version that performs the entire process within PostgreSQL itself using the dblink extension, offering a fully internal option ideal for controlled or restricted environments.
You can explore these examples in the samples/Z0DAN directory.
For more complex or scripted rollouts, pgEdge also provides spockctrl, a lightweight command-line orchestrator written in C. Spockctrl accepts a structured JSON plan that defines the sequence of operations to perform, including a sample JSON file demonstrating how to add a new node. Both the tool and sample configurations can be found in the spockctrl directory.
Ensuring Seamless Node Addition with LSN Checkpointing
pgEdge’s Spock 5.0 extension includes functions that make seamless node addition with zero downtime possible. LSN checkpointing (using the spock.sync_event() and spock.wait_for_sync_event() functions) allows you to create a logical checkpoint in the WAL stream on the source node, and then monitor another node for the arrival of that checkpoint's LSN to ensure that all transactions have completed. When adding a node, you can use this to guarantee that schema or data changes have been fully replicated to your source node before you continue.
Used in the context of zero-downtime node addition, these functions are critical. When you have confirmed that any in-flight transactions (from all nodes) have arrived on the designated source node, you can initiate a data copy to the new node. Without this precise synchronization, adding a node without interrupting cluster usage would not be possible; Spock's checkpointing guarantees safety and consistency as part of the overall orchestration strategy.
Feature Spotlight: Apply-Replay for Exception Handling
Spock has always been more resilient than earlier logical replication solutions. Neither pgLogical nor BDR 1 included any automatic error recovery. Spock, by contrast, introduced automated exception handling early in its lifecycle.
Exception Handling in Early Spock Versions
nitially, when a data conflict occurred during replication that couldn't be automatically resolved, the apply worker would restart itself, re-request the transaction from the origin, and continue from there. This was already a step above legacy approaches. Spock also allows you to configure how these exceptions are handled: you can pause replication and discard the problematic transaction, or Spock can step through each sub-transaction to isolate only the part of the transaction that caused the exception.
Although better than the other open source solutions, this exception handling process came at a cost:
The apply worker had to terminate and restart
The transaction had to be re-fetched over the network
XID resources were consumed during replay
The overall process could introduce some replication lag and resource overhead
Taking a New Approach in Spock 5.0
With Spock 5.0, this process has been vastly improved via a new “Apply-Replay” mechanism.
Now, when a replicated transaction encounters an exception, the apply worker does not terminate. Instead, Spock buffers transactions in memory up to a default of 4MB (configurable via a new GUC: spock.exception_replay_queue_size). If an exception occurs, the apply worker enters exception-handling mode and simply replays the buffered transaction from memory.
Benefits of Apply-Replay:
No worker restart required
No need to re-fetch from origin
Dramatic reduction in lag during exception handling
Reduced XID usage and less WAL churn
This enhancement significantly improves replication stability, especially in high-throughput environments or under intermittent network conditions where transient data conflicts could previously cause substantial delays.
In a synthetic benchmark test involving 10,000 intentionally triggered conflicts that would cause an exception, the Apply-Replay feature resolved all exceptions in just 3 to 4 seconds—compared to over 5 minutes using the previous approach. This represents a dramatic leap in both speed and efficiency for exception handling in distributed PostgreSQL clusters.
Large transactions that exceed the memory size will still use the old process, but we are currently at work making those transactions that exceed the allotted memory to be written to, and replayed, from disk, allowing us to completely retire the previous approach.
Why These Features Matter
Both zero-downtime node addition and Apply-Replay reflect a core philosophy behind Spock 5.0: to eliminate avoidable disruption and make high-availability PostgreSQL truly hands-off at scale. These improvements:
Reduce operational complexity
Lower the risk of human error
Improve cluster elasticity and resilience
Enable real-time distributed applications to stay online and in sync
Truly allow full zero-downtime for reads and writes across all nodes in a cluster while expanding or upgrading.
Final Thoughts
The pgEdge Spock 5.0 extension isn’t just a version bump—it is a major change to how robust, fast, and easy logical replication can be. Whether you’re managing global clusters or network edge deployments, the new features highlighted here will help your team scale smarter and operate more effectively. And these are just part of a broader set of enhancements: Spock 5.0 also includes improved automatic conflict resolution that can now handle more conflict scenarios without user intervention, along with other performance and usability upgrades that make it the most capable version yet.
Spock 5.0 is available now as a fully integrated component of both the self-hosted pgEdge Distributed Postgres: VM Edition 25.2 and the managed SaaS pgEdge Distributed Postgres: Cloud Edition offering. Whether you’re running your infrastructure on-premises or in the cloud, you can take advantage of the powerful features described in this post, plus many others, to build fast, resilient, globally distributed PostgreSQL applications.
To learn more about pgEdge’s distributed multi-master replication technology, visit the pgEdge website or explore the pgEdge documentation.
