pgEdge Posts from Shaun Thomas

Looking Forward to Postgres 19: Syntax Potpourri

Fri, 07 Aug 2026 10:41:27 GMT

Before diving into this final post covering new Postgres 19 features, I just want to say it's been a wild ride. Postgres 19 has been a veritable treasure trove of enhancements, perhaps more than any previous release; or maybe that's just my perception. Usually I just skim through the release notes and nod along, sometimes jotting down things that look interesting for later study. Maybe calling out each element that caught my eye was the right thing to do, to really show how far Postgres has come since the last release, rather than simply accepting the status quo. It's easy to miss an otherwise innocuous one-liner in a changelog.In any case, every major Postgres release ships with a marquee feature or two, and I've covered several of them in this series on Postgres 19. Some will garner dedicated conference talks or a flurry of frantic blogs from equally zealous authors. The big-ticket items always get their time in the spotlight. But I'm not here to talk about that this week. Like the last article on a handful of unassuming new function calls, this week will focus on some syntax tweaks that may never really trend anywhere—the lost and unsung.Why dedicate a whole article to syntactic sugar? Why fuss over features that, taken one at a time, save maybe four lines of SQL apiece? Because four lines of SQL, multiplied across every project we'll ever touch, becomes a veritable mountain of saved aggravation. Postgres 19 happens to bring a whole potpourri of these, and a few of them scratch itches I didn't even know I had.Let's wander through my notes and see what turned up.

Something Old, Something New

Prior to the newer MERGE syntax, the humble ON CONFLICT clause was the Postgres Way™ to combine and into a single statement. It can and shrug off the duplicate, or and merge new values into the old row. That's still its primary role.But there's a third thing we want constantly, and Postgres never had a clean way to express it: get-or-create. Insert this row if it's new, and either way, return the row that's there now. The clause is supposed to do this, but users encounter a rude awakening when they actually try to leverage it. You see, only works if something actually happened.Let's create a demonstration table with a sample row for this:Here's what happens if we try on a action:Technically correct, since did not produce any results to return. It's overly pedantic and ultimately not what the user likely wants. Here's the "fix" for that:But what did we actually gain here? We've abused the clause by performing a no-op update simply to ensure there was something to return. That causes a pointless write, bloats the table, and burns a tuple every single time. As a result, the syntax left us empty-handed the one time we most wanted the existing row's details.Now with Postgres 19, that behavior is finally a thing of the past thanks to the new DO SELECT syntax. Instead of updating the conflicting row or ignoring it, returns the existing row that collided with our proposed insert.Here's how that duplicate insert attempt turns out now:Look closely at that value. We proposed , but handed back , the value already stored. This action returns the existing row, not the one we tried and failed to insert, exactly as expected. When there's no conflict at all, the insert proceeds normally and reports the brand new row:One statement, two potential outcomes, exactly one returned row. It's what everyone thought was doing this whole time. The problem was never , but .But did you notice Grace landed on 5 rather than 2? If you have been following along with this example, you'll recall we've executed several statements until now. I covered this behavior in Logically Sequenced, but deserves special attention. In order to work properly, preemptively obtains a value from the sequence associated with the identity column. Since they're not transactional, sequences don't rewind. That gap is expected and harmless, but often surprises users when they end up burning through more values than expected due to high insert conflict counts.

You Shall Not Pass!

Returning the existing row is great, but a get-or-create rarely exists in isolation. It usually sits inside a transaction that's about to do something with that row, which means we have a classic race on our hands. Between the moment hands us the row and the moment we act on it, another session could waltz in and change it out from under us.Can we reserve the row for ourselves? It turns out that accepts the same row-level locking clauses as a normal , so we can lock the returned row against concurrent modification right in the statement:That locks Ada's row exactly as if we'd selected it the long way around, which closes the window where someone else could modify her account before our transaction commits. The whole family is supported: , , and all work too, so we can pick the weakest lock that keeps us safe.There's also an optional clause to decide whether the conflicting row even comes back. The condition filters the returned row, so we only get it when it matches:Ada's display name isn't , so the row is filtered out and we get nothing back, even though the conflict absolutely happened.There are two additional things to keep in mind when using . Because we're reading data back out, the statement requires privilege on the table, not merely . Secondly, a clause is mandatory. Leave it off and Postgres expresses its ire:Just think of the clause as the column list we'd normally specify for a statement.

Group Therapy

Things in Postgres 19 improve when we want to handle multiple rows at a time as well. One very long-standing complaint about is that it requires manually listing every single non-aggregate column. Boo! Hiss! Well, that all ends with the introduction of GROUP BY ALL. It gathers every select-list expression that isn't an aggregate and groups by all of them automatically.Consider a small table:The old way makes us name and twice:And here's the same query, letting Postgres infer the grouping for us:Identical results without all of the column duplication shenanigans. And it doesn't end there; computed expressions also come along for the ride. Consider this example with What about window functions, which can legitimately reference an aggregate? Postgres treats them the same way it treats aggregates and leaves them out of the inferred grouping. So we can rank our regions by total without accidentally becoming a grouping key:If the select list contains no aggregate or window function whatsoever, groups by everything, which makes it behave just like . Handy, occasionally surprising, and yet does what it says on the tin. And actually, given how flexible this is, I'm personally never going back.f there's any weakness here, it's that it might work too well. Since it always re-groups the results no matter how many columns are present, existing reports might suddenly grow unexpected new columns instead of resulting in an error. That could cause unexpected behavior in downstream parsers or spreadsheets that don't operate with column headers and only consider position. The lack of query errors makes that kind of thing harder to find. Of course, that’s what good CI/CD is for!It's still worth it in my opinion, but your mileage may vary.

COPY That

Now we've got a tidy aggregated result. Naturally, the very next thing someone requests is for that result as JSON, because it's always JSON.No worries! That's what , , and the other JSON companion functions are for, after all. But that only really helps if we already have a program which can immediately process JSON output. What about exporting that to an actual file on disk?Well, we can try to export a single-column CSV with no header:But that ends up looking like this:That's not... quite what we're looking for. Sadly, the only real way to perform this magic is by using some kind of program logic. Either to pipe through to reformat properly, or to otherwise JSONify the contents instead of Postgres. Who wants to maintain that?Instead, Postgres 19 teaches to speak JSON directly through a new format option. By default it emits newline-delimited JSON, one complete object per line, which is the format streaming tools and log pipelines tend to expect:There's a shorthand too. Since can only mean a format here, we can drop the keyword entirely and write . Naming a column list narrows each object to just those keys, exactly as it does for CSV.The real payoff shows up with nested data. There are ways to combine , , and the other JSON functions to build the nested structures, but that’s a lot of work. doesn't have to worry about any of that. Arrays become JSON arrays and columns nest as standard objects:The array and the object both land as real JSON, without the need for escapes or convoluted string mangling. Unfortunately this only works for JSON output; there is no equivalent yet:Additionally, don't try to use , , , or any of the other CSV-only format controls, as JSON has no need for them. But what it does support is a new parameter.Why? Newline-delimited JSON is perfect for streaming, where each line is independent and a consumer can process tuples as they arrive. But the output as a whole isn't technically valid JSON. A real JSON validator or parsing library will likely reject the output and leave us somewhat embarrassed.This is what happens if we add to the command:The leading-comma style looks a little unusual, but it's perfectly valid, and now the whole blob parses as a single JSON array containing three objects. That makes it a natural fit for writing a complete file suitable for a downstream tool. In natural Postgres fashion, this also applies to the psql equivalent:The result is actually well-formed, and as a result, it should be possible to hand to a JSON parser, nested objects and all. In fact, let's do that with the utility:That wouldn't have worked if the file wasn't actually valid JSON. And with that, we've gone from "export some rows" to "produce a valid JSON document" without writing a single line of application code.

Final Thoughts

None of these enhancements are likely to feature in the Postgres 19 release notes overview where the most prominent items appear. , , and are the small mercies, the quality-of-life touches that help us eliminate a few lines of boilerplate in favor of productivity. They're how we actually get things done day by day.Well... maybe. The jury is still out; there's a chance shows up in the overview. That's genuinely a massive shift in how people may end up using Postgres on an everyday basis. Imagine never having to modify the section of a query when adding or removing columns from the list ever again. The amount of unexpected syntax errors just dropped by an order of magnitude from that alone.Anyway, what are you waiting for? The Postgres beta is just sitting there waiting to demonstrate all the amazing new capabilities I've covered over the last few weeks. I know you have JSON. I know you have queries. I know plenty of users annoyed by how used to work with . If you're not already scrambling to download Postgres 19 purely based on this article, I'm not sure how else to convince you.And if Postgres 19 contained enough augmentations to fuel several weeks of exposition, I can only imagine what Postgres 20 will bring!

Looking Forward to Postgres 19: The Cult of Functionality

Fri, 31 Jul 2026 09:31:21 GMT

Postgres ships with a genuinely bountiful catalog of built-in functions. There are literally thousands of them, covering everything from trigonometry to JSON path queries to full-text ranking. It's a kitchen sink of staggering proportions. Somehow despite all of that, there are actually still a few gaps to fill.Need a random date for some test data? That's going to be an entire expression, I'm afraid. How about the exact definition of a role, tablespace, or database? You better have a GUI database tool like pgAdmin or access to the command with a little assistance from .It's little annoyances like this that Postgres 19 aims to alleviate. Let's explore these new quality of life enhancements!

In the Near Future

Suppose we want a random date sometime in 2026. The canonical approach for this has been to begin with January 1st and add a random amount of days:It's kind of a hack, but it works. Aside from not accounting for leap years, there's also something else to take note of:We asked for a date and got a , because adding an interval to a date promotes the whole expression. If we actually wanted a , we have to wrap the result in another cast. If we wanted a , that's a different cast. That's not exactly critical, but type sensitivity varies across platforms.Either way, that's a lot of ceremony for "give me a random date."So Postgres 19 adds a family of overloads that take an explicit lower and upper bound. The we all know and love is still around in the mathematical functions, but now there are three new temporal definitions in datetime functions.There's one for each of the valid Postgres date and time types:A random date is now just , and the result is a real . No interval multiplication, trailing cast, or surprise timestamp conversion. Ditto for and . The inclusive bounds also address the issue we had with leap years, where adding 364 days could mean never reaching December 31st.You may have also noticed the matching significant figures in the output. If you haven't ever used that before, it works universally. Here it is with four digits of precision:All of this takes the guesswork out of the function. Type goes in, type goes out. Everyone can explain that.

A Short Aside About Seeds

Random data is most useful when it's reproducible. A test that fails on one random dataset and passes on the next can be worse than no test at all. Sometimes, we want the same "random" values every time we run a script. For that, there’s the function:Execute those two commands in a fresh session and the random output will always be the same. That allows us to deterministically generate sample data in bulk with the aid of : Five hundred customers, orders scattered across the year, all in one readable statement. Bumping the series to a million nets us a respectable load-testing fixture. Neat!What else does Postgres 19 have for us?

The DDL Firehose

Quick! Without using some GUI, what is the exact definition of the role on this server? Until now, the only way to do something like that was with the pg_dumpall cluster-wide dump utility. Why? Because roles, tablespaces, and databases are global objects living outside of any single database. Short of mining the system catalog, there's been no historical way of doing this.Here's how it used to work:It gets the job done, but consider how awful that interface is. We can't use standard SQL, so it's not possible to join the result against the catalogs, filter it, or feed it into another statement. We're relying on an external binary, which means either local or remote shell access. If it's local, our must be the same version or newer than what's on the server. And the utility dumps every global object we request, meaning subsequent filtering. It's a mess!Well, it was a mess.

On a Role

Postgres 19 introduces three new information functions designed to interrogate the database on the previously obfuscated global objects. The first of these is , which shows exactly how to recreate a named role.Let's build a role with enough going on to be interesting:Now, instead of dumping the entire cluster's worth of roles and hunting for this one, we can ask for it directly:Everything is there: attributes, per-role settings, even the membership grant into . If you’re wondering why it says rather than , remember that the latter is just an alias for . And notice what is conspicuously absent: not even a hash of the password. This is a deliberate choice to avoid leaking sensitive material. That restriction doesn't apply to , so that information is still available, just not to a user-callable function.The function currently accepts two optional settings as name and value pairs after the role. If the membership grants are unnecessary, disable them with : And for a human to read the output rather than a script, breaks the attributes onto their own indented lines: One function, one query, no shell.Roles, though, are only one of the three global object types that used to send us reaching for . Let’s take a look at the other two.

Databases and Tablespaces

The other two new DDL information functions focus on databases and tablespaces. These are fairly low-level objects, and tablespaces in particular drive Postgres DBAs nuts. The view does not tell us where the tablespace lives. We need the function for that.Consider this tablespace:Say it's been a while and we forgot where the tablespace went, and it wasn't documented for whatever reason. Previously, we had to do this to get all of the tablespace information:Instead of that, why not just retrieve the tablespace definition itself? Now let's use the new function:Both the location and the storage settings arrive intact. The owner is also just a username rather than a system catalog oid that requires a join to resolve. Maybe I’m lazy, but sometimes I just want to see the DDL for an object without jumping through a bunch of hoops.Databases work the same way. Consider this busy database definition that even uses the new tablespace:The function makes short work of it:That's the identical output we once needed to obtain with the assistance of . Now it arrives as the result of a query with no external tooling or schema dump to wade through.Each of these accepts the same style of name and value options, so we can suppress the line, drop a database's clause, or request formatting:Imagine the possibilities! Check this out:Try doing that with ! Throw on a timestamp and we can audit database definitions over time. SQL does make everything better, after all.

Final Thoughts

Little tweaks like this improve QoL in a way users don't exactly expect, but may find pleasantly surprising. "Oh, Postgres does that? Nice!" Generating a random date is now much easier and less error-prone. Obtaining DDL for global objects no longer requires external tooling like a shell utility or a GUI admin client. Now it's all just a function call.I won't pretend the DDL functions are a complete analog for and . They currently cover global objects and nothing more, so there's still room to grow. There have long been functions for retrieving function, trigger, index, and view definitions, so it would be nice if every object had a DDL extraction equivalent. We don't have a full onboard introspection library just yet, but it's getting closer.In the end, Postgres 19 makes the engine just a tiny bit more appealing in ways that don't require a lot of fanfare. This isn't the headline catching query hint system, or temporal tables, or REPACK syntax, or any of the other more noteworthy topics I've covered in this series so far. They're just examples of the long and steady forward march of improvement each release tends to bring.And Postgres 19 still has a few more tricks up its sleeve - more coming next week. Stay tuned!

Looking Forward to Postgres 19: Autovacuum Tweaks

Fri, 24 Jul 2026 12:34:46 GMT

The autovacuum process has been dutifully tending Postgres tables since version 8.1, and almost every release since has acquired some new refinement: smarter thresholds, insert-aware triggering, resource cost limits, and so on. It's the reliable maintenance workhorse that heroically forestalls transaction ID wraparound and keeps statistics fresh. So how did the devs improve it this time around?For most of its life, an autovacuum worker built its list of tables needing attention and then worked through them in roughly the order they appear in the catalog. Very egalitarian. But a table one transaction away from triggering a wraparound shutdown got the same treatment as a table that crossed a statistical update threshold. Any implied urgency is lost in that context. So how do you teach a maintenance daemon the difference between sweeping the floor and putting out a five-alarm fire? Can an administrator dictate that the cluster cares more about freezing rows than reclaiming space without rewriting the whole scheduler? There's only one way to find out!

Keeping Score

The mechanism behind the change is a prioritization system that scores every table before the worker decides where to start. The launcher picks a database first, favoring any database skirting transaction ID or multixact wraparound, then deferring to whichever went longest without attention. Within that database, the worker builds its candidate list and sorts it by a numeric score instead of catalog order.That score is a weighted heuristic. Postgres computes five separate component scores for each table and keeps the highest as the table's overall priority. The components are the transaction ID age, the multixact ID age, the count of dead tuples waiting to be reclaimed, the count of freshly inserted tuples, and the volume of changes since the last . Whichever of those is highest decides the table's place in line.We don't have to take any of this on faith, because Postgres 19 also exposes a new pg_stat_autovacuum_scores view that reflects the current scores for every table in the database:The column is that overall maximum, and the five columns are the raw components feeding it. Numbers in a catalog view are abstract, though. Let's make some real ones.

Two's Company

Consider two tables with completely different patterns. The first is an append-only event log that only ever grows. The second is a work queue that churns constantly as jobs flip state and get cleared out. We can disable autovacuum on both so the daemon doesn't automatically clean up before we get a look at the scores:The log absorbed half a million clean inserts. The queue saw two hundred thousand inserts, then experienced a hundred thousand updates and forty thousand deletes, leaving plenty of dead tuples. After nudging the statistics collector with , here's how Postgres scores them:Each component is essentially a ratio: how far past its trigger threshold the table has drifted. The append-only log has zero dead tuples, so its is a flat zero. It has half a million inserts against an insert threshold of 1000, giving an of 500. And every one of those inserts counts toward analyze, which against the default threshold of fifty rows works out to an of 10000. The queue tells a different story: a hundred and forty thousand dead tuples push its up to 2800, while its mix of inserts, updates, and deletes lands its at 6800.The overall score for each table is simply the loudest of its constituents. For both tables here, that happens to be the analyze score, so the log outranks the queue, 10000 to 6800. Maybe that's exactly what you want. But maybe you run a system where reclaiming dead space matters more than refreshing a few statistics, and you'd rather the churning queue jumped the line. That’s now an option.

A Thumb on the Scale

This is where the six new tuning knobs earn their keep. Five of them are scaling factors meant to multiply against the matching component score before determining the maximum.Each weight defaults to , meaning all concerns are treated equally. Raising to doubles the priority of the analyze component. Lowering it to cuts analyze scores in half. There's even a documented escape hatch: set all five weights to and the cluster reverts to the pre-19 strategy of plain catalog order. Just in case the old behavior works better for some reason.So let's put our thumb on the scale. Suppose we decide reclamation is twice as important as it used to be, and statistics refreshes are worth only half. These are parameters, so a configuration reload is enough to apply them with no restart:Now the same two tables sort differently:How about that? The queue's vacuum component doubled from 2800 to 5600, its analyze component halved from 6800 to 3400. The log's analyze component fell from 10000 to 5000. Now the maintenance priority has flipped. The churning queue that used to sit behind the log now leads it, purely because we told the cluster that dead space bothers us more than stale statistics. That single policy decision rearranged the entire maintenance schedule.Notice that the reweighting touched the vacuum and insert numbers independently. Let's get into why inserts are tracked separately. Veterans are already aware of this distinction, but it's an interesting distinction worth exploration.

Inserts Are Special

It's tempting to lump all table activity together as "stuff that makes autovacuum work harder," but inserts and updates create fundamentally different messes. An or a leaves dead tuples behind, and dead tuples are the bloat that the vacuum in autovacuum exists to reclaim. A plain doesn’t have this effect at all, and in fact may even replace a previously reclaimed slot. The append-only log above proved it: half a million rows added, and a of exactly zero, because there was nothing to clean up.So why does an insert-only table ever need an autovacuum at all? First, those rows still shift the table's statistics. A query planner working from a stale row count on a table that grew by half a million entries will probably make some genuinely terrible decisions, so insert activity should eventually trigger an analyze. Second, every row Postgres has ever written carries a transaction ID that must eventually be frozen, whether anyone updates the row or not.That's the reasoning behind giving insert activity its own insert threshold back in Postgres 13. We could manage a data warehouse packed with append-only fact tables where the vacuum component is almost always zero and the real maintenance pressure comes entirely from inserts. A system like that would mainly focus on .That said, most systems should probably focus on the freeze weights.

Frozen Landscape

Postgres tracks row visibility with transaction IDs, and that counter is finite. Left unfrozen, the oldest rows in a table inch ever closer to the wraparound horizon. Once a database actually reaches that point, it stops accepting writes until Postgres completes an emergency vacuum. Freezing is the maintenance that rewrites those old transaction IDs as permanently visible, and unlike reclaiming dead space, it is not optional. Every row gets there eventually.That inevitability is exactly why the freeze scores deserve to outrank the others when things get tight. The transaction ID score measures a table's age against autovacuum_freeze_max_age, and the multixact score does the same for . As a table's oldest unfrozen rows age, its freeze score climbs, and a table approaching wraparound will eventually out-score all other weighted metrics.The two freeze weights, and , carry a subtlety the others don't. Increasing them above doesn't just multiply the score, it divides the thresholds by the weight. Pushing the freeze weight to 2.0 tells Postgres to start treating freeze pressure as urgent at half the usual . The max freeze age still applies, but score totals could mean a table gets frozen much earlier than we’d normally expect.There's also a hard backstop introduced in Postgres 18. The vacuum_failsafe_age and parameters, both defaulting to 1.6 billion, mark the point where Postgres stops being polite. Once a table crosses that age, autovacuum throws out its cost-based delays, skips non-essential work like index vacuuming, and bulldozes straight into freezing the table at basically any cost. So these two parameters aren't just an emergency brake anymore, but a heavy influence on the observable score.Sorting tables sensibly is only half the battle, though. Picking the right table to vacuum first doesn't help much if that table then takes an hour to finish.

Many Hands Make Light Work

The sixth new parameter is a different animal from the five weights. The autovacuum_max_parallel_workers parameter lets a single autovacuum worker recruit helpers to process a table's indexes in parallel during the index vacuuming and cleanup phases.Manual has been able to parallelize index work since Postgres 13, but autovacuum workers have long been relegated to handling one table and its indexes serially. As a result, a wide table with a dozen indexes could monopolize an autovacuum worker for excessive lengths of time.For now, it's necessary to set the new parameter to a non-zero value to opt in:Now a single autovacuum worker can launch up to four parallel workers when it reaches a table with enough indexes. Paired with the prioritization system, the two halves start to complement each other. The scoring determines the table priority order, and the parallel workers enable processing each as quickly as possible. Triage, then throughput.

The Shape of Things to Come

It's very tempting to look at six new Grand Unified Configuration (GUC) variables with names like and file them under "obscure dials I'll never touch." Most clusters will probably be just fine leaving the weights at . This is relatively unexplored territory, so inert defaults are a perfectly good starting position. The point of these knobs is merely that the autovacuum machinery finally has a vocabulary for priority at all.The fact that scores are constantly reflected in a view is great disclosure as well. Up until now, diagnosing autovacuum behavior meant reading source code and making educated guesses. Was it from a table approaching the max freeze age? Autovacuum threshold? Analyze scale factor? Something else? Now lays the reasoning out in plain sight. Heck, we can even monitor it in a dashboard.Then there's this note from the commit that introduced this prioritization system:That's pretty exciting, no? Autovacuum will continue to improve, just as it always has. Speaking purely as a former DBA in charge of a financial database that handled over a billion transactions per day, I can authoritatively state that we all sleep a little easier with every new autovacuum parameter. Even un-modified, it's comforting knowing they're available to address mysterious future emergencies. I can't wait to see what they come up with next.

Looking Forward to Postgres 19: Checkpoint Control

Fri, 17 Jul 2026 06:05:23 GMT

Postgres 19 is just a smorgasbord of new functionality; it's genuinely hard to believe they packed all these new features into a single release. To that laundry-list of new capabilities, it adds something I almost missed. We all know and love the Postgres Write Ahead Log (WAL), where all writes begin their lives. I even recently wrote about how the background checkpoint system can overwhelm storage when not properly tuned.What about the times when a DBA wants to purposefully invoke a manual CHECKPOINT to flush any pending writes to the heap? Just one quick command and Postgres invokes an immediate flush, then returns control once the dust settles. Until Postgres 19, that was the entire interface. . It's like a Zen koan.It turns out there were a few tricks we could teach this old dog.

A Blunt Instrument

A bare means "reconcile everything immediately, and don't come back until it's done." There are a few obvious scenarios where DBAs may execute this kind of targeted checkpoint:

Before taking a backup so the base image starts from a known-good state.

Prior to a
pg_upgrade
so the old cluster is fully settled.

Between benchmark passes so each run starts from the same clean slate. We even leverage that approach several times in this very article.

In every one of those cases, "as fast as possible" is precisely the intent. Given that only superusers or members of the pg_checkpoint predefined role can use it, that seems about right.The trouble is that its single behavior was also its only behavior. When the only tool you have is a hammer, everything becomes a nail. As we learned from the write storm article, sometimes a full and immediate flush is the last thing a busy server needs. If our means the next checkpoint won't happen for a while, maybe there needs to be an option to start the process early.And now with Postgres 19, there is.

A Challenger Appears

Improvements to the command in Postgres 19 came in three stages. The first added nothing user-visible at all. It simply taught the command to accept a parenthesized option list, the same design from , , and . The actual options arrived in the two commits that followed.The new grammar looks like this:Two options are recognized so far: and . It's possible to combine them in a single statement separated by a comma, in either order:Postgres enforces the grammar in a few ways. The parentheses are optional, but can't be empty if provided. Postgres also flags any invalid or unrecognized parameters:Newer clients also contain the usual helpful inline documentation:As usual, that's one of the reasons Postgres veterans find it hard to eschew psql in favor of other clients. It's just such a good CLI already!

A La MODE

The second phase of our new introduced the MODE option. It takes one of two values.

We can even examine how these modes differ in practice. Let's create a table with two-million rows, that ends up being about 517MB:Then we can dirty half of the pages in the table and request another checkpoint:A little over four hundred milliseconds, and the I/O all happened in that brief window. Now let’s try the same amount of dirty data, flushed with (after recreating the table for a fair comparison):That's over two minutes to flush the same amount of writes. Exactly as expected since Postgres is deliberately pacing the writes so the storage subsystem never sees a spike. The command still waits for completion before returning, so there's no ambiguity over when the flush has truly finished.

Two Men Enter, One Man Leaves

There's an interesting little quirk here that's easy to miss. What happens when two (or more) of these checkpoint requests collide? As it turns out, the server may consolidate them. Consider two sessions seeking a checkpoint at nearly the same moment. If one requests FAST and the other prefers , Postgres doesn't run two checkpoints back to back. Instead, it runs a single checkpoint with the attributes of the most urgent request. In this case, wins out and we end up with an immediate checkpoint. The same logic applies to , which we’ll cover in depth soon. If one session requests flushing unlogged buffers, that gets applied to the consolidated checkpoint.As a result, these options are more of a request than a contract. We can request a spread checkpoint or choose to ignore unlogged buffers, but any other session can effectively override those preferences. It's not totally unexpected, but something to keep in mind.

Unlogged Tables

The third and final phase of our new added a second option called . For the unfamiliar, Postgres provides unlogged tables which skip writing changes to the WAL. That makes writes dramatically cheaper with one major caveat: contents don't survive a crash. After an unclean shutdown, every unlogged table gets truncated to empty, because without WAL there's no way to guarantee its pages are consistent. Additionally, such tables can't be replicated.That bargain has a consequence for checkpoints. Since an unlogged table's data is forfeit on a crash anyway, an ordinary checkpoint doesn't bother flushing its dirty buffers to disk. Why pay for I/O on data that crash recovery will simply discard? As a result, those buffers sit in memory, ignored by checkpoint after checkpoint, and only get written out at a clean shutdown. We can watch that happen using the counter from pg_stat_checkpointer. Consider this short experiment using an unlogged table:Then like the logged variant above, we can modify the contents and invoke a plain checkpoint:No change. The checkpoint ran, dutifully reported success, and wrote none of those pages. For contrast, the identical operation against a regular logged table:Now we see a difference of 16,811 buffers. Logged data always gets flushed; unlogged data gets skipped. Now we can incur the new option against the same dirty unlogged table:This time we see a total of 13,989 buffers written, showing the pages actually land on disk. While it's disabled by default, there are a few scenarios where it might make sense to flush an unlogged table manually. Perhaps we're about to do a controlled, clean shutdown and would rather pay the flush cost early. Maybe data staged in an unlogged table should be physically persisted before some external process inspects the data directory. Whatever the reason, the choice is now available rather than forced by the engine.

Looking Forward

None of this changes what a checkpoint fundamentally is. The automatic checkpointer still hums along on its schedule, and the vast majority of installations will never need to type any of these new options. This is a small feature, and the documentation page is barely longer than it was before.But it's the kind of incremental improvement that DBAs probably notice. The manual checkpoint went from a single blunt instrument to something suitable to finer precision. Fast for hard flushes before a backup or an upgrade, spread to nudge the system without jolting it, and an option to flush unlogged tables to avoid unexpected shutdown delays. Christoph Berg's work here decorates the formerly unadorned we know and love, and it does so without breaking a single existing script.Postgres 19 is replete with such quality-of-life improvements, and we'll be exploring a few more of them in the weeks ahead. For now, if you've got a beta build handy, spin up a table and watch take its sweet time. Sometimes slower really is better.

Looking Forward to Postgres 19: Checksums For All

Fri, 03 Jul 2026 13:42:31 GMT

Data checksums are one of those Postgres features that, when they are doing their job, are easily forgotten. They sit quietly in the header of every data page as a small integer fingerprint, forever waiting to thwart the threat of cosmic rays or errant hardware failures. Most clusters run from cradle to grave and never trip a single one.For years, that decision was etched in stone at the time of database initialization. It wasn't until version 12 that Postgres introduced the pg_checksums utility to change it. And even then, doing so is a fully offline affair, grinding through every page on disk and incurring a long outage window.That's a fairly painful ordeal for a basic safeguard that wasn't even enabled by default until version 18. So why go through all the trouble in the first place? Do we really need data checksums in our Postgres cluster? The short answer is "yes". The longer answer explains why Postgres 19 continues to improve the checksum system by adding an online conversion capability.

Bit Rot Never Sleeps

Let's start with what a checksum actually defends against. Postgres is very good at protecting data from itself. Crash recovery, the write-ahead log, full page writes, all of that machinery exists to make sure a power failure mid-write doesn't leave a torn, half-updated page behind. But Postgres can’t help when the hardware itself lies about the data.Even with ECC RAM, that happens more often than you might expect. Cosmic rays can flip a bit in a memory cell. Failing drives may return stale sectors. Storage controllers could acknowledge writes that never made it to a platter. Any piece of the hardware, including the motherboard, CPU, and RAM, is suspect. In every one of these cases, Postgres asks for a page and the OS cheerfully returns something with no enduring validation. The data is just wrong, and nothing in the normal read path would ever know.A data checksum closes that gap. When checksums are enabled, every page written to disk carries a 16-bit checksum in the header computed from its contents. When that page is later read back into shared buffers, Postgres recomputes the checksum and compares. If the stored value and the computed value disagree, the page has changed underneath the database without the database's knowledge, and Postgres raises an error instead of returning garbage:That error is the difference between discovering corruption upon interacting with an affected page, and discovering it months later when the rot has propagated into replicas and backups. The full details live in the Data Checksums chapter of the documentation, and the short version is this: checksums hang a cowbell onto previously silent corruption.But wait, there's more!

Be Kind, Rewind

The second, sneakier motivation for enabling checksums is for the sake of pg_rewind. When running a high-availability Postgres cluster, a switchover or failover event means the old primary node eventually gets repurposed as a replica. The easiest and fastest way to do this is to compare the state of the new and old primary, and "rewind" differences until they're compatible.But has a prerequisite:pg_rewind requires that the target server either has the option enabled in postgresql.conf or data checksums enabled when the cluster was initialized with (the default). must also be set to on, but is enabled by default.So checksums are one of two ways to satisfy . The other is a parameter called wal_log_hints. As a subject, hint bits are basically boring Postgres bookkeeping meant for optimization. The important part is that, as an "optimization" which can change during a read, they are not logged to the WAL by default.But as we all know, checksums are highly dependent on page contents. Any change to its bytes, including a flipped hint bit, also affects the checksum. So when checksums are on, Postgres has no choice but to WAL-log the full page—even for a humble hint bit update—so that recovery and replication stay consistent. Both roads lead to the same destination, which is why either one satisfies .It’s still easier to simply enable if we only want rewind functionality, but checksums get us additional safeguards. With that in mind, how do we actually turn them on? Perhaps that process can explain why more DBAs avoid checksums than we would otherwise expect.

Born at Creation

Checksums have been available since Postgres 9.3, but only as a flag to at the moment of cluster creation. Past versions of Postgres worked like this:Omit the flag, and the cluster remained checksum-free for life. That initial decision baked checksums into the data directory before we ever wrote a byte. Postgres 18 changed the default for this, but that only helps brand new clusters. Every cluster created before that, and every one created with the old default, was on its own.A partial reprieve arrived in stages. Postgres 11 shipped , which could at least scan an offline cluster and determine whether existing checksums were intact. Then Postgres 12 renamed the utility to pg_checksums and taught it the two verbs everyone actually wanted: and . Finally, a way to add checksums to a cluster after it was born:Unfortunately, there was one major catch. Read the very first line of the requirements:The server must be shut down cleanly before running .The tool requires the cluster to be completely offline. Since the writable primary is the source of all truth, that means replicas would need to be rebuilt once the process completed as well. Then rewrites every single page of every single relation with its freshly computed checksum. On a modest database that's a mere inconvenience. On a multi-terabyte production system, that could result in hours or even days of downtime.And that assumes everything went as expected. If not, there's no partial credit here. An interrupted conversion process cannot resume. The cluster's checksum state simply remains unchanged, and we get to start over from the beginning. DBAs using this utility were probably praying to the Database Gods™ for salvation. Is it any wonder that so many existing Postgres clusters still lack checksums?The only real workaround is to create a new cluster with checksums enabled, and then use logical subscriptions to move the data over. This procedure is a kind of Swiss Army knife which can also ease major version upgrades, collation changes, and other major conversions where it's simply easier to abandon the old cluster rather than modify it. But that's not exactly a simple proposition either.Thankfully, there's now an alternative.

Behind the Scenes

Let's finally cut to the chase: Postgres 19 makes it possible to toggle checksums on a running cluster. Better yet, it's just a single function call:That single call kicks off the whole process in the background. Under the hood, Postgres spawns a worker for each database in the cluster. Each worker walks through every page, marking buffers dirty so that a valid checksum gets written on the next disk flush. The work is WAL-logged and replicated, so standbys are also implicitly converted. Just start the job and get on with your day.Processing every page on a busy system is still a lot of I/O, so the function accepts the same kind of throttling knobs that cost-based vacuum uses:The and arguments tell the background workers to ease off, trading a longer total runtime for less impact on production traffic. If the cluster is quiet, leave them out and it churns through every page as fast as the storage subsystem will allow. It's nice to have options.But how do we know when it's actually done?

Watching It Happen

This is where a small but incredibly convenient change shows up. The read-only data_checksums parameter has been around forever, but it used to be a plain boolean: checksums were either or . Postgres 19 changes that to an enum, exposing the other new states:The four possible values are , , , and . When being enabled, the cluster moves from to while the workers do their thing. It only flips to a confident once every page across every database carries a valid checksum.Disabling runs the same play in reverse, and it's even quicker:That moves the cluster from on to , and once every active backend has stopped trying to validate pages, it settles into . Turning checksums off doesn't require rewriting anything, so it's nearly instant; Postgres just stops checking. But really, why would we ever do that?

Final Thoughts

Looking back, it's kind of hard to believe how long this particular wart persisted. Checksums shipped with 9.3 back in September of 2013! The (offline) conversion tool didn't arrive until version 12, six years later. Postgres 18 finally switched to the safer default. And now version 19 makes it possible to change on a running system. The price of admission meant a lot of DBAs simply avoided converting existing clusters, and went without valuable protection due to a questionable default.Postgres 19 still doesn't make the cost of entry "free", and I wouldn't trust anyone who claimed it did. You still pay in WAL and background I/O during conversion, and the throttle settings aren't just there for decoration. That said, "pay in tunable background I/O" and "pay in a multi-hour outage" are not even in the same universe of impact. One is a mild annoyance, while the other probably relegated countless clusters to remain checksum-free.So if you've got an older cluster humming away without checksums, worry not! This is the release that removes the final obstruction to embracing full data integrity on such systems. If you have one of these clusters, try the latest Postgres 19 beta on a restored backup in a safe sandbox environment. Check how long the conversion process takes, try it with different IO tuning knobs, and develop a playbook for your production instance. Postgres 19 is coming very soon now, and you'll be ready when it does.

Looking Forward to Postgres 19: Split Personality

Fri, 26 Jun 2026 07:17:47 GMT

Postgres has had native support for declarative partitions since version 10, and every release since has filed off another rough edge. We got partition-wise joins, default partitions, hash partitioning, and the ability to attach and detach partitions concurrently. By any reasonable measure, declarative partitioning is one of the great success stories of modern Postgres.Despite the power here, it's always been a kind of one-way ratchet. Creating or dropping partitions was easy. But reorganizing existing ones was a different beast entirely. There's no syntax or tool to assist in the event we want to convert a table to a partition set, or revise an existing design. Instead, it's just a long series of manual statements to carve the table up into partitions, or perhaps leveraging pg_partman to do it instead.Well, Postgres 19 finally has an answer to that. Let's talk about the current state of the art, and how the new and syntax for ALTER TABLE may change the story.

The Way Things Were

Suppose we have a busy partition and want to break it into smaller pieces. The current procedure works something like this:

Create new partitions as standalone tables.

Copy the relevant rows into each one.

Detach the old partition.

Attach the new ones.

Drop the empty table husk.

It's serviceable, but also tedious, error-prone, and subject to various locks that can delay the process. Postgres 19 collapses that whole routine into one statement.Before diving into how, let's create something realistic to play with. How about an analytics pipeline containing a stream of events partitioned by time? The initial state will be one partition per quarter for 2026:Now let's inject a single event into every day of the year so each partition has something to hold:Let's assume that a quarter felt reasonable when we drew it up, but traffic in Q1 turned out to be heavier than expected. Now, querying three months of data to find a single morning's worth of clicks is wasteful. We'd much rather have monthly partitions there. In the past, that meant the migration shuffle.

Be Fruitful and Multiply

The new syntax reads almost exactly like what we might say out loud. "Take this partition, and split it into these smaller ones":That's it. The original partition is gone, replaced by three monthly children, and Postgres moved every row into the partition where it now belongs. Let's verify that:No temporary table -> copy -> detach dance. Instead, the rows land exactly where the new bounds dictate. Seventeen days of January data, a full February, and a full March. The whole operation is transactional too, so it's perfectly valid to abort in the middle and leave the original structure unchanged.One detail worth noting: the new partitions must cover the entire range of the partition being split, with no gaps or overlaps. Postgres will point out mistakes here:That's a good error; it tells us precisely what's wrong and exactly how to fix it.

E Pluribus Unum

Splitting is only half the story. The reciprocal scenario is just as valid. We could just as easily have a pile of old partitions that nobody queries individually anymore and want to de-clutter. When 2026 is well behind us, three monthly partitions for Q1 become pure overhead.Let's fold them back into a single quarter:The three monthly partitions collapse into one, and all 76 rows come along for the ride:Notice that we reused the name for the result. That's allowed, because the partitions feeding the merge are consumed in the process. The same trick works with , which is genuinely handy when carving up a default partition, as we'll see in a moment.Merging carries the same adjacency rule as splitting. The partitions being combined must sit next to each other on the number line with no gaps between them. Try to merge two partitions with a third wedged in the middle, and Postgres refuses:

On Distributing Defaults

Here's where these commands earn their keep in the real world. Imagine a table where data sometimes arrives for time ranges we have yet to allocate. That's what the default partition is for: a catch-all that scoops up anything unclaimed by existing partitions.Those two May and June rows had nowhere else to go, so they fell into : In the past, promoting that stray data into a proper Q2 partition was something of a pain. We couldn't simply attach a new partition that overlapped the default while it held matching rows. Consider:Now we don't have to worry about that:The May and June rows migrated into the brand new partition, and the default partition remains no worse for the wear. That alone is worth the price of admission.

Lost in Translation

But like any Ikea furniture, sometimes we lose a few pieces during assembly. Postgres drops any objects defined directly on the partitions during a split or merge procedure. Consider this scenario where we have a partition-local index on January and a constraint on February.This is what happens during a merge:Both the constraint and index are gone without a trace. Thankfully there's an important distinction between objects defined on the parent and objects defined on an individual partition.An index created on the parent table itself is a partitioned index, and Postgres maintains a copy on every partition automatically. When a merge or split produces a new partition, that partition gets its copy of the parent index just like any freshly attached partition would. Watch what happens when we add a parent index alongside a partition-local one, then merge:The merged partition comes back carrying the parent's indexes and nothing else:The parent index survives, recreated automatically under a generated name. The local does not. So we could look at it like this: anything inherited from the parent is rebuilt, and anything attached to a single partition is forfeit. If partitions carry their own bespoke indexes, constraints, or triggers, take note of these if any need to be recreated following a merge.

An Exclusive Club

The other major caveat is about contention. There is no variant of these commands yet. Both and take an lock, and they hold it for the entire duration of the operation.How heavy is that lock, exactly? Let's see:Look at the top row. The lock isn't just on the partition being split. It's on the parent table itself. For as long as the operation runs, the entire partitioned table is frozen. All reads and writes will wait until the completes. That's more than a little inconvenient.That's only a few milliseconds for a small partition, and nobody notices. Partitions containing hundreds of millions of rows could be locked for minutes or even hours during a split or merge. So treat these commands like any other statement: dangerous at peak traffic. Until a concurrent variant arrives, the heavy lock is the price of admission.

Hash Need Not Apply

Native table partitions in Postgres have come a long way, but the merge and split functionality are still early in their evolution. As a result, some omissions remain, such as hash-based partitions.Splitting and merging both work on and partitioned tables, but a hash-partitioned table will balk immediately:Perhaps this might come in the future, or perhaps not. How does one "merge" a hash, exactly? We'd have to change the base modulus, thereby affecting all child partitions rather than a distinct subset. Still, it pays to be informed.

Final Thoughts

The journey of 1000 miles begins with a single step. Declarative Postgres partitioning took that step way back in version 10, and now it continues the pilgrimage by adopting and syntax. Due to their relative youth, they're missing some creature comforts such as concurrent operation. That's no small omission, especially given the access exclusive lock they incur.But the foundation is there. If the history of Postgres partitioning tells us anything, the rough edges will be smoothed off release by release. A concurrent variant feels almost inevitable. Until then, maintenance windows are the only option when splitting or merging partitions. If availability is of utmost importance, the old table-swap polka is still an option.So pull down a Postgres 19 build and experiment a bit with the new and functionality. It's something a lot of users have requested, and now it's finally here!

The Long Road to Bottomless Postgres

Wed, 24 Jun 2026 10:05:14 GMT

Every database eventually runs into the same wall: storage costs money, and the data nobody queries anymore costs exactly as much as the data everyone does. A five-year-old row occupies the same expensive block storage as the order that came in thirty seconds ago. Postgres doesn't know the difference, and why would it? That's honestly a common refrain among most database engines.As a result, many have dreamed of fixing this by decoupling compute from storage. Push the cold, ancient data down to cheap object storage like S3, keep the hot data local and fast, and let a single query span both. Better yet, store that cold data in an open columnar format for optimal analytics. It's a vision that launched an entire cottage industry of extensions, forks, and re-architectures, each one tackling the problem in its own particular way. But there are so many approaches now, and none have "the" definitive solution. Each contender seems to demand a different sacrifice, whether that's a forked binary, an extra daemon, a duplicate copy of table data, or eschewing writes entirely. Let's explore the external storage ecosystem and see what we find.

Perchance to Dream

So what does "bottomless" actually mean, and how do we get there?Postgres declarative partitioning offers the first potential tool in our arsenal. The ability to split data into definitive chunks means the planner can leverage pruning to focus on the most relevant data based on our predicates. From there, we can consider purposefully tiering data based on storage cost and demands. It's not a huge stretch of the imagination to use tablespaces to further federate "hot" and "cold" storage this way. No longer would our critical transaction-sensitive data cost the same as accumulated metrics useful only to analytics.It's possible to take that idea even further. In the most extreme cases, innovative users utilize Postgres Foreign Data Wrappers to present entirely remote content as local data. Some wrappers are better than others, but this method of decoupling further taps into the potential of "limitless" storage. Indeed, Postgres doesn't even really care where that data lives at all, only that the FDW reacts as instructed. Postgres Table Access Methods can even obfuscate the foreign nature of the content entirely.And perhaps finally, there's the storage format itself. The Postgres data heap is great, but it's understandably tied to the engine, with all that implies. Storage isn't columnar for better analytic access, it's encumbered with reasonable atomic locking that actively resists multiple write sources, and has very real limitations. Apache Parquet or Iceberg on the other hand, have quietly become the lingua franca for data lakes of all description. Stored externally in those formats, suddenly multiple engines can interact with the "cold" data along with Postgres, in a format better suited for analytics.In that dream, Postgres is a familiar, operational query shell over literally any amount of data in any location. If that's not "bottomless", I don't know what is. And yet, every project that has chased that dream has missed the mark in one way or another. How? Well, let's take a look at some of the existing projects and see for ourselves.

Neon to The Extreme

The most thorough way to decouple storage from compute is to supplant the storage engine with an entirely new one. Neon decided that was the most direct way to attain "the dream"; just put all data in S3 where "bottomless" is in the product description.Neon replaces the Postgres storage manager so that pages no longer come from local disk at all. Compute nodes become stateless Postgres processes that fetch pages on demand from a separate service called the pageserver, while a redundant set of safekeepers durably store the write-ahead log before it's processed and uploaded to cloud storage. In a way, the WAL effectively becomes the database. There's no local heap to fsync, and the log sequence number doubles as a temporal key, so the pageserver can reconstruct any page at any point in time by replaying WAL forward from a base image.It's an elegant design that's entirely transparent to SQL clients. Applications have no idea that pages are being served over the network from layered files in S3. But that flexibility comes with a couple catches.The first is that this requires a patched Postgres. The storage substitution is woven through the codebase. Neon must start a node without reading a checkpoint record, track the last-written page LSN, and include copious prefetching to address S3 access latency. It's not possible to drop Neon onto a stock community build; it's barely even possible to run it outside of a cloud context. It's more of an ecosystem than the Postgres most users would recognize.The second catch is latency. Neon's own documentation is refreshingly candid that it "doesn't benefit from OS readahead, and the latency to pageservers is quite high compared to local disk." A cache miss is a network round trip, and a cold miss adds an S3 fetch plus page reconstruction on top. Aggressive prefetching / rehydration softens the blow, but the trade-off is inescapable. Neon is a remarkable piece of engineering and a fully managed platform. It's also the nuclear option: an entire reimplementation of how Postgres stores data. Perhaps there's a... less invasive way of reaching the same goal?

DuckDB Inside

We already mentioned Parquet and Iceberg storage formats, and they offer a convenient capability. Provided there's an access method, Postgres can treat it like any other externally available data. But something has to read those columnar files on S3 on its behalf. These days, that "something" is usually DuckDB.DuckDB is an analytical engine with first-class Parquet support and extensions for Iceberg and S3-style object storage. It's fast, columnar, and designed to be dropped into another process. For a Postgres extension author who needs to scan a pile of Parquet files on S3, DuckDB is the obvious shortcut. There's a reason it quietly powers essentially every project in this space.Practically every "data lake" Postgres extension embeds or leverages DuckDB in some manner, allowing a Postgres client to reach cold data on S3 through the power of SQL. DuckDB is the secret sauce for an all-you-can-eat buffet of storage if used correctly. It turns out, "using it correctly" is highly subjective, and has become its own handful of distinct approaches.

Magic Mirror on the Wall

Sometimes the easiest approach is to copy existing data and call it a day. The goal of the pg_mooncake extension is real-time analytics rather than cheap archival.Instead of moving data elsewhere, pg_mooncake mirrors it. The extension maintains a synchronized copy of a table in Iceberg alongside it. The approach is non-invasive and leverages the strengths of columnar storage. A Rust ingestion engine called moonlink streams changes through logical decoding, so the mirror stays current with sub-second resolution. Then DuckDB provides the analytical access to the mirrored data.It's trivial to create a mirror with a single procedure call:Notice the naming in that last query. We're selecting from , not . The mirror gets its own name, so the transparency stops at the table boundary; applications must know which copy they want. There are also two copies of the data and the additional continuous CDC to maintain alignment. It's an acceptable trade for fast analytics, but the cold storage isn't being offloaded in this case.Luckily, pg_mooncake is actively developed and has a healthy following. The architecture has even changed underneath it, moving from a Delta Lake columnstore in its earliest releases to the Iceberg and moonlink design on display today. It's a lakehouse for Postgres, and a good one. But it raises a couple of important questions. If those rows are cold enough to mirror, why keep paying full price to keep the originals warm? Why further compound it with the overhead necessary to continuously synchronize the data?

Moving Day

Extensions like pg_mooncake provide an important lesson: it's almost easy to access Parquet and Iceberg through Postgres. Taken a bit further, we can cut out the middle-man and consider the external data a true archive. If the data is important enough to remain local, it's a separate category of content. Data tiering is nothing new, but using an extension to manage that capability is.The pg_tier extension sought to provide that interface. Rather than mirror, it prioritized decoupling "old" data from the database entirely. The model is straightforward.1. Configure an S3 bucket.2. Point at a table.3. It copies the data to the bucket as a Parquet file.4. Then it swaps the local table for a foreign table.The tiering itself runs through a function call:After that, SELECT * FROM people still works, because the name now resolves to the foreign table. The most common way to do this is with partitions, allowing the root table to access local and remote partitions transparently. Tiering is a one-shot function call rather than an automatic lifecycle. The "real" table becomes an FDW stub, and the whole thing leans on the parquet_s3_fdw FDW from PGSpider. (Not DuckDB?! Gasp!)Unfortunately, pg_tier is effectively abandoned. Tembo, the original maintainers of the extension, pivoted to AI and eventually deleted the repository. What survives are legacy forks, unmaintained since mid-2024, carrying a beta warning and no published release. It was a good idea that lost its sponsor, and it's a useful reminder that an extension is only as durable as the people willing to maintain it.Despite that, it offers a valuable window into what's possible. And as with such value, another contender becomes inevitable.

Ready Player Two

While pg_tier provided an interesting prototype, the pg_lake "suite" (a bit more than an extension) was our first glimpse into a full-scale implementation with serious engineering behind it. It began its life as Crunchy Data Warehouse before arriving as open source after the company's acquisition by Snowflake.The stack of tools enables querying Iceberg and Parquet tables directly from Postgres, with the heavy lifting delegated to DuckDB for vectorized execution. The syntax is clean and feels native thanks to use of Postgres Table Access Methods:The architectural decision that defines pg_lake is how it runs DuckDB. Instead of embedding it in the Postgres backend, pg_lake relies upon a separate daemon called pgduck_server. It's a standalone executable that speaks the Postgres wire protocol over a local socket and runs DuckDB underneath. The project suggests it made this decision because embedding DuckDB directly would collide with Postgres's process-per-connection model, since DuckDB is multi-threaded and the Postgres backend is not. They believe that a separate daemon neatly sidesteps the threading and memory-safety headaches.While that decision buys performance and stability, there's an operational cost. A pg_lake deployment consists of two components to run, configure, and maintain: Postgres and pgduck_server. For some teams that's a non-issue. For others, an extra daemon is exactly the kind of moving part they adopted Postgres to avoid.

A Cooler Forecast

Which brings us to ColdFront, pgEdge's entry, and the reason for this whole survey. ColdFront tries to occupy the space the others leave open, and it makes a different set of trade-offs at nearly every decision point we've covered.Like pg_mooncake and pg_lake, it uses DuckDB for the heavy lifting. Unlike pg_lake, it embeds it in-process through , so there's no separate daemon or sidecar process. Like pg_tier, it moves cold data out to object storage rather than mirroring it, so there's only one copy and Iceberg becomes the system of record for archived rows. Unlike pg_tier, it doesn't leave us with a foreign-table stub or a one-shot manual move. Instead, a small Go archiver runs on a schedule and ages partitions from hot to cold automatically.The part that ties it together is the query path. ColdFront installs a in C that rewrites SQL at parse time. The table name is actually a view spanning a of the hot Postgres partitions and the cold Iceberg data. The hook inspects the clause against an archive watermark to decide which tier to include. Hot-only queries never go near S3, cold-only queries use DuckDB's Iceberg pruning, and a query straddling the boundary transparently hits both.Setup looks like ordinary Postgres:From there, we can query like any other table, and old partitions quietly migrate to Iceberg behind the scenes when the archiver runs. What's genuinely different here is that the cold tier stays writable through the same name. An or against archived rows just works, no rehydration and no special path. Postgres really does treat it just like any other table.ColdFront can also run in a fully decoupled mode where a table lives in Iceberg from its very first row, making Postgres a stateless compute front-end, or in a partition-only mode that does lifecycle management with no Iceberg at all. And because pgEdge specializes in distributed Postgres, there's a mutual-exclusion scheme modeled on Lamport's bakery algorithm. This allows multiple nodes in a cluster to write the same Iceberg table without stepping on each other. That makes it possible for nodes in an active-active cluster to share the same archival target (and write to it) without locking contention.For all that, ColdFront is still a brand new extension, complete with a "BETA PRE-RELEASE, NOT FOR PRODUCTION" warning in the repository. Both operating modes work end to end and the test matrix is green, but there are still a few usage caveats. While still not truly "battle tested", it also threads the needle to cover scenarios the other approaches don't.

Finding The Sweet Spot

What do we see in this short external data survey?Neon paid the steepest price for the most comprehensive result. Rewriting the storage manager buys perfectly transparent, effectively infinite storage. It costs a forked binary plus a network round trip on every page that misses the cache. For anyone who wants a managed platform and never wants to think about disks again, that's an acceptable trade. For anyone who flatly refuses to run a non-standard Postgres, it's a non-negotiable deal-breaker.The remaining solutions opted to remain as extensions on stock Postgres, but used radically different approaches. pg_mooncake barely moves the needle at all, keeping a live columnar mirror in sync for the sake of analytics. It proves the external data concept, but has limited utility otherwise. The remaining extensions go further and rely on external storage, keeping a single copy and letting object storage become the true home for Parquet or Iceberg data. Whether that access is in-process or an external daemon is the decision between pg_lake and ColdFront.So where does that leave someone who actually has to choose?I'll be honest that my own preference here isn't exactly neutral. What wins me over is the one thing none of the others quite manage together: coherence. ColdFront tries to be as transparent as possible, passing everything through a single table name. There's only ever one copy of the data, and there's no sophisticated cache refresh layer or annoying daemon to set up. The extension handles read and write routing, and that's the end of it.I'll admit the Go archival daemon struck me as an odd choice given Postgres background workers are available, but maybe that'll change in the future. Still, everything else about the extension is just easier to use and makes interacting with tiered storage a no-brainer.

Final Thoughts

The dream of bottomless Postgres is quietly converging. Open table formats won; nearly everyone now stores cold data as Parquet and Iceberg, which means archived rows stay portable no matter which extension wrote them. DuckDB won the engine war by default, sitting under almost every contender whether embedded or sidecar. The remaining arguments are about packaging and trade-offs, not fundamentals, and that's a healthy sign for an ecosystem.ColdFront is the newest answer with the simplest architecture. Whether the straightforward design survives contact with production is the question only time and a few brave early adopters can answer. It's beta software, and it says so loudly, so treat it as a glimpse of where this is heading for now. We at pgEdge hope to change that in the near future, but we want to stay honest.A default Postgres install still has no concept of data tiering aside from tablespaces, and that barely counts. But there are now several credible approaches to allowing Postgres to play to its primary strength: query execution. It's now possible to leverage Postgres as a compute layer for result aggregation regardless of data location. Foreign data wrappers are one way to do that, but now that Parquet and Iceberg are becoming the data lake standard, perhaps a more tightly-coupled design makes more sense.Either way, it's an exciting time in the Postgres ecosystem. Extensions like ColdFront finally allow Postgres to operate based on data temperature, even if that's a concept I made up just now. It's a common enough scenario that I think ColdFront will make a genuine difference. So please, put it through its paces, and let us know how it goes.

Looking Forward to Postgres 19: Logically Sequenced

Fri, 19 Jun 2026 11:01:55 GMT

Logical replication has been an integral part of Postgres since version 10 released in 2017. It's a very convenient system for synchronizing one or more tables from one running Postgres cluster to another, and the community has embraced it almost without reservation. It's a great feature we've all come to rely on.For all that, it has never been a flawless panacea. Perhaps the most glaring and conspicuous omission in Postgres logical replication is that of sequences. Novice users might run a logical migration or promote the new cluster, and meet a wall of duplicate key violations on the very first insert. The more experienced mind the gap and bolt on supplementary steps to circumvent this, quietly hoping they remember to run it during the cutover window.But why? Why were sequences left out of logical replication for the better part of a decade? Why let a freshly promoted cluster restart its counters at 1 and collide with the very rows it just spent hours copying over? That turns out to be a surprisingly long story, and one Postgres 19 finally gives a satisfying ending. So let's get into how sequences became the awkward holdout of logical replication, and what changed to bring them along at last.

The Sequence That Time Forgot

There’s a sequence hiding behind every or column ever created in a Postgres cluster. That sequence provides the next value to the related column when no default is supplied. It’s easy to take for granted because it’s always been there and everyone knows how they work.Any time a write happens in Postgres, the new row contents get written to the WAL. Logical replication works by essentially decoding the raw WAL tuple into an equivalent or statement and replaying it on the subscriber. When a row with arrives on the subscriber, that's the value that gets written. Meanwhile, the subscriber's own sequence object (probably copied from the publisher at some point) remains unused. That’s great until the subscriber has to generate a value of its own.Consider the migration including a table that is replicated all the way up to nine million rows. Once we redirect the application to the subscriber node as the new write target, it may be customary to expect the sequence starts at nine million and one. But that's not what happens; the sequence on the subscriber exists as it was initialized, and probably sits at the original value of 1. And that's the value the identity or serial will return, resulting in a cascade of "duplicate key value violates unique constraint" errors.Not exactly ideal, especially in production.The official restrictions page has spelled this out plainly for years: sequence data is not replicated. The standard remedy was to fix the sequences manually right before redirecting application and user traffic. Executing something like this on the publisher generates a script of setval() calls. Running the script on the subscriber essentially solves the problem:Notice the . That padding isn't decorative. Even with a highly ambitious activity drain on the old cluster, there's probably a short window where some pending writes could bump the sequence on the publisher. So we inject a 1000 value "buffer" to absorb any of those errant writes until we promote the subscriber to be the new permanent write target. But it's time sensitive: forget to run it, or run it at the wrong moment, and we're back to duplicate keys.There had to be a better way. For a long time, there was exactly one.

pglogical to the Rescue?

Before native logical replication existed at all, the pglogical extension from 2ndQuadrant was how serious users replicated Postgres data with a publish/subscribe model. Unlike the in-core feature that eventually supplanted it, pglogical learned to handle sequences from the beginning.The mechanism is a set of functions called on the provider to enroll sequences into a replication set. It's possible to add them one at a time:Or include everything in a schema at once:When enabled, the flag pushes the current value across immediately. There's also a function which can force a re-sync on every subscriber, handy after a big data load or partway through an online upgrade. But not even pglogical ever replicated sequences in real-time. Here's what the documentation has to say:That's basically our + 1000 trick. Pglogical maintains our handy safety buffer automatically with the help of a periodic sequence bump through a background worker process. There’s a reason it works this way, even when handled by an extension.

How we Got Here

So why couldn't native logical replication just decode sequence changes the way it decodes inserts and updates? Why didn’t pglogical? The reality is that sequences break one of the foundations logical decoding is built on.Sequences are non-transactional. Every generated value, even when rolled back, is permanently consumed. One superpower of sequences is that they're essentially asynchronous, but this comes at the cost of gaps. This allows multiple simultaneous transactions to obtain sequence values without waiting, but it also means any transaction can abandon whatever value it received.By contrast, logical decoding lives and breathes transactions. It reorders the WAL into neat transactional bundles and replays them in commit order. Sequences don't fit this model at all, making it essentially impossible to add them to a standard logical replication stream.This wasn't for lack of trying. Tomas Vondra spent years on a patch for logical decoding of sequences that would include sequence advances as part of the decode stream. It actually got committed during the Postgres 16 development cycle. And then it was reverted before the final release.Why rip out a feature that was already committed? The trouble was reconciling the transactional and non-transactional halves of sequence behavior, and it wasn't clear the design could be salvaged without substantial effort. An alternative idea to log sequence state at commit time was floated and then abandoned too, weighed down by overhead, tricky ordering between concurrent transactions, and reliability concerns.That's a sobering amount of effort to end up back at square one, but that’s what happened. Sequences became a permanent fly in the logical replication ointment, and everyone just kind of accepted it.

A Sync By Any Other Name

So how do you replicate a counter that isn't possible to cleanly decode? That's the really neat part: you don't! The breakthrough in Postgres 19, led by Vignesh C with Tomas Vondra, was to stop fighting the non-transactional nature of sequences altogether. Instead of streaming every increment through the decode pipeline, the new feature synchronizes sequence values at well-defined moments. Hopefully this is already starting to sound familiar. We never really needed the subscriber's sequences to track the publisher increment for increment. That gives us incredibly clean invocation points. The release notes lay out the three moments when this synchronization kicks in:Creating a subscription now pulls sequence values across alongside the initial table data. Then reconciles which sequences exist on the subscriber to match the publisher, updating their values while it's at it. The third command just does what we want outright, directly updating sequence values on the subscriber.This is not continuous or even periodic replication. Once a sequence value is synchronized, it goes stale the instant the publisher hands out another value. Applications that keep writing to the old primary means the subscriber drifts behind again until the next refresh. But that also dramatically simplifies matters: sync the sequences as the last step before promoting the subscriber.Let's see what that might look like.

Sequence Sync in Action

On the publisher side, Postgres 19 adds an clause to CREATE PUBLICATION, mirroring the long-standing : That single statement now publishes both the table data and every sequence in the database, including sequences created later. It's also possible to create publications purely for sequences. But take note: the clause is . There's no syntax for cherry-picking individual sequences into a publication the way you can with tables. This is an all or nothing proposition.On the subscriber, creating the subscription does the initial work for us:At this point the tables begin their initial copy and the sequence values are pulled across as part of the same process. Nothing should be surprising here. The interesting part comes at the time of the cutover. The initial sync may have happened hours or days ago, and sequences are probably vastly different from their initial state after the sync.Right before it's time to promote, give sequences one final bump with the new ALTER SUBSCRIPTION command:That re-reads every sequence value from the publisher and stamps it onto the subscriber without any further fanfare. If there are any new sequences on the publisher since the subscription was created, it's probably a good idea to use instead, since that reconciles existence as well as values.Compare that to where we started. The entire ritual from the top of this article collapses into one command that the database understands natively.

Peeking Under The Hood

Naturally, it's probably a good idea to validate the values that actually made it across the divide. Postgres 19 provides the new pg_get_sequence_data() function to inspect the synchronized state of a sequence directly:We can use that to verify all sequences on both the publisher and subscriber prior to committing to the transition. Errors during sequence synchronization get first-class tracking too. The pg_stat_subscription_stats view gains a column to spot sequence problems. This necessitated renaming the old to to distinguish the two states. Monitoring dashboards using this view may also need to be updated, so be careful.It's always a good idea to get a picture of the subscription before taking any drastic actions:If everything in this view looks good, feel free to continue.

Finally in Sequence

It's a little ironic that the solution which finally shipped is the same one pglogical leveraged since its inception, and the same one we've all been faking by hand with scripts. Synchronize the value with a minor safety buffer when requested—how anti-climactic. Then again, it's hard to beat the KISS principle.This removes one of the last manual gotchas from logical replication for anyone running cluster migrations or major-version upgrades. Given those are two of the biggest use-cases for logical replication, that's a major advancement. More real-time use-cases probably fall under Multi-Master territory, and are better served by UUIDs or Snowflake sequences anyway.In any case, Postgres 19 is only a few months away. We encourage you to experiment with the betas and release candidates when you get a chance. Try the new publication syntax. Fiddle with and see what happens. You've been waiting for the opportunity for years, after all!

Looking Forward to Postgres 19: It's About Time

Fri, 12 Jun 2026 04:31:56 GMT

Recently, a new type of question has entered the database arena: what did this data look like last Tuesday? Maybe it's the price of a product before the holiday sale kicked in, or which department an employee belonged to before that reorg nobody asked for. Short of adding an entire audit trigger system, how can we know what data looked like before and after a change at that exact date?The SQL:2011 standard formalized a proper solution over a decade ago with temporal tables. Other database engines adopted pieces of it relatively quickly. Characteristically, Postgres took its time. But Postgres 19 is finally bringing native temporal table support to the party — and it's been well worth the wait.Let’s see what we’re working with.

The Old-Fashioned Way

Before we get to the shiny new stuff, let’s check out the crusty old approach for some perspective. Suppose we want to track product pricing over time. A reasonable first attempt might look like this:Simple enough. We've got a product, a price, and a valid date range for the price. Unfortunately, nothing stops us from inserting two rows for the same product with overlapping date ranges. Product number 42 could be $9.99 and $14.99 on the same Tuesday. Your accountant might have some choice words upon discovering that.The traditional Postgres answer here is the btree_gist extension and an exclusion constraint:This works. If we try to insert a conflicting row, Postgres catches it:A problem solved by using ! So what’s the problem? Well, a few things:

Everyone knows BTREE and indexing in general, but
is specific to Postgres and thus requires experience to understand. This goes double for being an
extension.

The exclusion constraint syntax is quite non-intuitive. It’s in the documentation, but there’s no reason anyone would think of this as the standard approach otherwise.

There’s no temporal awareness baked into the table itself.

Basically, Postgres doesn't understand that this is temporal data. It's just columns and an esoteric constraint using a fancy index type. Every update that changes the time range requires manual splitting and stitching of rows, which means applications must carry the full burden of temporal correctness.It’s the barest of minimums, and frankly we can do better.

A Brief History of Time

The desire for proper temporal support in Postgres isn’t new. The SQL:2011 standard introduced periods, constraints, and syntax for temporal DML. 2011 was a long time ago.Henrietta Dombrovskaya (Hetti to her friends) was one of the earliest champions of temporal data in the Postgres ecosystem. Along with Chad Slaughter, she developed the pg_bitemporal extension. It's a framework for managing bitemporal tables entirely within Postgres using PL/pgSQL. She presented the concepts at multiple conferences since 2015, demonstrating how to track both valid time (when was this fact true in the real world?) and transaction time (when did the database record this fact?) simultaneously.The distinction matters. Valid time says "this price is effective from January through June." Transaction time is the database perspective, saying "this row was inserted at 3:47 PM on March 12th and superseded at 9:01 AM on April 3rd." Combining both produces a bitemporal table that can answer questions like "what did we think the price was last Tuesday, based on what we knew at the time?"The approach leaned heavily on the same mechanism we discussed earlier, but doubled up: one exclusion for the range (valid time) and another for the range (transaction time). A table definition looked something like this:That's two temporal dimensions enforced by a single exclusion constraint. The extension also introduced functions for bitemporal inserts, updates, corrections, inactivations, and deletes, plus implementations of Allen's interval relationships for temporal reasoning. It was a lot of machinery built on top of what Postgres offered at the time.And it worked! But an extension can only go so far. It can't change how the query planner sees temporal predicates, can't integrate with the constraint system at the engine level, and can't provide native DML syntax. For that, the feature needed to land in core.Now Postgres 19 accommodates the application-time half of a bi-temporal system. It's not the full picture, but that's still a huge step in the right direction.

Ranges to the Rescue

Let’s rebuild our products table the Postgres 19 way. Instead of separate and columns, we use a single range type column:That’s it. No extension needed. No exclusion constraint. The clause in the primary key tells Postgres that must be unique at any given point in time, but the same product can have multiple rows as long as their ranges don’t overlap.How does this compare to our old approach? Let’s put them side by side. The old way:The new way:It's cleaner, more expressive, and Postgres actually understands what we're doing. Under the hood, the constraint still uses a GiST index, and it still requires for the non-temporal columns in the key. But Postgres handles that dependency automatically when initializing the constraint. Very convenient.Let’s populate our table with some data to work with throughout the rest of this article:Notice the range notation: means inclusive, means exclusive. So includes January 1st but not July 1st. The final Gadget row has an open-ended range , meaning the current price has no defined end date. And the overlap protection works exactly as we'd expect:Two validation checks for the price of one! That’s what we get for working with ranges rather than two separate unrelated columns.

Slicing and Dicing

Now for the really fun stuff. Let's say Widget's price needs to change to $10.99, but only from March through September of 2025. In the old world, we'd have to manually split the existing row into pieces: delete or update the original, insert the new price range, and insert leftover rows for the portions we didn't change. Get any of that wrong, and we've got gaps or overlaps in our timeline.With temporal tables, we just say what we mean:Let’s see what the rows look like now:Wait... what happened here? We started with three Widget rows and now we have !Well, Postgres sees the tuples at range for $9.99 and range at $12.99 and needs to correct the overlaps to accommodate the new row. Several things happen as a result:

Postgres modifies the existing $9.99 row to only cover the
range.

Then it adds a new row for $10.99 over the remaining
range.

Then it modifies the existing $12.99 row to only cover the
range.

Finally, it adds another new row for $10.99 over the remaining
range.

Why two rows at $10.99 instead of one over the merged range? Because operates on each matching row independently. It doesn’t coalesce adjacent ranges afterward.The end result is gap-free and overlap-free, and that's something we didn't have using pure exclusion logic.That’s a lot of power packed into a single statement.What about the edge cases? If the range falls entirely within a single existing row, Postgres creates up to two leftovers (one before, one after). If it aligns perfectly with existing boundaries, no leftovers are needed. It just works.Interestingly, the newly introduced temporal leftovers don't require privileges. They're preserving existing data, not adding new information. But they do fire existing triggers. That's something to keep in mind for audit logging or trigger functions.

Erasing History

The clause also works with statements. Let’s say Gadget was temporarily pulled from the catalog between June and October of 2025:Let’s check the damage:The delete carved out the June-to-October window. The $22.99 row that originally covered was split into two leftovers: one ending in June and one starting in October. The pricing data before and after the gap is preserved with its original values. It's kind of hard to wrap my mind around a causing the row count to increase, but here we are.Either way, the mechanisms underlying temporal table management mean it’s all handled automatically. There’s no more risk of accidentally deleting too much or leaving orphaned fragments behind at the application layer.

Truth in Advertising

Temporal tables wouldn’t be complete without temporal foreign keys. Postgres 19 supports these using the keyword:The keyword tells Postgres that the foreign key itself is temporal. As a result, the referenced product must exist for the entire duration of the variant’s range. It's not enough for a matching product row to exist somewhere in time. The combination of all matching rows in the referenced table must fully cover the referencing row's period.If we try to create a variant that extends beyond the product’s known timeline:The Widget only has pricing defined through mid-2026, so a variant claiming validity through 2027 is rejected. Postgres checked the full temporal coverage, verifying that matching rows in the parent table span the variant's entire validity period.There’s one significant limitation here: temporal foreign keys only support for referential actions. That leaves out , , and . That means deleting a product row that a variant depends on always results in an error. This is understandable given the complexity of cascading temporal operations, but it does mean applications need to handle these cases explicitly. For now.

Baby Steps

So we’ve got application-time temporal tables with overlap prevention, temporal DML, and temporal foreign keys. What else is left?The big omission is system time, sometimes called transaction time. We already mentioned that application time tracks when facts are true in the real world, and system time tracks when the database knows about those facts. Systems that rely on temporal tables often make use of both. This is exactly the space that the extension has filled since 2015. It's possible to emulate system time with triggers, but it's not the same as having the engine manage it transparently like it does with the other new temporal features.The temporal table documentation acknowledges this directly, noting that system time is not natively supported but can be emulated. Whether it arrives in Postgres 20 or later is anyone’s guess, but the groundwork is already there.

Final Thoughts

Temporal tables have been missing in action for a while now. The approach works, but it’s a crude work-around by comparison. Extensions like proved the concept and kept the conversation alive. Now we're one step closer to the dream of deprecating the need for it.It’s also a much more intuitive approach than using GiST exclusion constraints. in a primary key reads like plain English as proper SQL should. says exactly what it does. The automatic row splitting during temporal updates and deletes eliminates an entire class of potential bugs. Anyone who's ever had to debug a gap in a temporal timeline knows how valuable that is.The road from SQL:2011 to Postgres 19 was a long one. Hetti and others in the community spent years proving these patterns were needed and workable. Now they're in core, and they'll only get better from here. Keep an eye on system-time support in future releases, because all bets are off once Postgres has both halves of the bitemporal equation.For now, go experiment with and on a test instance. There’s no (temporal) time like the present!

Looking Forward to Postgres 19: Query Hints

Fri, 05 Jun 2026 12:02:28 GMT

Well, the world has officially ended. Peter Venkman from Ghostbusters was right all along, and we'll soon be experiencing "human sacrifice, dogs and cats living together, mass hysteria!" Pack it in everybody; we had a great run. The feature freeze of Postgres 19 includes the one feature many claimed would never see the light of day: query hints. I guess "never say never" is pretty good advice.OK, so they're not technically called hints. The Postgres community would never be so pedestrian. Instead, Postgres 19 introduces two new contrib modules: pg_plan_advice and pg_stash_advice. It's "plan advice" you see. Totally different thing.An occasion this monumental deserves a bit more fanfare than simply describing the feature. So let's begin with a walk through one of the longest-running arguments in Postgres history.

A Brief History of "Never"

The Postgres community's position on query hints has been, shall we say, firm. The official wiki page on the subject states it plainly:"We are not interested in implementing hints in the exact ways they are commonly implemented on other databases. Proposals based on "because they've got them" will not be welcomed."Fair enough. The wiki goes on to list six solid reasons hints are problematic:

They create maintenance nightmares.

They break on upgrades.

They discourage root-cause analysis.

They scale poorly.

The optimizer is usually smarter than you think.

They actually
planner improvements because users stop reporting bugs.

Yeah, that sounds about right. And for years, that was the end of the discussion. Postgres doesn't do hints. Go fix your statistics. Next topic, please.But behind the scenes, the debate was never quite so settled. Back in late 2010, a legendary thread erupted on the pgsql-performance mailing list that raged on for several months and captured nearly every available sentiment. It started innocuously enough as a complaint about slow queries, veered through Oracle comparisons, and somehow spiraled into a full-blown existential crisis about whether Postgres needed hints.Robert Haas, threw the first real punch:He went further, arguing that Postgres needed to give DBAs an escape hatch for the edge cases:It's hard to argue with "please don't let people get fired."The legendary Tom Lane chimed in with similar sentiments:In doing so, he effectively left the door open just a crack.Kevin Grittner later observed a rather obvious paradox in the anti-hint argument:Was he wrong? How many of us have toggled to to force an index scan? Or thrown an into a subquery to prevent the planner from flattening it? Or wrapped a materialized CTE around something just to create an optimization fence? These are hints in all but name, with coarse knobs available only to seasoned experts. We were only pretending to eschew hints.Josh Berkus, one of the more vocal anti-hint advocates, drew a clear line in the sand:But even he wasn't against all forms of planner override. His position was against the Oracle model of embedding them in SQL comments. His preferred hierarchy was GUC cost parameters first, then cost parameters on database objects, then new statistical metadata, and query hints only as a desperate last resort. That's actually a reasonable action plan, but Postgres has always lacked that final option.The third-party pg_hint_plan extension eventually filled the gap for many users, borrowing Oracle's comment-based hint syntax. It worked, more or less, and the urgency to add hints to the core quietly dissipated. Or so we thought.

By Any Other Name

So what changed? Well, for one thing, the person who built these modules is none other than Robert Haas. Fifteen years later after that epic thread, he's the author of both and . Say what you will, but the man plays the long game.Haas didn't just slap Oracle-style hints onto Postgres and call it a day. He was deeply embroiled in the controversy from the very beginning, and knew all of the pain points. As a result, he'd been considering how to actually address every objection the community raised over the past two decades.Berkus insisted hints should never live inside query text. So keeps advice completely outside the SQL. Advice is set via a GUC () or stored in a separate stash keyed by query ID. Queries themselves remain unencumbered by extra slop.The community worried that hints would replace the planner's judgment entirely, producing plans that looked "correct" but were secretly catastrophic. So works by constraining the planner's search space rather than replacing it. The documentation is explicit: advice "can only produce plans the core planner considers viable." Advice only nudges the planner toward one it already considered.What about when advice is wrong or outdated? Instead of silently producing garbage or throwing an error, the planner marks affected nodes as and falls back to whatever it considers best within the remaining constraints. Bad advice degrades gracefully.And perhaps best of all, the system generates its own advice. It's not necessary to memorize a syntax reference and hand-craft advice strings from scratch. It is possible to ask the planner what it's doing, and it responds in a format that doubles as syntax:The planner literally hands over the advice string that reproduces its current plan. From there, it's just a matter of salting to taste. And we get all that just by adding to .

Whispering Sweet Nothings

The advice language is surprisingly expressive for something the community resisted for decades. There are several categories, each targeting a different aspect of query planning. control how individual tables are accessed:This tells the planner to use a specific index for the alias. The documentation lists several other handy scan types. Want to exclude a table from the final plan entirely? There's even for that. is where things get interesting. The tag supports a nested syntax with parentheses for strict ordering and braces for flexible grouping:The syntax here is great, offering a convenient distinction between required and optional ordering. That makes it possible to lock down critical parts of the join order while leaving the planner room to optimize the rest. The argument that a DBA may know better than the planner in certain cases is definitely possible with join ordering, and now it's tunable. control which algorithm the planner uses for each join, with a bevy of options. There's a subtle but important detail here: specifying a method implies that the target should appear on the inner side of that join type. So means "build the hash table from d." The documentation helps disambiguate here, and this is all still very new so is subject to further clarification.Want to specify that multiple relations should be on the inner side of a single join? Wrap them in parentheses: rounds things out with , , and for controlling where and whether parallel execution happens.Can we combine all of these? Of course:That's a single string controlling join order, join methods, and scan strategies all at once. And remember, if any part of this advice can't be honored, the planner degrades gracefully rather than catching fire, burning down, and then falling into a swamp.

Stashing it for Later

Using directly is great for ad-hoc tuning and experimentation, but what about production? It's pretty inconvenient to set a GUC before every query. That's where pg_stash_advice comes in.Once we've added it to , it must also be installed:The system works by establishing an "advice stash" as a named collection of query-to-advice mappings stored in shared memory. Create a stash, populate it with advice strings keyed by query ID, and tell sessions to consult it during planning:From this point on, every time the planner encounters this specific query pattern, it automatically applies the stashed advice. And since it's not embedded in the query itself, we can change it at any time. And don't worry about the query ID changing, that's essentially a hash of the query minus any dynamic parameters; the advice should apply universally.The stash can be scoped per-session, per-role, or per-database:Stashes persist to disk by default via , with a configurable write interval. This means they survive restarts, allowing DBAs to inspect their contents, update individual entries, or drop entire stashes when they're no longer needed.This is the part that addresses the production DBA's actual needs. In a scenario where there's a problematic query and it's not possible to change the application code, sometimes the ability to redirect the planner is critical. So create a stash, drop in the right advice, and the problem goes away without touching a single line of SQL. When statistics, the application, or the planner itself improves, just remove the plan advice on the database side. The application or end user never has to know.The advice stash is the last resort, and it's designed to be temporary.

Trust but Verify

One of the more thoughtful design decisions is the feedback mechanism. thoroughly clarifies the result of each piece of advice. Consider this setup using fact and dimension tables from earlier:The output provides impeccable feedback about our advice:We can see a few things from this:

means the targets were found and the advice applied.

means the targets weren't found in the query at all.

means the advice can't be honored (like referencing a nonexistent index).

means the final plan doesn't comply.

means two pieces of advice contradict each other.

Advice honors dollar
quoting syntax, making it more convenient and human-readable.

It's also possible to set to get these as messages during query execution. That's handy for catching stale advice in production logs before it becomes a problem.So what can't it do? For now, it's not possible to control whether aggregates use sorting or hashing. Nor can we supply the strategy for or operations. The planner is also free to reject advice on correctness grounds. There's also a small performance penalty to applying advice since it necessarily interrupts the planner loop even when it doesn't change the resulting plan. The documentation explicitly warns to use this judiciously for a reason.It's a scalpel, so use it like one.

The Long Game

There's something deeply satisfying about this saga. Somehow hints went from an accepted feature we would probably never have, to what looks like one of the better implementations available. The devs also did so in a way that reasonably addresses every one of the major complaints about hints:

Advice lives outside the SQL to avoid embedding them permanently.

Advice guides the planner rather than replacing it.

Advice degrades gracefully with detailed feedback to avoid worst case plans.

The planner generates its own advice strings for self-documentation purposes, enabling easier tweaks.

The community said they'd consider hints if someone could avoid the problems observed in other systems. It took a decade and a half, but someone finally did. Whether it's stubbornness or incredibly high standards, that's just the Postgres Way™.Will this end the debate? Perhaps not. There will always be folks who insist the planner should handle everything autonomously, and they're not wrong most of the time. The right approach is still to fix statistics, tune cost parameters, and leverage features like CREATE STATISTICS to give the planner better information. Plan advice is for when the planner still insists on a bad plan despite doing all of that.For those cases, we'll finally have a first-class, core-supported, well-designed escape hatch. No more toggling like savages. No more jamming into subqueries and praying. No more relying on third-party extensions crudely bolting on questionable syntax from a completely different engine.The Postgres community spent two decades saying they'd never add query hints and ended up adding advice instead. And as we all know, technically correct is the best kind of correct. Now go enjoy your hints, you certainly waited long enough!

Looking Forward to Postgres 19: The New REPACK Command

Fri, 29 May 2026 09:21:00 GMT

Postgres has had a thorn in its paw for a very long time regarding table size. Every modified tuple leaves an old version in the heap for use by older transactions. While locates these old tuples, it only marks them as reusable rather than returning the space to the OS. Tables only ever grow larger in Postgres.Maybe Postgres 19 can fix that for us.

Options Galore

Historically, the only sanctioned solutions to table bloat have been the VACUUM FULL or CLUSTER commands. Sadly, both of these require an lock for the entire duration of the operation. In a world where 1TB tables aren't uncommon, it's a ridiculous imposition that's entirely incompatible with a production system.So the community turned to third-party utilities like pg_repack and pg_squeeze, which rewrite tables with minimal locking by leveraging logical decoding under the hood. Some DBAs even run these on a regular schedule to keep tables at their ideal size at all times. But more conservative shops have always been wary of handing table rewrites to a project that operates outside the core engine's safety guarantees. I've personally handled support tickets to repair corruption caused by . While that may have been several years ago, it's enough to induce extra caution rather than reliance.Postgres 19 may prove to change things entirely. The new REPACK command brings this functionality into the core engine itself, complete with a option that eliminates the prolonged exclusive lock. And as an added bonus, and now use the same repack infrastructure internally, so every path to table compaction benefits from the new code.

An Ounce of Prevention

Before we get too excited about our shiny new tool, let's pull back and talk about not needing it. The best repack is the one you never have to run.Postgres ships with autovacuum enabled by default, and for many workloads, the defaults are perfectly fine. For tables with high churn, especially those that see large batch deletes or frequent updates, the defaults tend to be insufficient.The usual fix isn't to schedule periodic repacking, but to tune autovacuum so it keeps pace with the workload. Consider these configuration parameters:- : The fraction of the table that must be dead before autovacuum kicks in. The default of 0.2 (20%) is fine for small tables, but on a 100-million-row table, that means 20 million dead tuples can accumulate before cleanup starts. Lowering this default is one option for common use cases. Another trick for very large tables is to reduce this setting to 0 and rely on the next parameter instead.- : The minimum number of dead tuples in addition to the scale factor before autovacuum considers the table. So a table with 1000 rows would need 200 + 50 rows to change before autovacuum kicks in using the defaults. With this in mind, a large table that often sees 50,000 updates per batch might set this to 50,000 in conjunction with setting the scale factor to 0.- : How aggressively autovacuum can consume I/O. The default of 200 is conservative. Bumping this up lets autovacuum finish faster, especially on NVMe storage. I tend to recommend starting at 2000.These can all be set per-table with , which is the right approach for very busy or extremely large tables. When autovacuum is well-tuned, tables reach a stable equilibrium where dead tuples are cleaned up almost as fast as they're created, and bloat never accumulates beyond the scale factor. A few percentage points of extra space is worth avoiding additional maintenance steps.So tune autovacuum first. But sometimes things go out of control. Perhaps a stale or prepared transaction, or extreme lag while using hot standby replication led to bloat we need to clean up. What then?

The Miracle Cure

Let's see the new command in action. We'll create a table, fill it with a million rows, delete most of them, and watch what happens:One million rows of fake sensor data. Let's check the size:Now let's simulate a common scenario: a batch job deletes old data, removing about 60% of the rows. We'll run a manual afterward, as any well-planned bulk procedure should after such a major purge:There it is.We deleted 600k rows and the table is still 81 MB. The dead tuples are marked as reusable thanks to , but the file on disk hasn't budged. We're hauling around 138 MB of total relation size for 400k rows that should be much smaller.Let's fix that:The total used size went from 138 MB down to 52 MB, with the table and index size experiencing similar reductions. That space is genuinely returned to the operating system, not just marked for reuse.The option gives us a progress report at the level, and tells Postgres to update the table's statistics once the repack finishes. Both are optional, but there's no reason not to use them.

No Downtime Required

The basic command acquires an lock for its entire duration, just like the old . That's fine for a small table, but a 200 GB production table serving live traffic? Not so much.That's where comes in:When running concurrently, Postgres creates a replication slot behind the scenes to capture any changes that occur while the table is being rewritten. The lock is only held during the brief final step when the old and new files are swapped, similar to how TRUNCATE works. The rest of the operation proceeds while reads and writes continue as normal.There are a few restrictions worth knowing about. can't be used on tables, partitioned tables (though individual partitions are fine), or tables without a primary key or replica identity. It also can't run inside a transaction block. Postgres can also only run five of these concurrent repacks simultaneously by default, unless you increase the parameter.

Getting Sorted

So far we've been repacking in "physical order," which just means the rows are written sequentially into the new file without any particular sort. But can also replace the command by reordering rows according to an index:Why does this matter? When rows are scattered randomly across disk pages, a query that fetches all readings for a single sensor has to pull in pages from all over the table. Before clustering, a query for looks like this:The query touched over 3000 heap blocks to retrieve about 4,000 rows. That's because the rows for sensor 42 were sprinkled across nearly every page in the table. They're fetched in index order, but that's a lot of heap seeks.Here's what it looks like after repacking with : From 3300-ish buffers down to 49. From 3ms down to 0.6ms. The planner even switched from a bitmap scan to a plain index scan because the physical clustering made sequential access so efficient. That's the power of data locality.One important note: such data reordering is still a one-time operation. Subsequent inserts and updates won't maintain the physical order. Workloads that would benefit from index ordering need periodical . Alternatively, a lower table can leave room in the same page for updated values.

A Word of Caution

There's one caveat that deserves explicit attention: is not MVCC-safe.What does that mean in practice? If a concurrent transaction has taken a snapshot before the repack commits, and that transaction hasn't previously accessed the repacked table, it could see the table as empty. This is the same limitation that applies to and table-rewriting forms of . It's a narrow window, and in the case, getting empty results is essentially the expected outcome anyway. But for , where you're compacting a table that still has data, it could be surprising.The good news is that any transaction that accessed the table before the repack started will hold an lock, which blocks the final swap step until that transaction completes. So the risk is limited to transactions that have a snapshot open but haven't touched the table yet. It's rare, but it's worth understanding for long analytical queries that could run alongside maintenance operations. It may be a good idea to check for these before scheduling regular cycles. For monitoring, Postgres 19 provides the pg_stat_progress_repack view, which tracks the operation through various operating phases. It even reports tuple counts for scanned, inserted, updated, and deleted rows during the catch-up phase. No more guessing how long a repack will take.

Final Thoughts

For years, the Postgres community's answer to table bloat has been "tune your autovacuum" or perhaps "install pg_repack" for the more adventurous. The first is still the right advice; well-tuned autovacuum prevents bloat from accumulating in the first place, and that should always be the first line of defense. But when reclaiming space is critical, having a first-class operation in-core is a nice addition.Postgres 19 is still in development and this feature will go through extensive testing before release, so there's still a chance it could be reverted. But if it survives the grueling validation process, DBAs will have a whole new tool in their kit for keeping operations running smoothly. Sometimes bloat happens, and we could have an easier way out than . Keep your fingers crossed!

Why Postgres Lacks Transparent Data Encryption

Fri, 22 May 2026 11:14:31 GMT

If you've ever compared database feature matrices, you may have noticed something a bit peculiar. Oracle has Transparent Data Encryption. SQL Server has it. MySQL has it. Even MariaDB has it. But Postgres, which we all consider the best database engine? Conspicuously absent.It’s not that nobody wants TDE. Compliance frameworks like PCI DSS and HIPAA practically demand encryption at rest. Cloud deployments make the “stolen disk” threat model more tangible than ever. And the question comes up constantly on mailing lists, at conferences, and in every database evaluation checklist ever assembled by a procurement department. So what gives?It’s complicated. The real answer involves nearly a decade of mailing list threads, competing proposals, fundamental disagreements about threat models, and a problem scope so vast it makes most contributors quietly back away. Let’s trace the history and find out why the elephant in the room is still unencrypted.

The Grand Promise of TDE

The concept of TDE is straightforward: encrypt all data when it’s written to disk, decrypt it when it’s read back into memory. The “transparent” part means the encryption layer sits below the SQL engine so client applications operate normally without special tooling or requirements.TDE protects against an attacker who gains read access to the physical storage: a stolen or improperly decommissioned drive, a backup tape that fell off a truck, or perhaps a misconfigured cloud storage volume. TDE does not protect against a compromised application, a malicious DBA, those with superuser privileges, or SQL injection. The contents of the database can be fully decrypted while the server is running, which means anyone who can connect and query can read everything.This is the crux of a debate that has raged in the Postgres community for years. If TDE only protects against disk theft, and filesystem encryption (LUKS, ZFS native encryption, dm-crypt) already does the same thing, why should Postgres spend enormous engineering effort to reinvent it inside the database?The answers, depending on who you ask, include:

Compliance checkboxes are a real business requirement.

It’s genuinely more convenient to manage encryption at the database layer.

Doing so is a waste of time; just use LUKS.

These three positions play out repeatedly.

Four Forks in Search of a Standard

Perhaps in an effort to circumvent the ongoing debate, five separate companies have independently invented TDE for Postgres, each using its own particular approach. And of course, not a single one of them is compatible with the community release, and most are proprietary. One of these (Crunchy Hardened PostgreSQL) appears to no longer exist, bringing the list down to four.Here’s what remains:Percona’s pg_tde is the only open-source option, but its full capabilities (index encryption, complete WAL encryption) require Percona Server for PostgreSQL, their custom fork. It uses a two-tier key architecture with AES-256-CBC for data and AES-256-CTR for WAL, and supports external key management through HashiCorp Vault and KMIP. Temporary files remain unencrypted, which is a notable gap for result sets that spill to disk.EnterpriseDB offers TDE in their proprietary EDB Postgres Advanced Server from version 15 onward. Their implementation is arguably the most thorough, encrypting data files (AES-XTS), WAL (AES-CTR), and temporary files (AES-CBC). The catch, beyond requiring their proprietary server, is that TDE must be enabled at time. It’s not possible to retrofit encryption onto an existing cluster.CyberTec includes TDE in their PostgreSQL Enterprise Edition (PGEE). Their approach encrypts the entire cluster using AES-CTR with no option for selective per-table encryption. It’s all or nothing. CyberTec has been vocal about their TDE work influencing subsequent community discussions, and their original implementation dates back to around Postgres 12.Fujitsu bundles TDE in their Enterprise Postgres product, using hardware-accelerated AES-256 with FIPS 140-2 compliance. Their approach encrypts at the tablespace level, meaning encrypted and unencrypted tablespaces can coexist. It’s a pragmatic design, but documentation is largely confined to whitepapers and sales materials.Five companies, four remaining implementations, zero availability in community PostgreSQL. That should illustrate the difficulty of getting this feature into core. It’s not a lack of motivation or engineering talent. Every one of these companies did the work to make TDE a reality in their flavor of Postgres.

“What Threats Do You Want to Defend Against?”

The most instructive chapter in this saga began in May 2018, when Insung Moon from NTT proposed a comprehensive TDE design on the pgsql-hackers mailing list. The proposal was ambitious: per-table encryption granularity, a two-tier key management system with KMS plugin architecture, WAL encryption with separate keys, and buffer-level encrypt/decrypt during disk I/O. Masahiko Sawada co-designed the approach.The community response was… educational.Tomas Vondra got straight to the point:This is the fundamental problem that haunts every TDE proposal for Postgres. Before designing the solution, it’s necessary to agree on the problem being solved. And the Postgres community has never reached that agreement.Sawada himself pivoted the discussion toward threat modeling, asking what specific attacks TDE should defend against. The answers diverged immediately. Some contributors saw compliance requirements as sufficient justification. Others wanted protection against curious DBAs. Still others pointed out that if threat models involve a malicious DBA, server-side encryption can’t help because the DBA has access to the running server. The same server that transparently decrypts data during queries.Then Nico Williams delivered what might be the most devastating critique in the entire thread. He pointed out that the proposal didn’t encrypt , the internal metadata tables Postgres uses to manage everything. An attacker with storage access could manipulate catalog entries to escalate privileges and extract the TDE keys themselves. Per-table encryption without catalog protection was, in his analysis, fundamentally incomplete:Williams went further, arguing that any threat model where DBAs are not the threat “is just not that interesting to address with crypto within postgres itself.” Filesystem encryption already handles the disk-theft scenario. Cryptography inside the database should protect against someone with database access. And that problem, he noted, requires far more sophisticated protections (Merkle hash trees, MAC verification) that would be “performance killing.”The conversation branched into competing proposals: per-table vs. per-tablespace vs. per-column encryption, with Joe Conway from Crunchy Data proposing transparent column-level transforms. Debates erupted about cipher modes (CBC vs. XTS, with Fujitsu’s Tsunakawa advocating for XTS per NIST SP 800-38E). Arguments about key management mechanisms went unresolved. None of these branches converged. The thread effectively ended in January 2020 with no path forward.

The Scope of the Surgery

Part of what makes TDE so daunting for Postgres is the sheer number of subsystems it touches. The PostgreSQL TDE Wiki page details the work required, and it’s harrowing. Reading it feels less like a feature specification and more like a dive into The Money Pit.The wiki establishes a reasonable core design philosophy: TDE must be secure, but it must also have minimal impact on the rest of the codebase. Since only a small fraction of users would enable TDE, the less code added, the less testing is required, and the less likely TDE is to break as Postgres evolves.Fair enough. Now here’s what “minimal impact” actually entails. needs to encrypt on every write to disk and decrypt on every read. Shared buffers stay in plaintext (that’s the “transparent” part), but the transitions need to be airtight. is where things get genuinely hairy. WAL buffers are plaintext in memory but must be encrypted when flushed to disk, using a separate encryption key and CTR-mode cipher. Key rotation during streaming replication gets complicated fast. Does the WAL sender decrypt before transmitting? Can a primary and standby use different keys? What happens to key state during a standby promotion? need a unified I/O API so encryption can be applied consistently. Parallel query workers must share the temporary key. And because temp files are created and destroyed constantly during query execution, performance overhead here is especially visible.in the Postgres ecosystem needs modification: , , . Each needs a new passphrase command option. Only escapes, since it reads data through SQL (already decrypted in shared memory).Then there are the many edge cases. Hint-bit updates, which normally flip a single bit, produce entirely different ciphertext under block encryption, so becomes mandatory. must potentially decrypt and re-encrypt all copied files if the database OID is part of the initialization vector. Some GiST index pages have fixed LSNs assigned at build time, creating IV reuse scenarios. The page checksum question (encrypt the CRC or not?) remains unresolved. Encrypting it provides a weak form of integrity checking but forces every command-line tool that verifies checksums to also possess the encryption key.The wiki lists dozens of specific on-disk files requiring encryption, spanning table data, caches, dynamic shared memory segments, replication slot snapshots, statistics files, and all WAL segments. In the face of all this, the idea of “minimal impact” becomes a pipe dream.

Torn Arguments and Pages

One of the earliest attempts to get TDE into Postgres core came from Ants Aasma back in 2016. He submitted a cluster-wide encryption patch that proposed AES-128-XTS encryption of all data files with a single master key. The design was deliberately simple: one GUC, a narrow API (one key-setup function, two encrypt/decrypt functions), and key delivery via environment variable.Robert Haas raised the most consequential technical objection. Encrypting WAL a block at a time breaks Postgres’s torn-page safety model. If a crash tears an encrypted block mid-write, it becomes impossible to decrypt the partial result, and WAL replay stops dead. This undermines guarantees, which is to say, it undermines one of the most fundamental promises Postgres makes about data.Peter Eisentraut challenged the entire premise, noting that filesystem-level encryption already exists and is well-tested. He then raised the topic of tool integration burden and pointed out that environment variables are visible to other processes, making the key-entry mechanism insecure on its face.Bruce Momjian was characteristically blunt:Hans-Jurgen Schonig from CyberTec pushed back, arguing the demand was real, especially in financial services where regulations explicitly required database-level encryption. Stephen Frost emphasized backup portability as a genuine advantage: encrypted backups can be restored anywhere with the key, unlike filesystem encryption tied to a specific mount point.But the arguments for TDE never quite overcame the arguments against. Aasma himself seemed to sense this, calling TDE a “checkbox feature” and openly asking whether the community really thought it was worth the effort. Early benchmarks showed a 2x raw slowdown, and further performance optimization was promised but never delivered. The thread faded out with no consensus.

Keys to the Kingdom

By 2020, Sawada took a different tack. Rather than proposing full TDE, he submitted an internal key management system (KMS) as a foundation that future TDE work could build upon. The design centered on a master encryption key protected by a user-provided passphrase, with SQL functions and to encrypt and decrypt internal keys. Sehrope Sarkuni contributed a detailed cryptographic specification using a 64-byte master key split into separate encryption and MAC keys, backed by OpenSSL.The pragmatic “build the foundation first” approach made sense. Unfortunately, the foundation had cracks.Fabien Coelho spotted the first problem: if is a regular SQL function, any database user can simply call and extract the key. That rather defeats the stated goal of hiding keys from users.Robert Haas identified the second, arguably fatal problem. Passing encryption keys as SQL function arguments means they show up in query logs whenever is configured, or whenever a query throws an error. His verdict was unsparing:Then Tom Lane weighed in with a position that surprised many:This legal and policy argument had a chilling effect on the discussion. Andres Freund pushed back, arguing that crypto export restrictions are largely anachronistic in 2020, but the point was never fully resolved. Freund also took the opportunity to warn against building on pgcrypto, calling its code quality “well below our standards” with “very outdated copies of cryptography algorithm implementations.” That’s not a vote of confidence for the only crypto extension currently shipping with Postgres.Chris Travers and Robert Haas both proposed alternatives that bypassed the SQL layer entirely, using new protocol-level messages to exchange key material without ever touching the query parser or the log infrastructure. These designs were architecturally cleaner but also far more invasive, requiring changes to the client/server protocol itself.All of these observations apparently proved insurmountable, and the thread eventually ended in October 2020 with no resolution. The patch is still listed as “Returned with feedback.”

Meanwhile, in a Different Room

While the TDE discussion sputtered out, Peter Eisentraut started work on something related but fundamentally different. In December 2021, he proposed Transparent Column Encryption (TCE), a client-side feature that encrypts individual columns before data ever reaches the server. The server only sees ciphertext. It can store it, replicate it, back it up, but it cannot read it.Eisentraut was explicit about the distinction:The threat model is completely different from TDE. Where TDE protects against physical disk theft, column encryption protects against the DBA. The server never has the plaintext, so even a malicious administrator can’t read the protected columns. The canonical use case is PCI DSS compliance for credit card numbers, social security numbers, and similarly sensitive data.The design was both clever and intricate, requiring a boatload of core modifications:

Protocol-level changes.

Prepared statements for encrypted inserts, since the client must intercept parameter values to encrypt them.

A two-tier key hierarchy of Column Encryption Keys wrapped by Column Master Keys stored in an external KMS.

New DDL commands such as
and
.

Notice something that’s missing? Since Postgres can’t access the raw data values, it becomes impossible to compute on encrypted columns server-side. That means no indexing, sorting, or clauses against encrypted values. That’s more than slightly inconvenient.Despite that, the patch went through dozens of iterations over two and a half years. Jacob Champion raised thorny questions about NULL handling (unencrypted NULLs leak information about which rows have absent data), cryptographic algorithm selection, and AEAD integration. Tomas Vondra pragmatically argued for shipping a basic version that protected against passive attackers first, deferring stronger protections to future work. This patch also remains in “Returned with feedback” status since the thread ended in August 2024.Even the alternative approach to database-layer encryption hit the same wall: perfect becomes the enemy of good.

A Method to the Madness

Step back from the individual threads and a clear pattern emerges. Every TDE proposal for Postgres has died for some combination of the same five reasons.. This is the root cause, and everything else flows from it. Some contributors want protection against disk theft, but then LUKS works fine. Some want protection against DBAs, but server-side encryption can’t deliver that because the data is decrypted in memory. Some want a compliance checkbox, but is that worth the engineering cost of touching every subsystem in the database? Without consensus on what problem TDE solves, nobody can evaluate whether any particular design is the right one. Vondra’s observation from 2018 remains perfectly accurate: TDE keeps dying because the community can’t agree on what it’s for.. TDE touches buffer management, WAL, temporary files, replication, every backup tool, dozens of on-disk file types, and creates subtle interactions with checksums, hint bits, index construction, and database creation. This isn’t a trivial bolt-on feature. Properly adding TDE is a multi-year project plan, which can’t happen until everyone agrees on an approach.. The database needs the key to function, but storing the key near the data undermines the encryption. Passing keys through SQL exposes them in query logs. Environment variables are visible to other OS processes. External KMS integration adds deployment complexity and latency. Protocol-level key exchange requires invasive client/server changes. Can any approach address all of these concerns?. Bruce Momjian’s “unimpressed” comment, Tom Lane’s “never crypto in core” position, and Peter Eisentraut’s “filesystem encryption already exists” argument reflect real convictions from people who’ve been maintaining Postgres for decades. These are the perspectives of engineers who know exactly how much maintenance burden a feature like this creates, and who have watched the same proposal cycle through the mailing list multiple times without resolution.. As Robert Haas put it:This is technically correct, but try explaining that to an auditor who expects to see “TDE” in a feature checklist for compliance purposes.None of this means TDE is pointless, of course. Even Vondra, one of the more vocal skeptics about the threat model confusion, acknowledged that:There’s real value in a single flag that encrypts everything, manages key rotation, and produces encrypted backups by default. But is that enough to justify the work required to reach that point?Five years have passed since the last serious attempt at TDE in Postgres core. The Sawada KMS thread died in 2020. Eisentraut’s column encryption (a different feature entirely) stalled in 2024. No new proposals have appeared on the horizon. That’s a long silence for a feature with this much demand.

The Self Service Lane

It’s possible to have encryption at rest for Postgres today without adopting a proprietary fork. The options are well-established if somewhat mundane.with LUKS (Linux Unified Key Setup) is the most common approach. Create an encrypted volume, put the Postgres data directory on it, done. Performance overhead is modest (typically 5-10% with AES-NI hardware), key management is handled by the OS, and it’s battle-tested across millions of deployments. is available in ZFS, Ceph, and a handful of others. These don’t require the additional encryption layer, and often encrypt at the volume level. This enables tablespace-level granularity if necessary, without a lot of extra setup. These filesystems make transparent encryption practically “free”. such as AWS EBS encryption, GCP persistent disk encryption, and Azure disk encryption, are also available. They’re transparent, often enabled by default, and satisfy most compliance frameworks. It’s likely that Postgres cloud deployments already have encryption at rest without having done anything special.For column-level protection of specific sensitive values, pgcrypto remains available as a contrib extension. While it’s not “transparent” in any sense, it does keep ciphertext in the actual column storage, which satisfies certain compliance requirements.

The Debate Continues

Postgres is a remarkable database. It handles replication, partitioning, JSON, full-text search, geospatial data, and a hundred other things with a degree of sophistication that rightfully threatens many commercial offerings. But TDE remains the feature that “got away”.The community’s caution is understandable. Bad encryption is worse than no encryption because it creates a false sense of security. The scope of the change is genuinely massive. The threat model question has never been answered to everyone’s satisfaction. And the “just use LUKS” argument, while unsatisfying to procurement departments, is technically sound.But the gap is real. Every year that Postgres lacks native TDE, the four proprietary forks get a little more entrenched with customers who can’t fight the compliance battle anymore. Every abandoned mailing list thread makes the next proposal a harder sell. And every database evaluation matrix with a blank cell next to “Encryption at Rest” costs the Postgres ecosystem mindshare it didn’t have to lose. Many in the Postgres community like to poke fun at MySQL, but even they have it.Maybe the path forward is something nobody has tried yet: a modular, hook-based approach that lets extensions provide encryption without requiring crypto in core (neatly sidestepping Tom Lane’s objection). Maybe Percona’s pg_tde, imperfect as it is, eventually matures enough to become a de facto standard. Or maybe some future contributor will finally write the patch that threads the needle between skeptics and scope.Until then, format a LUKS partition, click the cloud provider’s disk encryption checkbox, keep an eye on the mailing lists, and move on with life. Given the community’s devotion to correctness in engineering, any solution that meets their standards is likely superior to competing implementations. When TDE finally arrives, it’s bound to be—like Postgres—the best there is.

Unlimited power: Letting go of Postgres Column Lengths

Fri, 15 May 2026 06:22:45 GMT

Somewhere in the dusty archives of database history, someone decided that column type declarations were the perfect place to enforce business rules. It made a kind of intuitive sense at the time: if a Social Security Number is always 11 characters, just slap on there and let the database handle it.Except that's not how it works in practice. All it takes is one entry that requires a longer field length to reveal the monstrosities lurking within this approach. Like most databases, Postgres tracks object dependencies. Column attributes get copied by numerous downstream database objects, and even a single such dependency prevents column modifications. The longer the dependency chain, the worse the situation becomes.Fortunately there's a better way, and it's been sitting right there in the SQL standard since the beginning: CHECK constraints. They do the same job without the collateral repercussions.Let's see how.

Catalogs Cast in Stone

When declaring a column as , Postgres dutifully records that type modifier in the attribute catalog, specifically in the column. That value becomes metadata about the column itself, baked directly into the type definition. It's actually a property of the column that Postgres enforces at the storage layer.Why does this matter? Because anything that references that column inherits the type definition wholesale. Views, materialized views, functions with typed parameters, composite types, you name it. They all pick up as if it were gospel, because from the catalog's perspective, it is.Consider a straightforward scenario:Everything looks fine, and the length limit works as expected:Great! The system did its job. But now business requirements shift (as they always do) and we need to allow slightly longer strings. Should be easy, right?Oof. The view uses the original type definition as part of its own structure, and Postgres refuses to let us change the underlying column as a result. This is actually correct behavior on the part of Postgres; silently altering the type of a column that other objects depend on would be far worse than refusing. It would be nice if Postgres had an option to cascade the change through all dependent objects, but that isn't available, so we are stuck with the original definition.

The Dependency Cascade

In a toy example with one table and one view, the fix is obvious: drop the view, alter the column, recreate the view. Annoying, but manageable.Now imagine a production database with 200 tables, many of which have multiple views layered on top of them. Some of those views feed other views. Some are referenced by functions. Others are exposed through an API layer that has its own set of assumptions about column types. Suddenly "just drop and recreate" becomes a multi-hour migration involving careful dependency ordering, and forbid something goes wrong halfway through.The pg_depend catalog tracks all of these relationships, and it can be genuinely alarming to see how deep the dependency tree goes for a single column type change. On a sufficiently mature system, altering to might require dropping and recreating a dozen objects in exactly the right order, all within a transaction that you're silently praying doesn't fail at step eleven.This isn't a Postgres limitation so much as a consequence of how SQL type systems work. The type modifier is part of the column's identity. Changing it is a structural alteration with many downstream effects.So what can we do instead?

Constraints to the Rescue

Postgres CHECK constraints take an entirely different approach. Instead of embedding validation into the type system, they sit alongside the column as independent database objects. They're evaluated at runtime during INSERT and UPDATE operations, and they have their own identity in the catalog. Most importantly, they can be dropped and recreated independently of the column definition.Here's the same table, rebuilt with a constraint instead of a type modifier:Notice that the column is now plain with no length specifier. The length enforcement comes entirely from the named constraint. Let's verify it works:The string "long string" is 11 characters, and our constraint requires , so it fails as expected. The error message is different (it names the constraint rather than citing the type), but the effect is identical: invalid data is rejected.Now for the moment of truth. Can we change the limit without disturbing the view?No error on the , no complaints about dependent views, and the insert succeeds because "long string" is well within our new 15-character limit. The view never knew anything changed because, from its perspective, nothing did. The column is still , same as before. Only the constraint was swapped, and constraints don't participate in type dependency tracking.

Unlimited Cosmic Power

Once you start thinking of validation as a constraint rather than a type attribute, the possibilities open up considerably. A only stipulates that "this string must be 10 characters or fewer." A CHECK constraint can enforce anything that evaluates as a boolean.Want to enforce a minimum length as well?Need to ensure a code follows a specific pattern like ?How about validating that a column only contains printable ASCII?None of these are possible with type modifiers alone. It's possible to drop, modify, and recreate them without any impact on dependent objects. The constraint is decoupled from the column's identity, which means it's decoupled from the entire dependency graph. We can add or remove as many arbitrary rules as we want, with only a short lock to update the system catalog.There's also a subtlety worth noting here regarding NULL handling. CHECK constraints evaluate successfully when the expression returns or NULL. This means a CHECK constraint on string length won't prevent NULL values on its own; it’s still necessary to specify if that's the intent.Alternatively, it's also possible to do this:Remember, any boolean evaluation works here. Yes it's better to use in the column definition itself, but what if the table already has billions of rows and you want to prevent more NULL values from being inserted? Normally that requires a trigger until it’s possible to change the column definition, and we can't change the definition until all existing rows contain non-NULL values. The CHECK definition bridges the gap without as much overhead.

The One True Type

Many experienced Postgres users will simply recommend using for all string columns. Postgres does, after all, treat , , and identically in terms of storage and performance. The (n) modifier merely adds attribute metadata, it doesn't optimize storage. A column doesn't use less space than a column storing the same string. The type modifier is purely a validation mechanism, and as we've established, a rather draconian one.Using everywhere with CHECK constraints for validation achieves the best of both worlds. This approach provides the same enforcement as a column length specification, and delivers improved flexibility to modify constraints without a multi-step migration. Constraints also simplify migrations from legacy schemas. An existing database chock-full of columns now has a modernization path that looks something like this:Yes, the migration itself still requires dealing with view dependencies; that's unavoidable for the initial type change. But it's a one-time cost. After that, every future length change is a simple constraint swap with zero dependency headaches.

The Performance Question

What about overhead? Is a CHECK constraint slower to enforce than a type length?The type modifier check happens at a very low level in the Postgres input function for the type. Postgres CHECK constraints are evaluated as an expression during the standard constraint-checking phase of an INSERT or UPDATE. As a natural consequence, the CHECK constraint involves marginally more overhead because it's running an expression evaluator rather than a simple integer comparison in the type input function.This is something we can actually quantify. Let's check a 10-million row insert using the standard approach: And now with a CHECK constraint:It's not exactly statistical noise, but a difference of ~66 nanoseconds per row is hard to really complain about. We may as well quibble about the weight of the paint on a freight train.Additionally, CHECK constraints convey better diagnostic information when they fire. Rather than the generic "value too long for type character varying(10)", Postgres reports the offending constraint name, which is presumably a more meaningful designation. On a table with many string columns, knowing that a value violated tends to be more helpful than knowing one of many column values was too long.

All in a Name

Speaking of constraint names, this is one area where discipline pays dividends. Always name CHECK constraints explicitly. Postgres will auto-generate names otherwise, and while those algorithmic names are functional, they leave a lot to be desired.Compare:Postgres produced a constraint named in this case. What was the check? Formatting? Validation of "name@tld" contents? Something else? Let's be more specific:The explicit name clearly communicates what the constraint enforces from the name itself. When a constraint violation shows up in application logs at 3 AM, the constraint naming-scheme should disambiguate what went wrong without having to examine the table schema. It also makes the drop-and-recreate pattern cleaner. Rather than looking up the auto-generated constraint name or assuming what Postgres will do based on inputs, all future operations reference a predefined object name without any ambiguity.

Learning to Let Go

The core insight here is really about where validation logic belongs in a database schema. Type modifiers embed validation into the column's identity, which ties it into the dependency graph and makes it resistant to change. CHECK constraints keep validation as a separate, independently manageable object. This results in the same business logic enforcement pattern, but uses a subtly different dependency graph that’s far more flexible.For legacy systems that already have everywhere, the migration to CHECK constraints (or with CHECK constraints) is a worthwhile investment. Yes, there's a one-time cost to untangle the existing dependencies, but every subsequent schema change becomes simpler. For new schemas, there's really no reason to use at all unless an external system absolutely demands it.Postgres gives us the tools to build schemas that are both strict and adaptable. CHECK constraints are one of those tools that have always been there, patiently waiting for us to stop hard-coding our validation into the type system. It's time we started using them.Give it a try on your next project, and you'll wonder why you ever put a number in parentheses after . True enlightenment comes with the second epiphany that TEXT takes this a step further, forever relegating legacy SQL types to the dustbin of history. If you’re not ready for that particular conversation just yet, the Postgres community is nothing but patient. Consider CHECK constraints your introduction to contemporary SQL the Postgres Way™.

No Compiler Required: Writing SQL-Only Postgres Extensions

Fri, 08 May 2026 12:11:40 GMT

Recently at Postgres Conference 2026 in San Jose, I presented a talk called Let's Build a Postgres Extension! Since that entire presentation was primarily focused on writing a C extension while exploring the Postgres source code, I only mentioned pure SQL extensions as an aside. But what's more likely in the Postgres community in general: C devs, or people who know SQL?It turns out that you can do a lot with functions, triggers, views, tables, and various other Postgres-native capabilities. The extension system doesn't care whether the contents are compiled C or plain SQL. It just wants a control file, a SQL script, and an optional to help with installation.So let's build a relatively trivial extension article entirely in SQL.

What Do We Want?

First things first: we need a plan. What should this extension actually do? I wrote about blocking DDL a while back with a C extension, so why not revisit that example with SQL?This being pure SQL, there are other handy elements we can add with very little effort, so how about:

A setting to enable or disable the extension.

A setting to allow or block superusers from executing DDL.

A role that allows members to bypass the DDL restriction.

A function to add users to the bypass role.

A function to remove users from the bypass role.

A view to see which users are in the bypass role.

An
event trigger
to actually block DDL attempts.

Rather than a simple event trigger to prevent DDL execution, we are building a kind of DDL execution management suite. That should hopefully demonstrate just how capable a purely SQL implementation can be.

Three Files and a Dream

Every Postgres extension, regardless of complexity, boils down to the same basic structure:

A control file to describe the extension.

A SQL script to create the tables, views, functions, etc.

An optional Makefile to copy the SQL script and control file to the right place. Unlike a C project, there's no build step for a SQL-only extension because there's nothing to compile.

Here's what our project directory looks like:Let's start with the control file. It tells Postgres the extension's name, version, description, and the settings of a handful of behavioral flags. Ours looks like this:The is what shows up in and the catalog view. The tells Postgres which SQL script to load when someone runs without specifying a version. The flag means only superusers can install or update this extension. That's the default, but it's better to be explicit.The flag deserves a quick explanation. A relocatable extension can be moved between schemas after installation with . Ours can't because the SQL script references a specific schema internally using the substitution token. It's still possible (and recommended) to define the schema during installation, but not afterwards.Next comes the . For a C extension, the Makefile orchestrates compilation and linking. For a SQL-only extension, it just copies the control and SQL file to the library folder where Postgres keeps extensions. Here’s the whole thing:There's normally also a line to specify C source files for compilation. Without it, simply copies the control file and SQL script to the correct directories. The PGXS build infrastructure handles the rest.With the boilerplate out of the way, it's time to have some fun.

A Bit of Bookkeeping

Before we really get going, the extension needs to live in a schema. Some of the objects in that schema need to be publicly accessible. So the first thing in our file needs to look like this:Usage just means the schema objects are visible. Users won't be able to create objects or even select from tables unless granted specifically.After that, we need to account for the configuration settings. You may think the first choice for this is to use session variables, but that's a subtle trap. The problem here is that SQL-only extensions don't have access to the finer points of system variables, such as limiting them to superusers, system start, service reload, etc. That means there's nothing preventing users from overriding them with a simple statement.The next option is a configuration table. The extension documentation says we can register these such that dumping and restoring a database retains values, and it's trivial to control table updates. So let's start our extension with this:Now only superusers can configure the extension! Regular users still need to be able to read the configuration table because the event trigger runs as that user. In any case, we now have a convenient configuration interface.

On a Role

The next step is to allow certain users to bypass the DDL restriction. The easiest way to do this is to create a role that a superuser can grant to these allowed users. We can also take care of our helpful grant/revoke functions here:The reason for the long role name which incorporates the extension name is to prevent name collisions. This role likely isn't already in use, and its purpose is obvious. The functions mean admins don't need to remember the role name itself, but they're not required either.The final thing to add is the view which lists bypass users:Is that a query you knew off the top of your head? Probably not. Now the extension takes care of it so you don't have to.

You Shall Not Pass

The core of our extension is a DDL blocker: an event trigger that fires on and raises an exception unless the session user is a superuser. The C version of this blocking routine was quite a bit more complex than what we're building here.Here's our DDL blocking function:The declaration makes this function eligible for use with . It's a special return type that signals to Postgres how to invoke the function.The superuser check queries the for the . This allows superusers to masquerade as other users for testing purposes, and potentially to prevent accidental DDL executions, provided they first. The final check is against the view we created. It may be tempting to use the information function for this, but that shows effective privileges, not actual membership. Superusers have all privileges, so would automatically pass this check if we didn't explicitly validate against role membership.With the function in place, creating the event trigger is a one-liner to call the function:The event fires before any DDL command executes. If our function raises an exception at this point, the command never runs. Easy peasy.What qualifies as "DDL" in the eyes of Postgres? Quite a lot, actually. The event fires for , and . It does not fire for commands targeting databases, roles, tablespaces, or ironically, event triggers themselves.We could also filter to specific command tags using a clause:But where's the fun in that?

Kicking the Tires

Time to see if this thing actually works. First, install the extension files:That copies and to the extension directory. Now connect to a database and create the extension:The extension is installed. Let's verify the event trigger is in place:The in means "origin," which is the default enabled state (fires in all contexts except replication). It's go time!

Testing the Blocker

Blocking is off by default. Let's confirm by creating a throwaway table:No complaints. Now let's enable the blocker:Then another test:Still works. Superusers get a free pass by default. Let's fix that loophole:Now DDL commands are now stopped cold. This should work for any potential DDL:Does our bypass system work?The explicit bypass now allows DDL. What about regular users? Let's create one and test again:Works exactly as advertised!

The Fine Print

SQL-only extensions are powerful, but they're not a complete replacement for C. A few tradeoffs are worth understanding before you commit to one approach or the other.The GUC security gap. In the C version of this extension, the GUC is registered with context, meaning only superusers can change it. In our SQL-only version, would be a custom parameter that any session can modify. We had to devise a somewhat convoluted workaround for this by using a configuration table. This wouldn't be necessary if there was some kind of SQL interface to true variable registration for extensions.Event trigger blind spots. Some DDL commands don't fire event triggers at all. Operations on databases, roles, tablespaces, and event triggers themselves are exempt. Operations such as or are entirely exempt. That's where Postgres's built-in privilege system (or restrictions) should do the heavy lifting. Once again, a C extension has access to capabilities our SQL version can only dream about.No background workers or hooks. C extensions can register background workers, intercept query planning, hook into the executor, and modify server behavior at a fundamental level. SQL-only extensions operate entirely within the SQL layer. If your use case involves any of those deeper capabilities, C is the only option.For everything else? Functions, triggers, event triggers, views, types, domains, operators, aggregates, tables, and more can all live inside a SQL-only extension. That covers a remarkable amount of ground.

Wrapping Up

The Postgres extension system is often perceived as something that requires C expertise, a compiler toolchain, and a deep understanding of the server internals. That’s only really the case if you need the deep internals. If you've ever written a collection of utility functions and wished you could install them with a single command, you're already thinking in extensions. The packaging is the point.Our extension demonstrates custom configuration tables, roles, functions, views, and event triggers. All of these are standard SQL primitives that any Postgres user already knows. The only new pieces are a minimal control file and . That's a few lines of overhead to gain clean installation and removal, version management, and dependency tracking.If you've got a handful of functions, views, or triggers that you deploy to every database in your environment, consider taking an afternoon to wrap them in an extension. Your future self, and anyone else who inherits those databases, may thank you for it.

It Depends: Using Session Variables in Postgres

Fri, 01 May 2026 05:36:14 GMT

There's been a kind of persistent myth regarding Postgres since I first started using it seriously over 20 years ago: "Postgres doesn't support user variables." This hasn't really been true since version 8.0 way back in 2005. Part of this stems from the fact it doesn't do things the same way as other common database engines.Why don't we spend a little time exploring the functionality that time forgot?

What Everyone Else Is Doing

Before I delve into the Postgres approach, let's take a look at the competition. If anyone wants to switch to Postgres (as they should), they'll bring along plenty of assumptions.Let's start with MySQL, the formerly undisputed database king of the LAMP stack. MySQL session variables merely prefix any name with to assign a value:Simple, right? It's even possible to use them directly in queries:We don't have to get into the finer minutiae here, as the MySQL documentation on user-defined variables does that job splendidly. The point is that some users expect this level of compatibility and balk when it's missing.When it comes to SQL Server, things are very similar to MySQL, though perhaps a bit more structured:Once again, the SQL Server documentation on variables is pretty clear about how these work. The primary caveat here is that these are limited to the current batch, making them somewhat tedious to work with in some cases.The picture for Oracle is a bit different. Oracle calls them substitution variables, and prefixes using rather than : This is also closer to a macro system than a true variable; the SQL*Plus or SQLcl clients substitute the values prior to sending statements to the server. It's not something other drivers or clients can use unless they added it themselves for compatibility purposes.

Postgres Has Entered the Chat

So where does Postgres fit into all of this?If Oracle's substitution is what you're accustomed to, Postgres actually has a direct equivalent. The psql client supports for defining client-side variables: The tool has supported these practically since the beginning, but some users find them insufficient. As with Oracle client-side substitutions, they only work specifically in the client. This can be a major limitation when interacting with Postgres in any other manner.The real answer lives server-side, and it's been hiding in plain sight since at least version 9.2 in 2012. It doesn't require any special syntax, extensions, or package declarations, and uses the same configuration parameter system that controls options like and . That's right, it's just the regular SET statement.The trick is in the usage. Postgres treats any name containing a period as a custom parameter:That's it. Those values now exist for the duration of the current session. Reading them back is just as straightforward with the SHOW statement: And clearing them:The dot in the name is mandatory for custom parameters. The part before the dot acts as a namespace (like , , or ). Postgres uses this to distinguish custom settings from its own built-in configuration parameters.But and are SQL statements, which means it's not conventionally possible to embed them in expressions or subqueries. So how do we use these handy new parameters?

Getting and Setting

Postgres provides two system administration functions that make manipulating user variables easy and convenient.Let's take a look at first. It's the functional analog of the statement:The third parameter just means: "should this only be a local parameter?" Setting it to allows the value to persist for the entire session, which is what most users may expect. Because is a regular function, it works everywhere a SQL expression is valid.Then we can use to retrieve that value:As the functional equivalent of , it returns the current value of any configuration parameter as text.The optional second parameter () is a small but important detail. By default, requesting an unset parameter raises an error:Rather than wrapping everything in an exception handler, it's possible to suppress the error and return by setting the second parameter to : Because and are plain functions, they integrate naturally with any SQL statement. Perhaps we need to convert a username to a tenant ID:Need to filter by the current tenant?Need to set context at connection time and have it flow through to every query automatically?Need to capture the current user in an audit trigger?Allowing unset variables in the trigger is a nice safety net. If the application forgot to set the variable, the audit column gets instead of aborting the insert. Whether that's the right behavior depends on the specific use case, but it's great to have options.

Properly Scoped

Remember that third parameter in ? When is , the value applies only for the current transaction. Once the transaction commits or rolls back, the value reverts to whatever the session-level setting was before. This is equivalent to the SQL statement:The same behavior works with : Why is this useful? Consider a web application where each HTTP request maps to a single database transaction. The application sets the request context at the start of the transaction, and every query within that transaction can see it. After the transaction commits, the context is automatically cleaned up. No manual needed, and no risk of context leaks between transactions.This is especially valuable with connection poolers like PgBouncer, where a single database session handles requests from many different application users. Transaction-scoped variables guarantee that context from user A never bleeds into a query for user B, because each user's transaction carries its own isolated state.Rollbacks work as expected, too:Local scope also applies when transactions rollback to a savepoint:

A Practical Exercise

Let's build something a bit more operational. We'll set up a multi-tenant orders table with automatic audit stamping and row-level security, all driven by session variables.First, the table:Next, a trigger that automatically stamps every new row with the application user:Now add a Row-Level Security policy that filters rows by the current tenant:With this in place, the application sets context when checking a connection out of the pool:And then every operation automatically respects the tenant boundary:If we switch tenants, the view changes automatically:No explicit in the query or risk of forgetting the filter. The session variable drives the RLS policy, and the RLS policy enforces the boundary. This is the same pattern SQL Server achieves with RLS plus , and what Oracle targets with Virtual Private Database plus (though Oracle requires considerably more setup to get there). And Postgres achieves it with far less ceremony than either.For transaction-scoped isolation (ideal with connection poolers), swap for :

A Few Caveats

Custom GUC parameters are powerful, but they come with some characteristics that deserve full disclosure.Everything is a string. Postgres stores all custom parameter values as text. There is no type enforcement whatsoever. If you , Postgres will utter nary a single complaint. You may have noticed the manual casting through all of the examples with .There is no access control. Any session can any custom parameter. RLS policies that rely on are only as secure as the application's control over SQL execution. Users with direct access can trivially and bypass the policy. A more reasonable RLS-based security system might leverage or and a mapping table to prevent such escape attempts.History lesson: . Before PostgreSQL 9.2, custom parameters required pre-registration. It was necessary to list extra namespaces in : Without this declaration, would raise an error. PostgreSQL 9.2 removed this restriction, and thus the freestyle dot syntax was born. If you encounter old documentation or Stack Overflow answers referencing , they're describing behavior from over a decade ago. Sadly, that's more common than it should be. They integrate with the full GUC machinery. Custom parameters aren't some bolted-on feature. They participate in everything the built-in parameters do: These defaults then act as the baseline that and override at the session level, and reverts to. It's the same layered configuration model that governs , , and other GUCs. Custom parameters are first-class citizens of the Postgres configuration system, especially in recent releases.

Reduce, Reuse, Recycle

Postgres didn't forget to implement session variables. Why invent a separate subsystem for functionality that was there from the beginning? The configuration parameter system, originally designed for things like and turned out to be a natural fit for arbitrary session state once the custom namespace restriction was lifted in 9.2.Does it lack the syntactic sugar of MySQL or SQL Server's approach? There's no denying that. But it also avoids MySQL's type ambiguity, Oracle's client-side-only limitation, and SQL Server's batch-scoping limitation. There's always a compromise and no approach is truly perfect; Postgres straddles the line between power and convenience.Either way, the next time someone claims Postgres can't do session variables, point them at and . They've been around for a long time, but recent versions have made them much more approachable. And if you haven't given them a try yet, there's no time like the present!

The Scaling Ceiling: When One Postgres Instance Tries to Be Everything

Fri, 24 Apr 2026 11:36:30 GMT

There's a persistent belief in the database world that vertical scaling solves all problems. Need more throughput? Add CPUs. Running out of cache? More RAM. Queries hitting disk? Higher IOPS. It's a comforting philosophy because it's simple, and for a surprisingly long time, it works. A single beefy Postgres instance can handle an enormous amount of punishment before collapsing under the strain.But there's a ceiling up there, and it's not made of hardware. Postgres was designed as a single-instance database engine, and many of its internal structures are shared across every database the instance contains. These shared resources are rarely concerning in a single modest instance. But with twenty databases running a mixture of heavy OLTP workloads, analytical queries, or even mostly idle, the shared nature of these internals becomes very relevant.Let’s talk about the barriers these over-provisioned instances eventually hit, with references to the Postgres source code itself for good measure. Some of these are well known, while others are the kind of thing that strikes suddenly at 2 AM when all the monitoring dashboards turn red simultaneously.

One Pool to Rule Them All

The parameter is probably the first tunable every Postgres administrator encounters. It controls the size of Postgres's own buffer cache, the region of shared memory where frequently accessed disk pages live so they don't need to be fetched from storage on every read. The documentation suggests starting at 25% of system RAM, and that's reasonable advice for a single-database instance. Most experts in the subject agree.It’s easy to forget that this allocation is instance-wide. The contents of bear this out, as the buffer pool gets allocated once at startup as a flat array of pages in shared memory:There is no per-database partitioning, no priority system, no reservation mechanism. Every database on the instance competes for the same pages in the same pool. An analytics query scanning a 500GB table in one database will happily evict cached pages that belong to a latency-sensitive OLTP workload in another. The buffer replacement algorithm (a clock-sweep LRU variant) has no concept of "this page belongs to an important database."The same applies at the operating system level. The kernel's filesystem cache, often called the "double buffer" in Postgres circles because accounts for it, is also shared across all processes on the machine. Two databases with fundamentally different access patterns, one doing sequential scans and the other doing random index lookups, will thrash each other's cached pages with no way to intervene.Will throwing more RAM at the problem help? Only until the largest working sets collide. At that point, it becomes the worst example of the Noisy Neighbor problem.

The 32-Bit Treadmill

The 32-bit nature of the Postgres transaction ID (XID) is practically venerated as something of an old joke by this point. Blogs warning about the dreaded "XID wraparound" terror are easy to find. The Postgres fix for this is VACUUM, specifically the operation. Most tuples have an associated XID, but since there are a limited number of those, tuples past a certain horizon get "frozen". Frozen tuples still have an XID, but Postgres ignores it and treats the data as if it has always existed. And thus by magic, that 4-billion transaction window only cares about "recent" transactions (for varying definitions of recent).Unfortunately, this counter persists across the entire instance. In , the function draws from a single global well:Read that error message carefully. The instance refuses all new transactions to protect a specific database. A single neglected database out of dozens can accumulate enough XID age and cause the entire instance to grind to a halt. Every tenant suffers because one database didn't get vacuumed in time, or some resource artificially held onto a visible tuple so long it couldn’t be cleaned up.The function in the same file makes this even more explicit. It computes the wraparound danger threshold based on "the oldest XID that might exist in any database of our cluster." One database's frozen-XID age becomes the constraint for every other database sharing that instance.

Multixact: The Other Wraparound

If XID wraparound is Postgres's well-publicized villain, multixact wraparound is the esoteric threat. Multixacts exist to track shared row-level locks; when multiple transactions hold locks on the same row, Postgres records them as a "multixact" group rather than storing each lock individually. Like XIDs, multixact IDs are 32-bit counters that wrap around, and like XIDs, they're instance-wide.But the member storage, the actual record of which transactions participate in each multixact, has its own nasty limit. The source code in spells out the on-disk layout with typical Postgres clarity:The math is straightforward but the implications are severe. With 409 groups per 8KB page and 4 XIDs per group, we can work out the total SLRU address space: 2^32 member offsets divided by 1,636 members per page, multiplied by 8KB per page. That comes out to roughly 21GB of multixact member storage for the entire instance.That 21GB ceiling might sound generous until you consider a multi-tenant setup with aggressive row-level locking. A workload that performs across many rows, or any application pattern that causes multiple transactions to hold shared locks on the same tuples, burns through multixact members quickly. Once exhausted, the instance starts refusing operations just as it does for XID wraparound, except the monitoring for multixact usage is far less mature in most environments.Worse, the same "slowest database wins" dynamic applies. The global minimum across all databases governs when the SLRU can be truncated. One database with inadequate vacuuming of multixact-heavy tables can pin that minimum in place for the entire instance. Similarly, a single database can greedily monopolize that precious resource simply due to unusual or aggressive locking behavior.

The One-Lane Highway of WAL Replay

Postgres streaming replication works by shipping Write-Ahead Log (WAL) records from the primary to replicas, which then replay them to stay current. It's a utilitarian and reliable workhorse, but there's a fundamental constraint: replay is single-threaded.In , the main redo loop that processes WAL on a replica is exactly what it looks like:One record at a time, sequentially, in a single process. The startup process (which handles WAL recovery) is the sole consumer of WAL data on a replica. There is no parallel apply. Even an over-provisioned 128-core machine acting as a replica can only leverage a single core for processing WAL data.The recovery_prefetch parameter (defaulting to since Postgres 15) helps when the bottleneck is IO. It looks ahead in the WAL stream and issues asynchronous reads for pages that will be needed soon, reducing stalls caused by cold cache hits. The prefetcher documentation in describes it as a "drop-in replacement for an XLogReader that tries to minimize IO stalls by looking ahead in the WAL."But if a primary generates WAL faster than a single core can process it, prefetching won't help. The bottleneck shifts from IO to CPU, and there's nowhere to go. A write-heavy primary with many concurrent backends can produce WAL at a rate that structurally outpaces what a single replay process can consume. The replica falls behind, and the gap only widens under sustained load. I've personally witnessed a replica where this process is pinned at 100% CPU for hours while replication lag continues to accumulate.This is especially painful in a multi-database instance. Every database's WAL goes through that same single-threaded funnel. A batch import into one database generates a torrent of WAL that delays replay of another database's critical transaction. On separate instances, each database has its own replica and independent replay process—no more cascading latency from a single busy database.

The Singleton Bottleneck Brigade

Beyond the big-ticket items, Postgres runs several background processes that are each a single worker serving the entire instance. Individually, they're rarely a problem. Collectively, they form a convoy of potential bottlenecks.Autovacuum gets a shared pool of workers, defaulting to a maximum of 3 (controlled by autovacuum_max_workers). The launcher process in schedules these workers across all databases in the instance. In an instance with ten databases and three workers, a couple of databases with heavy churn can monopolize the pool while others accumulate dead tuples and XID age. This kind of autovacuum starvation feeds directly into the XID and multixact wraparound risks discussed earlier.It's possible to raise of course, but those workers draw from the same CPU budget as application backends. How many workers will we need to accommodate all databases? It's not possible to assign workers to specific databases, so the problem never really goes away, it just becomes less likely. Separate instances would ensure that each database gets its own full complement of autovacuum workers without competing.The checkpointer is a single process responsible for flushing dirty buffers to disk at checkpoint intervals. A checkpoint triggered by one database's heavy write activity forces a flush of all dirty pages across the instance, including pages dirtied by other databases. The IO storm from a large checkpoint can cause latency spikes for every tenant, not just the one that triggered it.The background writer is also a single process that continuously writes dirty shared buffers to disk to keep a supply of clean pages available. It manages the entire shared buffer pool, and its pace is governed by instance-wide settings like and . There's no way to prioritize one database's dirty pages over another.

Splash Damage

Maybe the most straightforward argument against cramming everything into one instance is the blast radius of failure. When a Postgres instance goes down, whether from a crash, an OOM kill, a kernel panic, or just planned maintenance, every database on that instance goes with it.The postmaster treats many failure modes as potentially corrupting shared memory. A single backend crash triggers a full restart cycle and termination of all user sessions. This comment in the checkpointer code captures the philosophy:“If the checkpointer exits unexpectedly, the postmaster treats that the same as a backend crash: shared memory may be corrupted, so remaining backends should be killed."Maintenance windows compound the problem. A Postgres major version upgrade, an extension update, or even a configuration change requiring a restart affects all tenants simultaneously. Coordinating downtime across multiple teams with different SLAs, different peak hours, and different tolerance for interruption is an organizational headache that grows geometrically (or worse) with the number of databases.And then there's the dreaded emergency vacuum. If one database approaches XID wraparound, Postgres will refuse transactions for all databases (as we saw in ). An urgent maintenance task on one database is now a high-severity outage incident for everyone. The blast radius of a forgotten cron job or a stuck long-running transaction just expanded to encompass the entire data tier.

Splitting the Atom

The solution to most of these problems is, perhaps counter-intuitively, not beefier hardware but more instances. Take the same physical machine, carve it into virtual environments (VMs, containers, or even just multiple Postgres installations on different ports), and run one database per instance.What changes? Let's see...

Each instance gets its own
shared_buffers
, sized appropriately for its workload. An OLTP database can have a large, hot buffer pool while an analytics database gets a smaller one tuned for filesystem cache access. No more buffer thrashing between incompatible access patterns.

Transaction IDs become per-instance. One database's vacuum debt can't drag others into wraparound territory. The same applies to multixact members; that 21GB ceiling now applies to a single workload rather than the sum of all tenants.

WAL replay is per-instance. A write-heavy database generates WAL that only its own replica needs to replay. A latency-sensitive OLTP replica isn't waiting behind a batch import's WAL records destined for a completely different database.

Autovacuum workers, the checkpointer, and the background writer each serve a single database. No more starvation, no more shared checkpoint storms, no more one-size-fits-all background writer pacing.

Failures become isolated. A crash in one instance is invisible to the others. Maintenance windows can be scheduled independently. Emergency vacuums don't trigger cross-tenant incidents.

The trade-off is operational complexity. More instances means more configuration to manage, more backup schedules to maintain, more monitoring dashboards to watch. But with modern infrastructure tooling (Ansible, Terraform, Kubernetes operators), the marginal cost of an additional Postgres instance is low compared to the cost of debugging an emergency multi-tenant resource exhaustion event.

Knowing When to Quit

Vertical scaling is a perfectly valid strategy, and there's a reason so many Postgres installations run happily on a single large instance. For moderate workloads, the shared nature of Postgres internals is not just acceptable but efficient. Shared memory, shared processes, shared caches: they all reduce overhead when the workloads are playing nicely together.The trouble starts when "playing nicely" is no longer a given. Databases with fundamentally different I/O profiles, vacuum requirements, availability SLAs, activity patterns, and other concerns, don't always mix well. Resources become contested rather than efficient. No amount of RAM, CPU, or storage can counteract that because the constraints are architectural.The signals are usually subtle at first. Autovacuum can't keep up across all databases. Replica lag increases during batch jobs in an unrelated database. Checkpoint duration creeps up. Multixact warnings appear in the logs that nobody configured alerts for. By the time XID wraparound threatens to lock the whole instance, there have usually been many other signs that simply went unseen. There's a reason many in the community consider multiple-database instances a type of anti-pattern; shared resources are also a shared throttle.So if you're staring at a single Postgres instance that hosts a growing number of databases, or a shrinking number of exceptionally large ones, take a hard look at the shared internals. Read the source. Do the math on your multixact headroom. Check whether your autovacuum workers are keeping pace across every database, not just the ones you're watching. And if the numbers start looking uncomfortable, consider splitting before it becomes absolutely necessary.It's a lot easier to plan a migration than to execute one during an incident.

Enforcing Constraints Across Postgres Partitions

Fri, 17 Apr 2026 05:48:50 GMT

Postgres table partitioning is one of those features that feels like a superpower right up until it isn't. Just define a partition key, carve up data into manageable chunks, and everything hums along beautifully. And what's not to love? Partition pruning in query plans, smaller tables, faster maintenance, easy archiving of old data; it's a smorgasbord of convenience.Then you try to enforce a unique constraint without including the partition key, and Postgres behaves as if you just asked it to divide by zero. Well... about that.

The Rule Nobody Reads Until It's Too Late

The Postgres documentation on partitioning spells it out pretty clearly in the limitations section:Read that again. The constraint's columns must include all of the partition key columns. Not "should." Not "it would be nice if." Must. And the reasoning is maddeningly justified: each partition maintains its own index, and a local index can only enforce uniqueness within its own partition. Postgres has no concept of a global index that spans all partitions simultaneously, so it has no mechanism to check whether some value in partition A already exists in partition B.Other database engines (Oracle, for instance) have global indexes that solve this at the storage layer. Postgres does not, and there's been no serious movement on the mailing lists to add them. So we're left to our own devices.

When Theory Meets the Event Pipeline

Consider a fairly common (if somewhat contrived) scenario: an table partitioned by range on an identity column. The table includes a that the application uses to prevent duplicate event processing. Naturally, that should be unique across all partitions.Now try adding to that table without an error. The partition key is , and doesn't include it, so Postgres rejects the constraint. You could make a composite unique constraint on , but that's effectively useless for deduplication since is already unique. Every row would satisfy the constraint regardless of duplicate values.This is especially painful for date-range partitions, which are probably the most popular partitioning strategy in the wild. It's common to partition by month or week, but there's no universal uniqueness strategy there, just distinct intervals. The partition key is there for data management, not for data integrity, and Postgres can't separate those concepts without assistance.So what can we do to help it?

Brute Force to the Rescue

If Postgres won't enforce global unique constraints for us, can we do it ourselves? This is Postgres after all, so there are many tools at our disposal. Triggers, for example, exist for exactly this kind of scenario.The simplest approach is a trigger that scans the entire partitioned table set for duplicates:The index on is critical here; without it, every insert triggers a catastrophically slow sequential scan across all partitions. The index enables a much more optimal index scan across all partitions. This is still more overhead than ideal, but it could certainly be worse.Let's throw two million rows at it and see what happens:Without the trigger, this insert completes in about 7 seconds. With the trigger? 25 seconds. That's more than a 3x overhead, and we only have three partitions. The trigger must probe every partition's index for each row to confirm no duplicate exists. As the partition count grows, so does the probe time, because the query planner has to touch more and more partition indexes. Fifty partitions, a hundred partitions, three hundred partitions... each one adds another index lookup to every single insert.Does our duplicate check work, though? Absolutely:That's pretty satisfying, but performance at scale has much to be desired. Is that a problem we can solve?

Fake it 'till You Make it

What if instead of scanning the partitioned table, we maintained a separate, unpartitioned table whose sole job is tracking which values already exist? Then we can leverage the primary key of that table to do the uniqueness check for us, and Postgres handles all the heavy lifting with a single B-tree lookup.Now the trigger changes from a cross-partition scan to a simple insert into the (dedup) table. If the insert violates the primary key, Postgres immediately catches the duplicate. No scanning, no partition probing, no existential dread:But we also need to keep the dedup table honest. If rows get deleted from , the corresponding dedup entries should be cleaned up. Otherwise we'd reject future inserts for values that no longer exist:Now let's run the same two-million-row insert:This time the inserts completed in 14 seconds. Roughly double the baseline of 7 seconds without any trigger. This makes sense, as every insert now performs one additional B-tree insert into the dedup table. On the other hand, that overhead doesn’t change whether there are three partitions or three hundred. The dedup table is a single unpartitioned table with a single index. The cost is constant regardless of how many partitions exist on the table.That trade-off is fairly compelling: pay a fixed 2x overhead per insert in exchange for partition-count independence. For a system that might grow to dozens or hundreds of partitions over its lifetime, the dedup table approach is clearly the more sustainable choice.And yes, the dupe check still works:Even better, we get Postgres's own constraint violation message now, complete with the offending key value. No custom error formatting necessary.

Feedback Loop

There's a natural concern with this approach: the dedup table itself could grow enormous. If the table accumulates billions of rows over time, the dedup table will have billions of entries as well. That's a lot of B-tree to maintain within a monolithic index.The solution is almost comically recursive: partition the dedup table:Hash partitioning is well suited for this because it distributes values evenly and Postgres can prune directly to the correct partition for any given . The primary key constraint works here because, well, the partition key is the column we're constraining. No uniqueness problem on the uniqueness-enforcement table. Very convenient.The trigger code doesn't change at all. Postgres handles the partition routing transparently. We've effectively built a scalable uniqueness enforcement layer using nothing but declarative partitioning, a trigger function, and a table that exists purely as an optimization.

Going Virtual

What if the deduplication ID is something that we can derive from the row's content rather than storing in the table? That's worth yet another tweak to our approach. Say the uniqueness guarantee is based on the content. We could use a hash function to generate the dedup key on the fly:The function is one of those perilous corners of Postgres that doesn't show up in casual documentation browsing. It produces a 64-bit hash of the input text, which delivers a collision space of about 18 quintillion values. For most practical purposes, that's unique enough. For deduplication requirements that are truly ironclad, stick with a deterministic identifier rather than a hash.The beauty of this approach is that the table stays lean. The dedup table absorbs the storage cost, and since it's just a single column, it's remarkably compact.

The Fine Print

No solution is without caveats, and intellectual honesty demands we acknowledge a few.First, triggers add complexity. Every trigger is a piece of business logic that lives in the database rather than the application. Some teams are fine with that; others treat it like putting ketchup on a steak. Organizations with strong opinions about keeping logic in the application layer may balk at using a trigger in this situation regardless of its technical merits.Second, concurrency deserves attention. Both trigger approaches rely on seeing committed data (or data visible within the current transaction). Under high concurrency, two transactions could simultaneously check for the same , both find nothing, and both proceed with the insert.The dedup table approach handles this more gracefully because the will block on the primary key's underlying unique index, causing the second transaction to wait. Once the first commits, the second will get the constraint violation.The scan-based trigger is more vulnerable here because might not see uncommitted rows from other transactions. The trigger would need to use more sophisticated logic including advisory locks to prevent that.Finally, there's the matter of updates. If it's possible to update , there should be additional trigger logic to handle that use case. This is territory where implementation details tied to business logic proliferate and escape the simple confines of our demonstration case. But you get the idea.

Closing Thoughts

So where does this leave us? The documentation is refreshingly honest about the constraint limitation, but honesty doesn't enforce uniqueness across partitions.The brute force trigger works without much fanfare, but it scales poorly with partition count. The dedup table approach trades a small, constant overhead for partition-count independence, which is almost always the right trade-off in production systems. And for those feeling adventurous, the algorithmic variant omits the extra column entirely by computing dedup keys on the fly.Sadly, none of these techniques are as clean as a native constraint. They all involve triggers, which means more moving parts, more things to test, and more things to document. But they work. Sometimes the best engineering isn't about finding the perfect solution, it's about finding the most tolerable compromise.Keep these tricks in your back pocket. Partitioning is still the best approach for horizontally scaling large data sets, and the uniqueness problem isn't going anywhere until Postgres adds global indexes. Until then, triggers and dedup tables will do just fine.

Checkpoints, Write Storms, and You

Fri, 10 Apr 2026 06:06:37 GMT

Every database has to reconcile two uncomfortable truths: memory is fast but volatile, and disk is slow but durable. Postgres handles this tension through its Write-Ahead Log (WAL), which records every change before it happens. But the WAL can't grow forever. At some point, Postgres needs to flush all those accumulated dirty pages to disk and declare a clean starting point. That process is called a checkpoint, and when it goes wrong, it can bring throughput to its knees.

A Bit About Checkpoints

Under normal operation, Postgres is remarkably polite about checkpoints. The parameter (default 5 minutes) tells Postgres how often to perform a scheduled checkpoint, and (default 0.9) tells it to spread the resulting writes over 90% of that interval. So a checkpoint timeout of 5 minutes means Postgres trickles dirty pages to disk over roughly 4.5 minutes, keeping IO impact to a minimum.This only applies to timed checkpoint behavior.The parameter sets a soft limit on how much WAL can accumulate between checkpoints. When the WAL approaches that threshold (1GB by default), Postgres doesn't wait for the next scheduled checkpoint. Instead, it forces one immediately.These forced (or requested) checkpoints do not honor . Postgres needs to reclaim WAL space, so it flushes every dirty buffer to disk as fast as the IO subsystem will allow. On a busy system with a large pool full of modified pages, this can completely saturate disk IO in seconds.It's like trying to drink from a firehose.

Rubber Meets the Road

To see this in action, we set up a modest test environment:

Hypervisor:
Proxmox

CPU:
4x AMD EPYC 9454 cores

RAM:
4GB

DB Storage:
100GB @ 2,000 IOPS

WAL Storage:
100GB @ 2,000 IOPS

OS:
Debian 12 Bookworm

We initialized the database with at a scale factor of 800, producing roughly 12GB of data (3x available RAM to reduce cache hits). We also followed the traditional advice of setting to 25% of RAM, or 1GB in this case. All other settings remained at their defaults.Each test followed the same pattern: issue a manual to start clean, then run pgbench for 60 seconds with per-second progress reporting and 16 concurrent clients to keep all of the CPU cores busy:We started with the default of 1GB to see how the system behaves. This setting is frequently overlooked during optimization, so it should illustrate a good example of baseline operation.Throughput holds steady between 1,000 and 1,100 TPS for the first 41 seconds of the test. Buffers began to warm, the IO subsystem was keeping pace, and latency remained low. At the 42-second mark, WAL output reached 1GB and Postgres forced a checkpoint. TPS immediately cratered to roughly 620—a drop of nearly 40%! It never recovered for the remainder of the benchmark run.We increased to 4GB for the second test. It's a modest bump, but should be sufficient for the purposes of this demonstration. Throughput started around 1,000 TPS this time around and gradually climbed as shared buffers warmed up, reaching 1,200 TPS by the end of the test. One minute of pgbench activity isn’t enough to produce 4GB of WAL on this hardware, meaning no forced checkpoint.The results basically speak for themselves:Ouch! Both tests tracked nearly identically for the first 40 seconds. Then the 1GB configuration hit a wall while the 4GB configuration kept climbing.

The Cost of Forced Checkpoints

Postgres normally uses during a timed checkpoint to calculate a write budget. If it has 5 minutes between checkpoints and the target is 0.9, it can spread dirty page writes over 270 seconds. That's a lot of time to trickle data to disk, and the IO impact per second is minimal.A forced checkpoint has no such luxury. The WAL is full (or nearly so), and Postgres needs to reclaim space now. It writes dirty buffers as fast as it can, competing directly with active queries for disk IO. This competition is fierce on a system limited to 2,000 IOPS. Every IOPS spent flushing checkpoint data is essentially stolen from user queries.The severity is largely hardware dependent. Systems with fast NVMe storage and tens (or hundreds) of thousands of IOPS may barely notice. But cloud instances, virtualized environments, or anything with IO throttling (which is extremely common) will feel the pain. We provisioned our test system at 2,000 IOPS per volume, which is relatively generous by cloud standards, and still experienced a marked impact.The benchmark itself is only half of the story. Prior to that, we had to initialize the 12GB test database with . While pgbench generated the sample data with the default 1GB , Postgres triggered 18 forced checkpoints. Trying again with set to 20GB brought that number to zero.So what? It's just initialization, right? Consider that this same pattern applies to any bulk data operation: imports, large statements, on big tables, , or even hefty batches. If any of these operations are running alongside a production OLTP workload, that's 18 IO storms competing with application queries.An ETL job that loads a few gigabytes of data every night could trigger a string of forced checkpoints that spike latency for every other query on the system. The bulk operation itself will also slow down since it's fighting its own checkpoint IO for disk bandwidth.Everyone loses when checkpoints can't spread write activity.

Spotting the Problem

Postgres tracks checkpoint statistics, and checking them should be part of any regular health assessment. The system catalog you should use depends on the Postgres version.In Postgres 17 and later, use pg_stat_checkpointer:In older versions, the same information lives in :The critical ratio here is: timed versus requested checkpoints. In a well-tuned system, (or ) should be close to zero relative to . If requested checkpoints are a significant proportion of the total, is too small for the current write workload and performance is likely sub-optimal.It's also worth keeping an eye on and . If is consistently high, the storage subsystem is struggling to keep up with checkpoint flushes, which further confirms an IO bottleneck during checkpoints.As for logging, we highly recommend setting log_checkpoints to to capture checkpoint activity:This causes Postgres to log detailed information about every checkpoint, such as the number of buffers written, how long it took including sync time, and other useful metrics. When enabled, the Postgres log should show checkpoint activity like this:That line is the smoking gun. It means this checkpoint was forced because WAL hit the limit, not because the timeout expired. A timed checkpoint would say instead.This is free forensic information. The logging overhead is negligible and provides a clear trail of checkpoint behavior. This is another one of those settings that should be enabled by default, and has been since Postgres 15. Those with older clusters will have to enable it manually.

Finding the Right max_wal_size

So should everyone just crank to some enormous value and forget about it? Not exactly. There are trade-offs.A larger allows more WAL to accumulate between checkpoints, which means more data must be replayed during crash recovery. If Postgres crashes with 20GB of WAL to replay, startup necessarily takes longer than it would with 1GB. This difference in recovery time is usually only a matter of seconds, but it's worth acknowledging.The other consideration is disk space. WAL files consume storage, and is a soft limit. Under heavy write loads, WAL can temporarily exceed it. There should be enough headroom on the WAL volume to accommodate bursts without running out of space entirely. That would be a much worse problem than slow checkpoints.A reasonable starting point for write-heavy OLTP workloads is 10GB to 20GB. Systems with aggressive bulk loading or large batch operations might benefit from 50GB or more. The goal is to make forced checkpoints rare enough that essentially all checkpoints are timed and spread gracefully over .We recommend validating the setting by monitoring (or ) over time. Let the system run under typical load for a day or a week, then check the ratio. If requested checkpoints have increased, bump higher and repeat.If you'd rather not do this yourself, I actually wrote a Postgres extension called pg_walsizer. It launches a background worker that monitors checkpoint activity and automatically increases based on how many checkpoints occur within the configured . Just set it and forget it!

Wrapping Up

Checkpoints are one of those Postgres internals that most people never think about until something goes wrong. Periodic latency spikes can have any number of causes after all. Not all DBAs consider checking WAL activity, nor realize the relationship it has with disk flushes—most people blame vacuum at first.As is tradition, the default value of 1GB for is a conservative one. It minimizes crash recovery time and works fine for light workloads without using a lot of storage. Unfortunately, busy systems will quickly exceed this default and begin to suffer. Our test showed a 40% TPS plunge on modest hardware; production systems with heavier loads and tighter IO budgets will likely fare worse.For most production environments, we suggest starting with something more appropriate. If isn't already enabled, prioritize enabling that as well. And finally, either or should feature prominently in a monitoring dashboard. Look upon any increases in requested checkpoints with suspicion.In the end, is one of those rare occasions where a single parameter can confer a substantial improvement with virtually no downside. So go check your checkpoint stats. You might be surprised by what you find!

What is a Collation, and Why is My Data Corrupt

Fri, 03 Apr 2026 05:36:03 GMT

The GNU C Library (glibc) version 2.28 entered the world on August 1st, 2018 and Postgres hasn't been the same since. Among its many changes was a massive update to locale collation data, bringing it in line with the 2016 Edition 4 release of the ISO 14651 standard and Unicode 9.0.0. This was not a subtle tweak. It was the culmination of roughly 18 years of accumulated locale modifications, all merged in a single release.Nobody threw a party.What followed was one of the most significant and insidious data integrity incidents in the history of Postgres. Indexes silently became corrupt, query results changed without warning, and unique constraints were no longer trustworthy. The worst part? You had to know to look for it. Postgres didn't complain. The operating system didn't complain. Everything appeared normal, right up until it wasn't.This is the story of how a library upgrade quietly corrupted databases around the world, what the Postgres community did about it, and how to make sure it never happens to you again.

What even is a Collation?

Before we can understand what broke, we need to understand what a collation actually does. At its core, a collation defines how text is compared and sorted. That sounds simple enough, but collation rules become much more turbulent outside of the English alphabet.Consider the German letter ß. Does it sort the same as "ss"? Usually. What about accented characters like é and è? Should they be treated as equivalent to "e" for sorting purposes, or should they have their own distinct positions? What about the Swedish alphabet, where ä and ö come after z rather than being treated as variants of a and o?Every language has its own answer to these questions, and a collation encodes those answers into a set of rules for a database to follow. When Postgres needs to sort a column of text, enforce a unique constraint, or build a B-tree index, it asks the collation: "Which of these two strings comes first?" The collation's answer determines everything from query results to whether an index lookup finds any data at all.Historically, Postgres delegated that question to the operating system's C library. Postgres doesn't have its own implementation of American English sorting rules baked in, so databases created with the locale rely on external libraries. More specifically, Postgres used to simply invoke from glibc (on Linux systems) and trusted whatever answer came back.That trust worked fine for years. And then one day it didn't.

The Day the World Changed

So what actually changed in glibc 2.28? Consider a simple example using the locale. Before the update, strings containing special characters sorted like this:After glibc 2.28, those same strings sorted like this:That's not a minor adjustment. The relative positions of strings containing punctuation, mixed case, and special characters shifted dramatically. Another well documented example: the sort order of and simply flipped between the old and new versions. Data containing strings with hyphens, underscores, or currency symbols now experienced seemingly inconsistent sorting rules.This wasn't a bug in glibc. The glibc developers were correcting years of accumulated divergence from the Unicode standard. The new sort orders were arguably more correct. But correctness is cold comfort for indexes founded upon faulty assumptions.

Anatomy of a Silent Catastrophe

Why did this cause so much trouble, though? Let's examine how Postgres B-tree indexes work with text. When Postgres builds an index on a text column, it sorts the values according to the active collation and stores them in that order. Postgres builds the literal on-disk tree structure based on these results. Later, Postgres navigates the B-tree by comparing search terms against stored keys, following the tree left or right based on which string "comes first" according to the collation.Imagine the library underneath changes its mind about which string comes first. The physical layout of the index reflects the old sort order, but every new comparison uses the new sort order. Postgres navigates right when it should go left, or left when it should go right, and previously valid data becomes invisible. The row is still in the table, a sequential scan still finds it, but the index lookup misses it entirely.The consequences cascaded outward from there:

Queries using index scans could silently skip existing rows. A
might return 999 rows when 1,000 actually matched the predicate.

Unique constraints backed by B-tree indexes could fail to detect actual duplicates because the index traversal couldn't find the existing entry. Alternatively, they might wrongly reject valid entries because the traversal landed on the wrong node.

Any query relying on
with a text column would produce different results before and after the upgrade. Merge joins, which depend on both inputs being sorted identically, could produce silently incorrect results.

If a primary ran one glibc version and a replica ran another, identical queries against identical data produced different results. The
demonstrated this exact scenario with streaming replicas, though few understood the full implications at the time.

All of this happened silently. Nobody knew (or could know) anything was wrong until it was already too late.

The Domino Effect

The glibc 2.28 release didn't hit every Linux distribution simultaneously. Instead, it rippled outward over the course of about a year as each distribution adopted it on their own schedule:

Fedora 29 and Ubuntu 18.10 shipped with glibc 2.28.

Christoph Berg, the Debian PostgreSQL maintainer, raised the alarm on the
, describing the situation as critical. He proposed automated warnings for Debian users with PostgreSQL clusters.

RHEL 8 and CentOS 8 arrived with glibc 2.28, making the leap from glibc 2.17 in RHEL 7. That's an 11-version jump in a single upgrade cycle.

Debian 10 (Buster) followed suit.

The RHEL jump was particularly brutal. Many enterprise shops run on CentOS or RHEL, and an OS upgrade from version 7 to 8 was just a common eventuality. Nobody expected that a routine distribution upgrade would quietly corrupt their database indexes. Arch Linux users, running the bleeding edge as always, were among the first canaries in the coal mine.Daniel Verite published "Beware of your next glibc upgrade" in August 2018, one of the earliest public warnings. The Postgres Wiki created dedicated pages for Locale data changes and Collations to track the evolving situation. Blog posts from Crunchy Data, Citus Data, and CYBERTEC followed, each emphasizing the same uncomfortable truth: if you upgraded glibc and didn't rebuild your indexes, your data might already be corrupt.

Finding the Damage

The first step for any affected system was identifying at-risk indexes. Any B-tree index on a text, varchar, char, or citext column using a locale-dependent collation (anything other than or ) was potentially corrupt. The Postgres community settled on a diagnostic query that looked something like this:Every index returned by that query needed to be rebuilt. For Postgres 12 and later, that meant:The option proved to be a lifesaver for production systems, as it allows the rebuild to happen without locking the table for the entire rebuild duration. For those stuck on Postgres 11 or earlier, the workaround was uglier: create a replacement index concurrently, drop the old one, and rename the new one. Primary key indexes and unique constraints made this particularly painful.The scale of the problem was staggering. A database with hundreds of tables and thousands of text indexes needed every single one rebuilt. And this wasn't a one-time fix. Any future glibc upgrade that changed collation rules would require a repeat performance.

Déjà vu

Ironically, the glibc 2.28 incident wasn't even the first time glibc caused index corruption in Postgres. Several glibc versions in 2015 shipped with a buggy implementation that produced results inconsistent with , violating both ISO C90 and POSIX standards.Postgres 9.5 had introduced "abbreviated keys" to speed up text index builds, and the glibc bugs caused these to produce corrupt indexes. The fix in Postgres 9.5.2 was to disable abbreviated keys for non-C locales entirely, a performance regression that persists to this day for libc-based collations. Users had to then, too.Two major incidents in three years, both caused by the same fundamental problem: Postgres trusted an external library for a critical operation, and that library's behavior was neither stable nor guaranteed. The writing was on the wall.

The Long March Away from glibc

The Postgres community's response to these incidents was measured but determined, playing out over nearly a decade of incremental progress. introduced initial support for ICU (International Components for Unicode) as an alternative collation provider. Peter Eisentraut's work here was prescient, landing a full year before glibc 2.28 shipped. For the first time, it was possible to create a collation backed by ICU instead of libc:ICU maintains its own collation data independent of the operating system. ICU updates its rules through a strict versioning system, meaning Postgres can detect the change and emit a warning. added collation version tracking for glibc. Postgres began recording the collation version during index creation, and issuing warnings when the underlying version changed. This was the first real "early warning system" for the problem. It couldn't prevent corruption, but at least the logs told the full story. was an event many anticipated: ICU could now be used as the default collation provider for an entire database cluster:Before this, ICU was only available for individual collation objects, which was both inconvenient and prone to mistakes. delivered what many consider the real solution: the builtin collation provider. This provider compiles collation logic directly into Postgres itself, with no external glibc or ICU dependency. The builtin provider ships with two primary collations:

: Unicode code point sorting with POSIX-compatible pattern matching and simple case mapping.

: Unicode code point sorting with full Unicode case mapping and standard pattern matching behavior.

Both are guaranteed to be immutable within a major Postgres version. The entire class of "the OS changed my sort order" bugs simply cannot happen with these collations.

The Right Way to Initialize a Cluster (Today)

Despite all of this history, as of Postgres 18. Newly initialized clusters will use glibc libraries unless told otherwise. That means every new database is potentially vulnerable to the same class of problem that glibc 2.28 caused, just waiting for the next major library update to trigger it.As a result, I recommend always specifying the builtin provider when creating a new cluster:Two flags to eliminate an entire category of data corruption risk. The locale provides proper UTF-8 character handling while sorting by Unicode code point order without any lingering surprises.For those creating databases within an existing cluster, the same principle applies:It’s necessary to specify as the template database when doing this, as it’s normally impossible to use a collation different from the source database.Until the Postgres project changes the default (and there are ongoing discussions about exactly that), every DBA needs to make this a conscious choice for every new cluster or database.

Never a Free Lunch

If the new internally-provided collations are so great, why isn't everyone using them?The first reason is that few stop to consider the topic at all. They may trust that the problem is being handled and addressed by future versions, and it'll happen as if by magic. Nobody wants to face the ugly reality of a data migration. Perhaps they’re simply new users who missed the uproar.The other reason is more subtle. Both and sort by Unicode code point value. This is essentially a byte-order sort for UTF-8 encoded text. It's deterministic, fast, and perfect for indexes. But it doesn't match what a human would expect for linguistically correct sorting in most languages.Consider German names:Yet a native German speaker would expect this instead:For most application workloads, this doesn't actually matter. APIs return JSON, frontends sort data client-side, and search operations care about matching rather than ordering. But linguistic sort order is still relevant for applications that display sorted lists directly to users, such as a directory, a catalog, a report, and so on.

Applying Linguistic Sorting

This is where ICU earns its keep. Clusters can run on the builtin provider for safety and performance, and then apply ICU collations precisely where necessary. There are two approaches. is ideal when a particular column always needs linguistic sorting:Now the column always sorts according to German linguistic rules, while every other text column in the database uses the safe, deterministic builtin collation. Indexes on the column will use the ICU collation, so becomes dependent on ICU's versioning for that specific index. ICU tends to exercise more discipline about version management than glibc, so this isn't quite as risky. is better for occasional linguistic sorting:This applies the German ICU collation just for this specific sort operation. The underlying column and its indexes remain on the builtin collation. This results in linguistic sorting without altering the storage or index behavior. This is the safest approach as a result, though admittedly more inconvenient due to the additional syntax.

The Ghost of Clusters Past

New clusters are easy. Specify the provider at init time and move on with your life. But what about the millions of existing Postgres clusters already running on libc collations? Those don't just go away, and it's not possible to change a database's default collation after creation. That said, there are a few options:

Use
and
or logical replication to move data into a fresh cluster initialized with the
provider. This is the cleanest approach, but requires planning and potential downtime.

Individual columns can use a different collation within a libc-based cluster. Perform a staged rollout by moving tables or affected columns to a safer collation, and always create new columns with that collation. A future cluster migration is still necessary for a permanent fix, but this process provides a path to safety.

Postgres 13+ will log warnings when a collation's underlying version changes. Pay attention to those warnings; they're screaming that a REINDEX is necessary. Look for these messages:

The safest approach for glibc-based clusters is to rebuild every text index following a major distribution upgrade. No exceptions. Use the diagnostic query from earlier to identify affected indexes, and
for identified candidates.

Moving On

The glibc 2.28 incident changed how the entire Postgres community thinks about external dependencies. Before 2018, the idea that an OS library update could result in database corruption was something only a handful of people worried about. That it did so silently—allowing the corruption to fester for weeks, months, or even years—just poured extra salt in the wound.The Postgres community responded in its typical heroic fashion. Collation version tracking, ICU provider support, and ultimately collations, show just how far the Postgres devs are willing to go to solve a problem. It's not a complete rebellion against trusting OS-provided libraries, but decoupling from external collation resources remains a prudent reaction given the circumstances.And yet we shouldn't rest on our laurels; glibc remains the default even now, suggesting the lesson hasn't fully sunk in for everyone. Every run without is another cluster carrying the same risk that nearly spelled disaster for many. I've personally encountered clusters with this kind of corruption as recently as 2025, fully seven years since everything went wrong. Why propagate that mistake?So the next time you spin up a Postgres cluster, do yourself a favor and use the builtin locale provider. Your future self, the one who just upgraded to the latest Ubuntu LTS without thinking twice about it, will thank you.

PG Phriday: Absorbing the Load

Fri, 27 Mar 2026 06:53:43 GMT

Recently on the pgsql-performance mailing list, a question popped up regarding a Top-N query gone wrong. On the surface, the query merely fetched the latest 1000 rows through a join involving a few CTEs in a dozen tables with a few million rows distributed among them. A daunting query with an equally daunting plan that required about 2-3 seconds of execution time. Not ideal, but not exactly a show-stopper either. But high concurrency compounded the problem, with 40 sessions enough to completely saturate the available CPU threads.In the end all it took was a new index to solve the conundrum, as is usually the case with queries like this. Problem solved, right?Well... maybe. But let's play a different game.What if the query was already optimal? What if every join was fully indexed, the planner chose the most optimal variant, and yet the sheer volume of data meant two seconds was the fastest possible time? Now imagine not 40 concurrent users, but 4000. What happens then?

It's Just Math

Suppose a query takes two seconds after being fully optimized and there are 4000 users that each need that result once per page load, spread across a 60-second window. That's roughly 67 concurrent executions at any given moment. A 2-vCPU machine can only handle two of these, what about the other 65?That's an irreducible capacity mismatch. If we can't reduce the execution time of the original query, we must provide some kind of substitution instead, one that reduces the number of times Postgres runs that query at all.The good news is that Postgres has built-in tools for this. The better news is that there are external tools that extend those capabilities substantially. Sometimes—though a Postgres expert is loathe to admit it—the right answer rests outside of Postgres.Let's explore the available tools.

It's a Material World

Postgres has supported materialized views since version 9.3, and they are exactly what they sound like: a view whose results are physically stored on disk, rather than recomputed on demand. Creating one is straightforward:Now instead of hammering the source tables on every request, thousands of concurrent users query a single pre-materialized result set. The read path becomes trivially fast. You can even add indexes to the materialized view itself:The catch is that the data is frozen at the moment of last refresh. To update it, you run:By default, this takes an AccessExclusiveLock on the view during the refresh, meaning all reads are blocked until the operation completes. It's possible to refresh a materialized view concurrently, but this requires a unique index and is considerably slower:It's typical to schedule such refreshes using a tool like pg_cron or an external scheduler:What do we actually get here? Zero external dependencies, native Postgres, indexable, and queries like any other table. A multi-second query transforms into a sub-millisecond scan to clients. Unfortunately, the data is stale between refreshes, the concurrent refresh is slower and requires a unique index, a full refresh blocks all readers, and the refresh time scales with the base table size rather than the size of what actually changed. It's a blunt instrument, but an effective one.Materialized views excel most when you can tolerate stale data on a predictable schedule. Think dashboards, reporting aggregates, and leaderboards where "as of 5 minutes ago" is perfectly acceptable. For real-time transactional Top-N results? Less so.Can we improve on this?

Incrementally Yours

The pg_ivm extension addresses the core weakness of standard materialized views: the full recomputation on every refresh. IVM stands for Incremental View Maintenance, and the premise is straightforward. When a base table changes, only apply the delta to the materialized view, rather than rebuilding the entire thing from scratch.The performance difference is striking; a full refresh of a large materialized view might take 20 seconds. An incremental update after a single-row insert may only require a few milliseconds. It's not uncommon to experience one or more orders of magnitude of improvement.Creating an incrementally maintained materialized view (IMMV) looks like this:Behind the scenes, attaches triggers to every base table referenced in the view definition. When a row is inserted, updated, or deleted in , , or , those triggers fire and update in the same transaction. The view is always current without any need for a job scheduler or update process.Beautiful. But there are some cracks in the facade.Trigger overhead is real. Every write to every base table now carries a few microseconds of extra work. If those tables receive thousands of writes per second, every single one of those must update the IMMV. We've essentially traded faster reads for slightly slower writes. Whether that trade is favorable depends entirely on the database's read-to-write ratio and write latency thresholds.There's also the matter of query restrictions. does not support , , , window functions, , or subqueries in the view definition. Notice that the from the original view definition is missing and left as an exercise on the read side. Base tables must be plain tables too, not views, partitions, or foreign tables. And finally, data types need to support btree indexing, which rules out JSON, XML, and similar types.The extension is best suited for aggregate-style views (sums, counts, averages) where the base data is relatively stable. For the specific Top-N use case from the mailing list, these limitations narrow its applicable scope considerably. But if your scenario fits within those constraints, the always-fresh guarantee is hard to ignore.What other options are there?

Out of the Loop

Neither materialized views nor pg_ivm fully address highly dynamic real-time query results accessed by thousands of concurrent readers. The real insight is this:The TTL of your cache only needs to be greater than or equal to the query execution time.Think about that for a moment. A query with a two-second run-time has a minimum resolution of two seconds. By widening that window slightly to five seconds, cache refreshes can absorb slow executions while still remaining current. Every concurrent request in between just reads the cached result instantly, condensing thousands of database roundtrips into one.This is the Shared Cache pattern, and it works at the application layer. The key design decision is to separate the writing of the cache from the reading of it. Rather than having client requests trigger the expensive query on a cache miss, you dedicate a background worker whose entire job is to keep the cache warm. Clients never touch the database directly, they only read from the cache.Here's what something like that may look like in Python:This separation is what makes the pattern so effective. The background worker runs on its own schedule, executing the expensive query once per TTL cycle regardless of whether 1 user or 4000 users are waiting for results. The clients are completely decoupled from the database. There's no risk of a sudden surge of users all triggering the same expensive query at once, because clients never trigger it in the first place. The worker owns the query. The clients own the read. Those two concerns never intersect.The dance is entirely agnostic to what Postgres is doing. No schema changes, no extensions, no triggers. And with a TTL measured in seconds rather than minutes, the data is effectively real-time from a user's perspective. Unless it's a high-frequency trading platform, few users will notice data that's a few seconds stale.The obvious question is: where does this cache live? If it lives in-process memory, each application server maintains its own cache, and you still get N database queries for N application servers. Which brings us to the next stop on our tour.

The Shared Nothing Problem

Redis (and variants like KeyDB) is the most common answer to the shared cache problem in production environments. It's fast, simple, and provides a single cache shared by all application servers. That last part is important because it instantly solves the multi-instance cache coherence problem.The same background worker pattern applies, but now the worker writes to Redis instead of a local dictionary, and every application server reads from the same shared store.Let's see that with some more pseudo-python:Redis handles the TTL natively via , so even if the worker hiccups, stale entries expire on their own. The result: one database round-trip per refresh cycle, regardless of how many application servers or concurrent users there are. The worker is the only thing that ever talks to Postgres for these queries, and it does so at a calm, predictable cadence.There are some things worth considering though. JSON serialization of large result sets has a cost. For very large payloads, MessagePack or Protocol Buffers will do the job with less overhead. Cache granularity matters too; if every user has a highly unique set of filter predicates, cache hit rate plummets and essentially negates any previous benefit.Then of course there's cache invalidation. We can actually ignore that for the purposes of this discussion. Why? It's often better to have some data than no data. If the worker hasn't refreshed the cache, either the database is unreachable, or the refresh took longer than expected. In that event, the front end can display the existing cache with a tooltip or some other disclaimer disclosing the information may be slightly stale. While not ideal, the alternative is redirecting traffic back to the database, inevitably overwhelming its resources and resulting in no data for anyone.Tools like Memcached deserve a brief mention as well. It's a reasonable alternative when you don't need persistence or data structures beyond simple key-value storage. It's slightly faster at raw cache operations but lacks many of Redis's operational niceties. It does the job in a pinch.

Pushing It to the Edge

If your Top-N query results are not user-specific (think a public leaderboard, a global trending list, or a site-wide feed) then there's one more powerful option that lives even further from Postgres: a Content Delivery Network (CDN).CDNs like Cloudflare, Fastly, and AWS CloudFront can cache HTTP responses at edge nodes geographically close to your users. A CDN will cache the partial or fully-rendered results from a properly configured API endpoint with an appropriate HTTP header:This instructs the CDN to serve the cached response for 5 seconds, and then for a further 10 seconds serve the stale result while asynchronously revalidating in the background. Your origin server, and by extension Postgres, sees one request per cache expiration cycle per CDN edge node. That could collapse millions of user requests into dozens of origin requests for global deployments.The CDN approach follows the same philosophy as the background worker pattern, just at a different layer. The CDN itself handles revalidation, fetching a fresh copy from the origin on its own schedule rather than letting every user request pass through. It's elegant, requires zero application code changes for HTTP-based APIs, and the cost is typically a fraction of your dedicated compute resources.The limitation, of course, is that CDN caching only applies to public data. Anything that varies by authenticated user (personalized feeds, account-specific results, user-filtered Top-N queries) can't be cached at the CDN layer without significant complexity involving headers, cache partitioning, or edge authentication. For those cases, intermediate caches like Redis remain a simpler solution.There's also Varnish, which deserves a mention for teams that want CDN-like caching on self-hosted infrastructure. Varnish sits in front of an application server and caches responses based on configurable rules. It's extremely fast, delivers fine-grained control over cache behavior, and avoids CDN cost overhead.That makes three different tiers of caching!

Know Your Terrain

There's no universal answer, but the decision isn't all that complicated:In practice, most high-scale systems use multiple layers simultaneously. Postgres materialized views pre-aggregate data that rarely changes. Redis caches user-specific query results with short TTLs. A CDN handles public-facing endpoints. Each layer absorbs load that doesn't need to reach the next one, and Postgres ends up only seeing the requests that genuinely require fresh database access.

Sometimes the Question Isn't the Question

Ostensibly, Yudhi asked about query tuning. Most of the early replies focused on missing indexes, and they were ultimately proven correct. A well-placed moderately improved that specific query. But that's not how every story ends.What about the other questions Yudhi asked? The ones about thousands of concurrent users, CPU spikes, and capacity planning? Those are really architectural questions in query-tuning suits. Answering those questions evades any amount of database or query performance tuning.When you've done everything right in Postgres (optimal queries, correct indexes, well-chosen configuration) and you still have more demand than your instance can absorb, the solution isn't to make the query faster. It's to make the query happen less often. Materialized views bestow that capability natively. The pg_ivm extension adds a few more bells and whistles in exchange for minor write throughput. Redis brings a shared cache that spans the entire application fleet. CDNs push the cache to the edge of the internet itself.The magic, if we can call it that, is the layering. Each level absorbs the load that doesn't need to penetrate further. By the time a request actually reaches Postgres and runs a query, it genuinely needs to. Everything else was handled or deflected by an upstream layer.I suspect most performance problems that look unsolvable at the query level are really just missing a layer. People don't usually think like databases, and really, who can blame them? But the next time you find yourself staring at a perfectly optimized query that still can't keep up with demand, consider: you may not have a query problem, but an architectural one. And the good news is that the solutions are well-understood, battle-tested, and surprisingly straightforward to implement.Now go add that index. Then think about the cache. Your database will thank you.

Using Patroni to Build a Highly Available Postgres Cluster—Part 3: HAProxy

Fri, 20 Mar 2026 06:47:06 GMT

Welcome to Part three of our series for building a High Availability Postgres cluster using Patroni! Part one focused entirely on establishing the DCS using etcd to provide the critical DCS backbone for the cluster, and part two added Patroni and Postgres to the software stack. While it's entirely possible to stop at that point and use the cluster as-is, there's one more piece that will make it far more functional overall.New connections need a way to reach the primary node easily and consistently. Patroni provides a REST interface for interrogating each node for its status, making it a perfect match for any software or load-balancer layer compatible with HTTP checks. Part three focuses on adding HAProxy to fill that role, completing the cluster with a routing layer.Hopefully you still have the three VMs where you installed etcd, Postgres, and Patroni. We will need those VMs for the final stage, so if you haven't already gone through the steps in part one and two, come back when you're ready.Otherwise, let's complete the cluster!

What HAProxy adds

HAProxy is one of the most common HTTP proxies available, but it also has a hidden superpower: it can transparently redirect raw TCP connections as well. This means it can also act as a proxy for any kind of service such as Postgres. Here's how it works:

HAProxy connects to the Patroni REST interface and gets the status for the "/" URL.

Patroni will only respond with a "200 OK" status on the primary node. All other nodes will produce a "500" error of some kind.

HAProxy marks nodes that respond with errors as unhealthy.

All connections get routed to the only "healthy" node: the primary for the cluster.

Of course that's not the end of it; the Patroni REST API is incredibly powerful, as it provides multiple additional endpoints. For example a check against:

/replica will succeed if the node is a healthy streaming replica of the primary, a good match for offloading intensive read queries from the primary node.

/read-only works on any healthy node in the cluster—perfect for connections that don't care how they interact with the database.

/synchronous only succeeds on healthy synchronous streaming replicas, for operations where it's important for read durability.

There's also an HTTP parameter (lag) that limits success on replicas to a specified maximum. Want to target only replicas with less than 1MB of replication latency? Simply add that parameter to the HTTP check operation in HAProxy. This enables creation of multiple proxy definitions for each dedicated requirement, and HAProxy maintains everything automatically based on Patroni status codes—no manual intervention necessary.

Installing HAProxy

Thankfully installing HAProxy is pretty easy because it's so ubiquitous. It should be available in the default repositories of every major Linux platform without any special steps. In the case of Debian, one command should do it:

Building a useful HAProxy configuration

Unlike Patroni, configuring HAProxy is a much simpler affair. The default configuration file should already exist in the directory following installation. Since the defaults depend entirely on the version of HAProxy and the target distribution, let's replace it with something that should work everywhere and specific to the Patroni cluster being built.Begin with the preamble of all the service defaults. There are globally applied settings to all defined listeners, and default values for standard operation. For example:In this case, HAProxy will stop allowing connections after 100—a perfect amount for a demo cluster, but you may want to increase it in a production system.The remainder of parameters define log output, set the connection type to TCP rather than HTTP, ensure checks are repeated twice to prevent false positives, and establish a few basic timeouts to avoid stale connections or server states.The next step is to define a block. This is what actually binds a port to a group of servers and defines the health check against Patroni's REST API. Based on the VMs we've built so far, it should look something like this:The Postgres service port of 5432 is already in use in the VMs we created, so the proxy must use a different port. It's probably not necessary to explicitly set the mode to TCP again in the listen block, but is a good practice just in case. Checks can be either TCP or HTTP, and despite the fact connections should be TCP in nature, the health check itself is HTTP thanks to Patroni's convenient REST interface. And of course we only want to acknowledge a 200 status as success.The line has a special importance here. Translated, it means:

Perform the health check every three seconds.

Require three failures before marking the host as down.

Set a failed host as healthy after two successful checks.

When a node is marked down, disconnect any established sessions.

This is necessary because HAProxy has many operation modes, many of which are permissive or otherwise optimistic. Just because new connections shouldn't be sent to a new server doesn't mean old ones are suddenly invalid. In the case of Patroni however, it means exactly that!If Patroni fails, the REST interface also disappears, and HAProxy will interpret that as a node failure. But in the event Patroni fails before it can properly stop Postgres, Postgres will remain online, possibly accepting writes in the case of a primary node. We need to terminate all connections just in case to prevent split-brain scenarios.This is one situation where simply relying on Postgres connection protocol parameters like reveals an underlying weakness. That parameter only ensures connections reach a writable node when set to , it does not guarantee that only one node in the specified list of hosts is writable! If two nodes are promoted to read-write status, that's simply too bad, and cleaning up afterwards is up to you.Every safeguard matters, and so the safest course of action is to immediately terminate connections to hosts that fail three consecutive health checks.The next block simply consists of one line for each server, including the port to direct connections to, and the check port itself. Now connections to port 6543 on this server will be redirected to port 5432 on whichever node is the current primary. By running HAProxy on each VM, connections to any node will go to the primary without any additional configuration.

Starting and testing HAProxy

Once the configuration file is complete, start HAProxy to activate the new routing layer:That's really all there is to it. But we also want to test the proxy to verify that it's working as expected. The easiest way to do that is to connect from any node aside from the primary and then check to see where the connection went.Use to find the current primary:This output indicates node 1 is still the cluster primary. In that case, connect from node 2 or 3 on the proxy port and check the server IP address:That's definitely node 1. Success!

Adding endpoints

Remember how we mentioned the ability to add additional endpoints? Here's an example where connections will only go to replicas with less than 1MB of replication latency:After restarting HAProxy, it will also listen on port 6544 and only send connections to one of the replicas. These servers are idle, so there should be no lag, allowing us to test the connection this way:Go ahead and execute that command as many times as you want, it will never execute on node 1. How's that for convenience?

Finishing up

We've now completed the proverbial "HAProxy and DCS sandwich" Postgres cluster, made possible by Patroni. Here it is, in all of its glory:Both HAproxy and etcd (the DCS) are running on the same nodes as Postgres and Patroni for the purposes of this demonstration, but this is hardly a standard configuration. We only did it this way to simplify the example and require a minimum of virtual machines. A more typical cluster is more likely to decouple the HAProxy layer to a separate system, allowing it to act as a dedicated endpoint. It's usually easier to connect to "postgres-proxy.company.net" than arbitrarily assign Postgres VMs to various applications, or use multi-host connection strings. Putting HAProxy on its own host also allows it to revert to the default 5432 Postgres port and simply masquerade as Postgres.Another interesting variant includes running HAProxy at the application layer itself. Applications connect to the local server or server group and transparently route to the current Postgres primary based on HAProxy checks.The DCS (etcd here) is also likely to exist on a separate set of hosts. This allows multiple Patroni clusters to share the same consensus layer, and having a separate DCS layer makes two-node Postgres clusters a valid solution. For clusters that need to conserve storage or compute, reducing excess Postgres deployments is a great way to go. Larger or more established organizations may already have a consensus system (such as ZooKeeper or Consul) in place, so it makes sense to reuse these resources.Regardless of what your deployment model resembles, Patroni acts as the glue binding everything together: all Postgres nodes, the DCS, and the routing system. Everything acts as a single coherent cluster, perhaps in spite of how each component may act in isolation. In the end, you'll have the best HA solution available for Postgres short of a full Kubernetes solution, but in a much simpler package.We at pgEdge use Patroni in our own clusters, and now you know why. Let us know if you found this series useful, educational, or at least entertaining. Power to Postgres!

Using Patroni to Build a Highly Available Postgres Cluster—Part 2: Postgres and Patroni

Fri, 13 Mar 2026 06:12:14 GMT

Welcome to Part two of our series about building a High Availability Postgres cluster using Patroni! Part one focused entirely on establishing the DCS using etcd, providing the critical layer that Patroni uses to store metadata and guarantee its leadership token uniqueness across the cluster.With this solid foundation, it's now time to build the next layer in our stack: Patroni itself. Patroni does the job of managing the Postgres service and provides a command interface for node administration and monitoring. Technically the Patroni cluster is complete at the end of this article, but stick around for part three where we add the routing layer that brings everything together.Hopefully you still have the three VMs where you installed etcd. Those will be the same place where everything else happens, so if you haven’t already gone through the steps in part one, come back when you’re ready.Otherwise, let’s get started!

Installing Postgres

The Postgres community site has an incredibly thorough page dedicated to installation on various platforms. For the sake of convenience, this guide includes a simplified version of the Debian instructions. Perform these steps on all three servers.Start by setting up the PGDG repository:Then install your favorite version of Postgres. For the purposes of this guide, we’re also going to stop Postgres and drop the initial cluster the Postgres package creates. Patroni will recreate all of this anyway, and it should be in control.It’s also important to completely disable the default Postgres service since Patroni will be in charge:Finally, install the version of Patroni included in the PGDG repositories. This should be available on supported platforms like Debian and RedHat variants, but if it isn’t, you may have to resort to the official installation instructions.Once that command completes, we should have three fresh VMs ready for configuration.

Configuring Patroni the easy way

The Debian Patroni package provides a tool called that transforms a Patroni template into a configuration file customized specifically for Debian systems. Before using it, it’s necessary to modify part of that template to use etcd, as ZooKeeper is the default. Perform these steps on all three servers.Note that the YAML header shows “etcd3” rather than simply “etcd”. Patroni uses etcd2 by default for backward compatibility purposes, and version 3 requires a much different communication protocol.Then create the rest of the config with a single command:This creates a file named in the configuration directory, which systemd uses when managing this specific cluster. We’ll also need this for invoking .

Understanding Patroni configuration

Despite the fact that the configuration file is already complete, it’s important to actually understand the purpose of each section and what it does. This will enable users of other platforms to manually configure Patroni if necessary.Let’s start with the topmost section dedicated to the DCS:When Patroni writes to the DCS, all keys start at the path specified by the parameter. Similarly as one DCS may host multiple clusters, keys for this cluster must include in the key path. The indicates how Patroni should refer to this individual node. The configuration tool actually uses the DCS to see which names are already reserved so each VM will be uniquely identified. Go ahead and check all three to make sure they’re correct.The next section, labeled , determines how Patroni should create the initial Postgres cluster, the parameters to use, and other important information. It’s also pretty long, so let’s look at each individual portion:Normally Patroni uses when creating a new cluster, but for full compatibility with Debian organization quirks regarding Postgres, the configuration specifies an alternative command. This short section will likely only appear on a Debian system.Next comes the section under . All of these parameters should be covered in the Dynamic Configuration Settings documentation, but we’ll explain the important ones. It’s important to note that any settings defined here actually persist in the DCS layer and apply to Patroni on all nodes. After initialization, the only way to change these parameters is through the utility. It’s a good idea to make sure all of these are set properly, as changing them later is somewhat inconvenient.These parameters define how Patroni interacts with the DCS layer and how it should manage certain Postgres features. Remember that the leadership token determines which node is the primary, so defines how long that lease should last, controls how long to wait between lease renewals, and says how long to wait for a response from the DCS.We’ve included in this output because the leader race isn’t quite absolute. If Patroni promotes a node to primary, or determines Postgres has failed, it has up to this timeout before it forces a failover. The provides a grace period for crash recovery to complete, but you may find the default of five minutes is much too long. Another important parameter here is , which tells Patroni it should manage the Postgres configuration setting by automatically using names of other nodes in the cluster. This is how you would enable synchronous replication in Patroni.Next is the section under : This section defines how Patroni should operate the Postgres service. The first few parameters control how Patroni recycles old primary nodes, such as using the utility when possible, and whether it should erase the data directory as a last resort. Patroni also uses replication slots for replicas by default to prevent unnecessary replica rebuilds in failure scenarios.You can also pass GUC settings directly to Postgres on all nodes through the section. This is useful for providing important cluster-wide settings that may not be hardware dependent, such as , , or .The final section under the heading is : You’ll want to customize this section before starting Patroni; it uses this to build the pg_hba.conf file that controls incoming connection access. The default will allow connections on the server’s subnet if you uncomment the disabled line, otherwise it’s local access only.Next is another section, but this is a top-level header meant to tell Patroni how it should handle Postgres on this specific server. These sections are explained in more detail in the Patroni YAML Configuration Settings documentation.This example starts with some Debian-specific content:As before, this is so Debian can integrate with the other packaged Postgres tooling, so it’s safe to skip on other platforms. After that comes a few pertinent parameters for handling connections: This sample effectively tells Patroni how it should connect to the local Postgres service for administrative actions. Patroni uses unix sockets when possible using these settings, which makes sense as Patroni runs as the postgres OS user and has direct socket access.Then comes a fun section that defines several paths: Patroni knows it will be installed in several different environments where Postgres and configuration directories may be in completely arbitrary locations. These are the defaults for Postgres 18 running on a Debian system.Lastly there’s a second parameters section, meant for parameters that should only apply to this specific Postgres server: Nothing here should be surprising; it’s mostly just log storage for the local instance and where the unix socket directory is located. These are likely to be universal across the cluster, but it’s safer to leave them out of the DCS section. If there is ever any variance caused by a hardware or OS distribution migration, you’ll want to have the ability to change these locally.In any case, take some time to examine the file on each node to spot-check it for any mistakes.

Starting and validating Patroni

The Patroni package provides a standard systemd service file; simply enable and start the service on all VMs.One of the three nodes will “win” the leader race and become the primary for the cluster. Patroni then invokes the command on that system to create the data and configuration directories before starting Postgres. On the other nodes, Patroni calls instead to create new streaming replicas. If you want a specific node to start as the primary, simply start Patroni on that node and wait for it to establish a cluster before starting the service on the other two.The end result on all three systems should be a new “demo” database visible to : The next step is to check the status of the Patroni cluster itself. You should be able to run this command from any node as the OS user. It will also work as , but now that Patroni is installed and managing the cluster, it’s best to avoid relying on the root user.This output tells us the cluster is healthy and operational, node 1 is the current primary, both replicas are streaming, and there’s no replication lag. Success!

Editing the cluster configuration

The last step that might be necessary is to modify the cluster configuration stored in the DCS layer. These are the Postgres parameters and pg_hba.conf entries used to bootstrap the initial state of the cluster, and it’s easy to make mistakes early on.Once again, comes to the rescue: Patroni loads the current DCS config into the current default editor, and in our case it looks like this:Use this as an opportunity to fix any missing HBA lines, or add any Postgres parameters that should apply to all nodes. For example, add under to enable logical replication:Since changing the parameter requires a Postgres restart, use to restart the nodes in the cluster: Then check with Postgres to verify that the setting changed as expected. This is the output from node 3, even though I modified the DCS and restarted the cluster from node 1:

Finishing up

Now you know why this series was broken into three parts! Setting up Patroni isn’t too difficult by itself, but getting the configuration right, knowing how and why each section works the way it does, and continuing to modify the cluster after deployment, is a complex process. But if you followed along, you should have a fully operational Patroni cluster at this very moment.Technically you can even stop here and skip the third and final installment of this series. Postgres supports multi-host connection strings, and specifying for the restricts connections to the primary node. Connecting with psql might look like this:But what if, in some distant future, we change server names, or add more nodes to the cluster, or want other connection restrictions? That’s where the routing layer comes in, and what fully completes a Patroni deployment.So come back next week to learn about HAProxy and how it provides that critical and final component!

Using Patroni to Build a Highly Available Postgres Cluster—Part 1: etcd

Fri, 06 Mar 2026 07:48:55 GMT

The last PG Phriday article focused on the architecture of a Patroni cluster—the how and why of the design. This time around, it’s all about actually building one. I’ve often heard that operating Postgres can be intimidating, and Patroni is on a level above that. Well, I won’t argue on the second count, but I can try to at least ease some of the pain.To avoid an overwhelming deluge consisting of twenty pages of instructions, I’ve split this article into a series of three along these lines:

Etcd

Postgres and Patroni

HAProxy

This establishes each of the three layers that represent the full Patroni stack, and provides a convenient reference for later regarding each.With that out of the way, let’s get started!

Why etcd?

The last article should have made it abundantly clear that the DCS is the nexus of communication and status for the whole cluster. As a result, it’s important to install it first and certify that it’s operational. Etcd is the default and the example most often deployed in Patroni clusters. It’s also the key/value storage system Kubernetes uses as a default, so it should be reliable enough for our needs.Don’t forget to keep a browser tab opened to the etcd documentation handy.

What you’ll need

If you want to follow along with this demonstration, you’ll need:

The ability to create three VMs. Whether it’s
Amazon EC2
instances,
Microsoft Hyper-V
,
Xen
,
QEMU
,
Proxmox
,
Oracle VirtualBox
, or even
VMWare Fusion
, make sure you have a hypervisor and know how to use it.

Three VMs running
Debian Stable
version 13. At the time of writing, this should be the Trixie release.

SSH access as a root-capable user on each VM.

An internet connection. If you have the first three, it’s likely you have this as well.

Believe it or not, that should actually be all that’s necessary. While these instructions focus on Debian packaging when possible, feel free to substitute RedHat equivalents if you’d rather be adventurous. Most of these instructions should work on any Linux system if you’re familiar with your platform of choice and know how to improvise.If you want to make your life easier, add some lines to on the VMs to give each a name. IP addresses are great, but they’re not as convenient as “pg1”. Here’s an example:Unless otherwise noted, execute commands described in this guide on each of the VMs.

Preparing each VM

Prior to installing etcd, let’s create a user named “etcd” to own the service and related data using a quick command:It’s important to create the user as a “system” user, as these are often treated differently by Systemd.

Installing etcd

The first lesson is that most of these tools are not “properly” packaged. By that, I mean that there are no official .deb or .rpm packages that should be considered recent. The etcd software maintainers do not provide anything aside from Zip, tarballs, or source code. That means the first step is to visit the etcd GitHub release page and find the URL for the latest release.With that URL, install with these commands:Then we want to invoke the Debian alternatives system to make the binaries easier to use:Finally, create a systemd service file to control the etcd service:Don’t start the service; we haven’t configured it yet.

Configuring etcd

For etcd, the “hard” part is getting the configuration right. Take note of all bolded parameters, including in the following block. These should all reflect the name or IP address of the server you’re configuring! The easiest way to do this is to use environment variables as shown in the example.The listen URLs are IP addresses because etcd attempts to resolve hostnames when specified for these parameters.Once the configuration file exists on each of the servers, enable and start the etcd service itself.Since the configuration file states that this is a “new” cluster, the cluster won’t consider itself bootstrapped until all three servers are online and connected to each other. Give it a minute or two before continuing.

Validating the service

Once etcd starts on all nodes, it’s a good idea to verify that it’s working as expected before handing it over to Patroni. Start by launching the etcdctl tool to view the cluster member list, which should include all three nodes:We can see here that all nodes are accounted for, but this doesn’t actually show the state of each, just that they have joined the cluster. For that, we need a different command:And this is what we see if a node is offline:Finally, write a sample value to the DCS, retrieve it from a different node, and delete it from a third. This ensures that all nodes can write based on the consensus between them.This full lifecycle proves etcd is operating as expected, all nodes are fully operational, and this cluster is ready for Patroni.

Finishing up

You should have three VMs equipped with an etcd service at this stage, and that provides a convenient stopping point for the next article. If you were wondering why we’re installing everything on three nodes, this is because it’s the minimum viable HA cluster with any real meaning.While it’s possible to run a two-node cluster, quorum requires a majority to guarantee consensus. Any node that disagrees is simply treated as incorrect, and should re-synchronize with the majority. So a two node cluster must always keep both nodes online or it can’t be trusted. A three node cluster has a spare, as it becomes possible to stop a single node while the other two maintain the consensus.As a result, no real cluster has fewer than three nodes. Please note that this only applies to the etcd layer! Consider this cluster design:This is the same as our original diagram, but with only two Postgres / Patroni elements. This is perfectly valid because the DCS layer itself maintains the quorum, so we don’t have to enforce that same constraint on Postgres or Patroni. This means we could theoretically operate two Postgres nodes in different regions under the assumption that there’s an externally managed DCS layer.In the case of this demonstration however, we don’t have that luxury. To decouple Patroni from etcd that way requires a five node cluster: three for etcd, and two for Patroni and Postgres. That’s actually the superior approach for more sophisticated architectures since multiple Patroni clusters can share a single etcd resource.We may explore that kind of advanced use case in the future, but for now, experiment with your new etcd cluster and we’ll see you next week!

How Patroni Brings High Availability to Postgres

Fri, 27 Feb 2026 05:33:52 GMT

Let’s face it, there are a multitude of High Availability tools for managing Postgres clusters. This landscape evolved over a period of decades to reach its current state, and there’s a lot of confusion in the community as a result. Whether it’s Reddit, the Postgres mailing lists, Slack, Discord, IRC, conference talks, or any number of venues, one of the most frequent questions I encounter is: How do I make Postgres HA?My answer has been a steadfast “Just use Patroni,” since about 2017. Unless something miraculous happens in the Postgres ecosystem, that answer is very unlikely to change. But why? What makes Patroni the “final answer” when it comes to Postgres and high availability? It has a lot to do with how Patroni does its job, and that’s what we’ll be exploring in this article.

The elephant in the room

By itself, Postgres is not a cluster in the sense most people visualize. They may envision a sophisticated mass of interconnected servers, furiously blinking their lights at each other, aware of each computation the others make, ready to take over should one fail. In reality, the “official” use of the word “cluster” in the Postgres world is just one or more databases associated with a single Postgres instance. It’s right in the documentation for Creating a Database Cluster.“A database cluster is a collection of databases that is managed by a single instance of a running database server.”The concept of multiple such instances interacting is so alien to Postgres that it didn’t even exist until version 9.0 introduced Hot Standbys and streaming replication back in 2010. And how do hot standby instances work? The same way as the primary node: they apply WAL pages to the backend heap files. Those WAL pages may be supplied from archived WAL files or by streaming them from the primary itself, but it’s still just continuous crash recovery by another name.This matters because each Postgres node still knows little to nothing about other nodes in this makeshift cluster over 15 years later. This isn’t necessarily a problem in itself, but it betrays a certain amount of willful ignorance on the part of each node. Why doesn’t each node care about, or even really acknowledge the other nodes exist at all?Of course each node should care that the other nodes exist! And that’s how every Postgres HA tool was born. Slony and PgPool-II were probably the first of these, using Pacemaker and Corosync was always popular in the early days, then came Bucardo, repmgr, and EFM. Those are just the more noteworthy examples known by most of the community.But a funny thing happened after Patroni’s initial release: the relentless torrent of Postgres HA tools suddenly ceased. Everyone immediately understood something made it fundamentally different from its predecessors. Let’s talk about why.

What Patroni does

Patroni does something Postgres still doesn’t do: it builds a cluster of Postgres nodes. It does this by facilitating what I like to call a “HAProxy and DCS Sandwich” that looks something like this:Think of it like a Postgres BLT where Patroni acts as the lettuce that brings everything together. It’s the missing communications nexus that records the composition of the cluster, the status of its members, and routes connections where they need to go.Let’s dive a bit deeper into how it does all of this, and why that matters to everyone from hobbyists to fortune 500 corporations.

Quorum

The first and most important aspect of Patroni’s operational role is that of maintaining quorum. Here’s a handy definition for a quorum:The minimal number of officers and members of a committee or organization, usually a majority, who must be present for valid transactions of business.The critical aspect here is the voting majority, otherwise known as a Consensus. The standard formula for this for some number of nodes N is: N/2 + 1. While a two-node cluster would need both nodes to remain online to maintain a majority, a three-node cluster would also require two nodes to maintain a majority. It’s this “extra” node that creates resilience in a network cluster. Should one node become isolated from the others, either through failure or a network partition, the quorum remains and the cluster stays operational. More nodes usually also confers better protection; three is best out of five, after all. Due to communication overhead caused by node topology, most consensus layers suggest staying below a “handful” of nodes, which tends to mean “fewer than ten”.Ironically, Patroni handles quorum by delegating that responsibility to another piece of software entirely. Patroni reports compatibility with four different key/value or Distributed Configuration Store services, including etcd, Consul, ZooKeeper, and even Kubernetes. In reality, Patroni doesn’t really care where the DCS layer lives or what it’s composed of, just so long as it responds to read and write requests.That’s why the “DCS” layer in the diagram is a flat plane supporting all of the Postgres nodes. The DCS could be anywhere, using any number of nodes, and Patroni doesn’t have to manage it.

Orchestration

Patroni is a specialized high-availability tool designed specifically for Postgres. As a result, it knows how to manage anything associated with a cluster of Postgres instances, including but not limited to:

starting and stopping the Postgres service.

promoting replicas.

bootstrapping new replicas.

demoting primary nodes.

Log Sequence Numbers (LSNs).

replication slots.

Patroni stores all metadata in the DCS layer, updating it regularly for every node. The “cluster” always knows the status of all nodes, including any replication lag. The magic of how Patroni works so well is how it knows which node is the Primary for the cluster: the leadership token. Here’s how it works:

Patroni checks to see if the current node owns the leadership token.

If yes, refresh the token and restart the loop.

If no, can this node
take
the leadership token?

If yes, take the token, promote this node, and restart the loop.

If no, act as a normal replica, reconfigure for the current primary if needed.

There are other steps involved of course, but since the consensus layer is distributed, there can only ever be one leadership token. Once a node has the token, no other node can claim to be the primary node. Based on which node has the token, Patroni will reconfigure all other nodes to use it as the primary. If a replica encounters replay errors, or was a previous primary, Patroni will use pg_rewind, pg_basebackup from the current primary, or even recover from a stored backup to rebuild the node.That’s something almost none of the other HA tools do. Not only will Patroni promote a replacement primary, but it will rebuild the failed one if it can. If you add a new node to the cluster, it creates the data directory on your behalf based on the cluster configuration. The DCS is the single source of truth Patroni operates from, and in a very real sense, the DCS itself is the cluster.Things really start to get interesting when the DCS layer itself experiences failures.

Fencing

The idea behind fencing is that a misbehaving node should be decommissioned. The reasoning for this is deceptively simple: in the absence of consensus, you can’t trust any written data. There are many reasons a node could lose contact with the DCS, or the DCS refuses to respond, and none of them matter at all. The safest course of action is to stop Postgres.If the primary node can’t maintain its ownership of the leadership token, another node seizes it. The Patroni process on that node promotes it to leader, the cluster reconfigures itself around that new primary, and the beat goes on. Isolated replicas don’t have to worry about writes, but they also can’t participate in the leadership race.An isolated primary knows another node has been promoted in its absence, that it should reject writes to prevent split brain risk, that it should no longer accept new connections. Similarly, a replica cut off from the DCS can’t be monitored, is likely accumulating replication lag, and is otherwise suspect. As a result, Patroni stops the Postgres service on that node.Believe it or not, most Postgres HA solutions omit this critical factor. Almost all of them will detect a primary failure and promote a standby, but almost none consider what happens if the failure is the network and not the node or Postgres itself. In these systems, an isolated node keeps accepting writes from colocated systems or established connections, keeps operating normally, and doesn’t know or care that a promotion happened elsewhere.The Postgres service on isolated nodes absolutely must self-terminate, and Patroni ensures that outcome by its very design. Lose contact with the DCS, or if the DCS refuses requests for any reason, shut down. Easy.Note: One failure scenario for the cluster is that the DCS itself loses quorum. In a five node cluster, a network error could split two nodes from the other three. In such a situation, two nodes lost the majority and will refuse to operate in that state. The Patroni service for any affected nodes doesn't know this, and indeed, it doesn’t matter. The end result is always to stop Postgres.

Routing

The final thing Patroni does to establish a Postgres cluster is manage connection routing. It does this by tracking the ownership of the leadership token and providing an HTTP REST status interface. Any front-end routing system can interrogate a Patroni node for its current state. Whether or not Postgres is online, if the node should be considered writable, if there’s too much replication lag, and so on.The usual choice for this routing layer is HAProxy as reflected in the architecture diagram, but it could easily be an F5 load balancer, an Amazon ELB, and so on. This determines which connections reach what node—or whether a node should allow connections at all. Users who wish to connect to the Postgres primary simply need to connect to the routing layer. Is it important to connect to a replica that has less than 5MB of replication lag? Routing layer. Patroni evaluates criteria encoded in the health check request and responds accordingly.Fencing is one half of the equation, and routing control is the other. If Patroni determines a node should not be routable for any reason, it simply returns a failure on the REST interface. A properly configured routing component will then immediately cut any established connections and refuse future routing for that node until it is healthy again.More importantly, users and applications don’t need to know anything about the node they’re connecting to. In a very real sense, they’re not connecting to a node at all, but the cluster itself. Now, finally, it’s possible to accurately describe Postgres as a cluster of nodes. Each individual node doesn’t actually operate any differently, but Patroni and the underlying DCS establish an underlying fabric that binds everything together.

What about Kubernetes?

Kubernetes solves the problem of truly establishing a Postgres cluster in a similar fashion. Kubernetes operators like CloudNativePG either take the same role as Patroni, or literally use Patroni under the hood like the Crunchy operator. But Kubernetes is a rich ecosystem of inter-operating components with its own design ethos. A Postgres cluster emerges from this design as a consequence of its underlying fundamentals.Kubernetes isn’t specific to Postgres, and a Postgres HA solution in Kubernetes cannot exist outside of that environment. It is a perfectly valid way to transform Postgres into a cluster, but software outside of a Kubernetes context can’t leverage those capabilities. At least for now, the vast majority of Postgres installations still exist on bare metal, VMs, and other ad-hoc manual deployments.

Final thoughts

Let’s look at MongoDB for a second. The following is a diagram taken directly from their architecture documentation:We’re not here to discuss the merits of Postgres versus MongoDB or NoSQL in general. However, look very carefully at this structure. The design manual describes shard architecture in further detail: “If the primary replica for a shard fails, secondary replicas together determine which replica should be elected as the new primary using an extended implementation of the Raft consensus algorithm.”Does any of this sound familiar? Ignore the sharding aspect for a moment and consider what’s happening here. The cluster has a routing layer and nodes coordinate by consensus through Raft. MongoDB was designed from the very beginning to be a cluster, while Postgres treats node interaction as an afterthought. Outside of extensions like Spock from pgEdge, BDR from EnterpriseDB, or forks like YugabyteDB, every Postgres node is an island. Even Citus, an extension known for using coordinator nodes and data nodes and should be thought of as a cluster, needs Patroni to handle failover between data node replicas.Postgres simply isn’t a self-organizing cluster without some external orchestration layer. For now, Patroni is the best of these. There’s a reason we use it at pgEdge as part of our Ultra HA architecture. It’s impossible to say what the future might hold, but for now, just use Patroni.

Returning Multiple Rows with Postgres Extensions

Mon, 27 Oct 2025 05:09:23 GMT

Creating an extension for Postgres is an experience worthy of immense satisfaction. You get to contribute to the extension ecosystem while providing valuable functionality to other Postgres users. It’s also an incredibly challenging exercise in many ways, so we’re glad you’ve returned to learn a bit more about building Postgres extensions.In the previous article in this series, we discussed creating an extension to block DDL. That sample extension was admittedly fairly trivial, in that it only added a single configuration parameter and utilized one callback hook. A more complete extension would provide a function or view to Postgres so users could interact with the extension itself. So let’s do just that!

Deciding on Data

Once again we’re faced with choosing a topic for the extension. Users will sometimes ask the question: how much memory is Postgres using? Using various tools like top or ps will show a fairly inaccurate picture of this, limiting results to opaque fields like VIRT, RES, SHR, PSS, RSS, and others. Some are aggregates, others include Postgres shared buffers, and none really describe how memory is being used.Luckily on Linux systems, there’s an incredibly handy /proc filesystem that provides a plethora of information about several process metrics, including memory. The smaps file in particular reports several process-specific memory categories, and does so on a per-allocation basis. What if we could parse that file for every Postgres backend process and return output in a table? Admins could then see exactly which user sessions or worker processes are using the most memory and why, rather than an imprecise virtual or resident memory summary.Sounds interesting!

Starting the Extension

As with our previous extension, we need to bootstrap the project with a few files. Start with creating the project folder:And create a file with these contents:As before, we just need to name the extension, give it a version, and provide an installation path for the resulting library file. Nothing surprising so far.Next we need a makefile; let’s start with this:Look familiar? Simple extensions won’t likely need to customize the makefile very much, and in this case we’re just describing the extension and its contents. More sophisticated extensions require more complicated makefiles, but we’re still covering easier ground.Don’t forget your copy of the Postgres source code; it’s going to be incredibly valuable in this exercise.

Functions and Macros

Now it’s time to work on the body of our extension. Before that, we need to learn a bit more about some of the macros and API functions Postgres provides for this kind of extension. We already know about the macro which prepares the extension for Postgres, but there are several more.The C-Language Functions documentation says that any function exposed to users must be declared with the macro, and the function should use rather than an actual argument list. The function we’re building won’t take any arguments, but the macro is required anyway.The “Returning Sets” section in particular explains that there are two types of functions which can return a set: value per call, or materialized. Given we’re parsing potentially ephemeral kernel memory mappings, we probably want to return the entire data-set at once. We also have to consider the fact it’s necessary to obtain the list of running processes, and if we used value-per-call, we’d have to effectively cache that output so the function could use it for each call. It’s simply easier to materialize for now.Since we’re writing a function, the src/backend/utils/fmgr/README file also presents a few pertinent structures we’ll need to know:

The documentation also says a set-returning function will need to interact with tuplestores. That means we should examine src/include/utils/tuplestore.h to understand that API. The most interesting function here is , which takes a , a , an array of values, and an array of booleans defining which are NULL. This will allow us to pass an entire array of values representing the row, and another array specifying which (if any) are null.The tricky part is figuring out what and are. It turns out that in this context is a struct, and if we examine src/include/nodes/execnodes.h where it’s defined, we’ll see that the field is a which is intended to contain our result set! Similarly, is a which describes the tuples we’re returning. That should be all we need for storing the return values.We don’t quite have everything, however. Postgres expects result sets to be composed of structures, which encapsulate supported data types like , , and so on. In order to transform “raw” C types into these, we need to use another set of functions defined in src/include/postgres.h and src/include/utils/builtins.h. Primarily we are looking for these functions:

Why is defined in a completely different file than ? Who knows. It might be tempting to use , but that would be a mistake; Postgres Text and CString types are treated quite differently and this would make it difficult to manipulate the fields in queries.Next we need the ability to capture the contents of so we can use the PID of each process to obtain all of the memory mapping information. Finding that is a bit of an adventure. First we start at src/backend/catalog/system_views.sql where it’s defined, to find that it’s a view which calls the function. That function is defined in src/backend/utils/adt/pgstatfuncs.c and provides a great example at how to make internal system calls. This function is meant to be used by Postgres user sessions, so we want to effectively “mine” it for internal routines. Here are the important ones:

These are all defined in src/include/utils/backend_status.h , which is fairly convenient for us.Finally, we can’t forget about security. This function returns a lot of potentially sensitive information, so we want to restrict it to users who are either superusers or who have access to the pg_read_all_stats predefined role. If we check out src/include/miscadmin.h, the function retrieves the UID of the current user.Nailing down the ability to check the role is a bit trickier. Searching the source basically only shows this string in source comments, but the usage is clear: use to check a user’s roles against . That means we also need the src/include/utils/acl.h header.There’s one final piece that’s slightly non-obvious. The definition doesn’t actually exist yet. That header file is generated during the build process and placed in a file included by src/include/catalog/pg_authid.h. This file is then included by src/backend/utils/adt/acl.c and not the previously mentioned This happens frequently when working with catalog data since these are defined as files ending with a extension that are converted into actual headers at build time.It’s admittedly a bit confusing, but forewarned is forearmed!

Ins and Outs

Isn’t it interesting how much we had to search the code for various Postgres internal system calls? The published documentation really only scratches the surface of everything necessary to write even a fairly basic extension, so always have a copy of the actual source handy!Given our discussion so far, we’ll need several includes:It’s possible to reduce this list by almost half by accounting for the dependency graph, but in the interest of being complete, it’s best to simply include every header which declares a referenced symbol. Also note that as with the previous extension, the header always comes first.Then start the module, declare the function we plan to expose to Postgres, and begin with the function definition:These are all things we’ve discussed thus far, and should present no surprise. The function takes which expands to , a variable we’ll be using frequently. The function must return a , even if we’re returning a set.For the function body, we need to do a bit of housekeeping. As discussed, the first step is to limit execution only to allowed roles. That means we need some variant of this code in the body:The only thing we haven’t discussed here is the error code. These are described in src/backend/utils/errcodes.txt , so always check this to see if there is a more applicable error when rejecting user actions.Next we call the indispensable helper function that makes all of this possible:This single function call prepares our function environment for returning a set, performing all of these actions:

Ensuring the caller can receive a result set.

Asserting materialize mode is allowed.

Creating a tuple store and description in the correct context to store the result set.

Setting several result-set attributes to enable returning a result set.

It’s about forty lines of code we no longer have to worry about.Next would come the actual program body, which we’ll discuss at greater depth in the following section. The end of the function should always end with this:This indicates to Postgres that row production is complete.

The Main Loop

Now we’re ready to build the central loop for the set results. Recall how we have a function to retrieve the number of existing backend processes, and another to fetch the contents of each. Here’s how we could use those to locate the appropriate smap file for parsing:It’s quite beneficial that we can retrieve all backend status simply using the index of the backend. In our case it’s only necessary to retrieve the PID. Once we have that, we just need to loop through each of the smap files using . Perhaps it would be better to include some error-handling here in case the files are unavailable or can’t be opened, but this is fine for demonstration purposes.The parsing loop is a bit more complicated. Since this is a blog, we’re using a magical function called which will return a struct of all known smap fields. For the actual implementation, we’ve put together a GitHub repository you can use for a full understanding of how we built this extension.The inside of the parsing loop needs a few things for the tuple functions we discussed earlier: an array of Datums, an array of nulls, and a call to the function. Given our magic function and struct, here’s how that might work:Note from this example that we never set any values in the array under the assumption that zero is the default and preferred value. Aside from the metadata rows about the memory segment itself, most of the other fields are some amount of kilobytes. Rather than showing all 33 possible fields, we’ve elected to omit them for the sake of brevity.So long as the parsing function returns records from the memory mapping, the materialized result will continue accumulating rows. This will repeat for each Postgres backend until every one has been processed, and then we exit the loop. Once the function returns, the accumulated tuples are handed to the caller.

Becoming a Postgres Extension

Writing the function is only half of the story, it’s also necessary to create a SQL to C binding. This accomplishes a couple of things:

Provide a SQL wrapper so the function is usable within Postgres sessions.

Defines all of the types for each column in the result set. This is how Postgres builds the
setDesc
portion of the result set struct.

This has a naming convention based the Packaging Related Objects into an Extension documentation. For our extension, this would be: . The contents should be the function declaration including the entirety of the result set. For this function and all fields in the memory mappings, it would look like this:The portion will automatically be substituted for the installed location of the shared library in Postgres. Running this SQL after the extension shared object files are installed will create an SQL function named that would call our C function.This would be the minimum to get a working Postgres extension. Assuming all of the files are complete, installing consists of a few steps. First is to build and install the actual library:Next, restart Postgres using your favorite method. And finally, install the extension itself:This actually executes the SQL file we prepared above, ensuring that the function mapping exists. Once that happens, all subsequent Postgres sessions should be able to utilize the function like this:It sure was a lot of work getting to that point, but it was all worth it!

Odds and Ends

We definitely glossed over a few pertinent details here, the most obvious of which is the implementation of the memory map parser itself. Unless all intermediate structs and functions are defined in the same file as the extension function itself, these are probably in a separate C file for organization purposes. (Again, full sources for this blog are available in a dedicated GitHub repository.)Imagine these are in a parser.c source file that should be part of the extension build procedure. The Makefile example we used doesn’t really account for building multiple sources, or binding them together into a single module. It’s easy to remedy however by using dedicated build variables in the Makefile itself:The assumption is now that and exist, produce the resulting object files during the build step, and are then combined into a single module. Everything else is the same as before and the extension will work as described once installed, we simply have more organized source files as a result.

What's Our Vector, Victor? Building AI Apps with Postgres

Wed, 15 Oct 2025 04:14:58 GMT

Something I’ve presented about recently (a couple times, now!) is how we can make AI actually useful with Postgres. Not the mystical robot overlord kind of AI that people worry about, but the practical, math-heavy kind that can actually help solve real problems with the data you already have.Let me be clear up front: AI isn't magic. It's math. Lots and lots of math. And if you can write SQL queries (which, let's be honest, is why you're here), you can build AI applications with Postgres. Let me show you how.

Breaking Down the AI Buzzwords

Before we get too deep into the technical stuff, let's clarify some terms that get thrown around way too much these days:LLM (Large Language Model): This is what people are actually talking to when they chat with AI systems like ChatGPT or Claude. It's the thing that generates responses that sound surprisingly human.RAG (Retrieval Augmented Generation): We're all tired of AIs making stuff up, right? RAG forces them to use reference material - your actual data - instead of just hallucinating answers. It's basically saying "here's some context, now answer based on this instead of whatever random training data you remember."Tokens: Text gets broken into chunks, and those chunks get meaning associated with them. Think of it as building up from word parts to bigger concepts, piece by piece.Embeddings: Here's where it gets interesting, and where Postgres really starts to shine. An embedding is a vector, essentially a coordinate system for those tokens. You know how a Cartesian plot has X and Y coordinates? Vectors are like that, except they're hundreds or even thousands of coordinates long. That's where AI concepts live and get their granularity.

Understanding Vectors: The School of Fish

I love using this analogy because it actually captures what's happening under the hood. Picture a school of fish in a coral reef. A vector isn't just representing the species of fish, it's capturing everything about that fish in context. The position of its fins, what other fish it's swimming near, its place in the family tree, the rocks around it, what it eats, how it behaves.That's exactly what a vector does for text or concepts. It's not just the word itself, it's all the surrounding context and meaning that gives it substance. It's like having a 3D hologram where you capture not just the position of the fish, but everything that makes that fish what it is at that moment.This is why when you talk to an LLM, you get all that surrounding context. All the related concepts, the metadata, the contextual relationships - they're all encoded in those coordinate arrays.

The Postgres AI Stack That Actually Works

Here's what makes this all work with Postgres, and why I think this approach is superior to the usual Python-heavy AI stacks:pgvector: This is the foundation. It adds vector similarity searches to Postgres, along with different vector types (binary, sparse, various array types), operators for comparing vectors (cosine, euclidean, dot product), and crucially, new index types like HNSW that make searching through thousands of coordinates actually feasible instead of impossibly slow.pg_vectorize: This is where things get really interesting, and it's the extension I personally prefer for real-world work. Written in Rust by Adam Hendel (@chuckhend on GitHub), it builds on pgvector but adds the functionality that makes everything actually usable in production. It uses pgmq for background queuing and can integrate with pgCron for scheduled operations.The beauty of pg_vectorize is that it handles all the tedious orchestration work that usually requires a bunch of Python code and multiple API calls to different services.

What pg_vectorize Actually Does

Here's where the magic happens. pg_vectorize gives you several key capabilities:

1. Text Encoding

Using vectorize.encode, you can transform individual prompts to search terms using whatever compatible vector you prefer. The output is compatible with pgvector vector search.Obviously, the answer is yes, but let's keep going.

2. Create and Maintain Embeddings

This is where pg_vectorize really shines:When you specify realtime, something clever happens: any time you insert or update a row, it generates an embedding in the background using pgmq. Your insert completes immediately; the embedding generation happens asynchronously. This is absolutely crucial because generating embeddings can be slow (it's either a local CPU operation or a remote GPU call), and you definitely don't want that latency directly impacting your write performance.I saw a Microsoft demo at a conference where they were generating embeddings synchronously on every insert. That's not a great approach if you value database performance.Embeddings are maintained by pg_cron job, or pgmq live updates.

3. Vector Search

This searches through all your vectorized content and returns the most semantically similar matches. It's not just keyword matching, it's understanding the intent and context behind your query.

4. Complete RAG in One Query

Here's the real magic trick:This single function takes your question, compares it against your vectorized content, pulls back the best matches, combines that with a prompt, sends it to the LLM, and returns a complete response as a JSON object that includes context if we need it. All the orchestration that used to require dozens of lines of Python code - gone.

The Architecture Simplification

The traditional RAG architecture is a nightmare of moving parts: User → App Layer → Transformer API → Database lookup → Combine context → LLM API → Parse response → App Layer → User. That's a lot of potential failure points and a lot of code to maintain.With pg_vectorize, your stack becomes: App Layer makes SQL query → Postgres handles everything → Response back.I rebuilt a RAG application I had originally written in Python using this approach, and I'm not exaggerating when I say it cut my codebase by at least half. Much easier to maintain, much easier to reason about, and much fewer things that can break.

Practical Implementation: The Chunking Strategy

Here's something important that doesn't get talked about enough: embeddings are less granular than LLMs themselves. Embeddings are usually “fuzzy” (only 384 coordinates), while LLMs have billions of parameters. This means you need to think strategically about chunking your content.Don't try to create embeddings for entire 10-page articles. Break them into logical chunks: paragraphs, sections, whatever makes sense for your content. Each chunk gets its own embedding. When someone asks a question, you're matching against the specific chunk that's most relevant, not trying to capture the essence of an entire document in a single vector. This will provide sharper context.
-- Example chunked table Then you vectorize the chunks table instead of the articles table. Much better semantic matching.

Advanced Technique: Query Rephrasing

Here's a trick I learned from Adam that's incredibly useful: users don't always phrase questions in ways that match well with embeddings. If someone asks "how do window functions work?", a naive embedding search might just return content that contains the words "window" and "function" frequently, not necessarily content that explains what window functions actually do.However, there’s an elegant solution. Use the LLM to rephrase the query first:-- Step 1: Get a semantically better queryThis might return something like "SQL window function syntax row partitions unbounded practical examples" which is much richer semantic content.-- Step 2: Use that improved query for your RAG searchThis gives you much better semantic matching instead of just literal word matches. The difference in result quality is dramatic.

Configuration and Setup

You can configure pg_vectorize to work with external services like OpenAI, or you can run everything locally. Want to keep your data completely internal?-- Search using Ollama or vLLM instead:-- Or, use a custom transformer service:With distributed Postgres systems, you can co-locate your AI processing with your data nodes, reducing latency and keeping everything within your security perimeter. Using this approach, no data leaves your infrastructure.

Why This Matters

The bottom line is this: if you can write SQL queries, you can build sophisticated AI applications. You don't need to become an expert in transformer architectures, manage API tokens across multiple services, handle complex JSON parsing from different AI providers, or write orchestration code to coordinate between embedding services and LLMs.Your data is already in Postgres. Your application logic can stay close to your data. The AI functionality becomes just another database capability rather than a separate system to manage.And here's the real advantage: when ChatGPT is having an outage or when your OpenAI bill is getting out of hand, your local setup keeps running. When you need to meet compliance requirements about data not leaving your infrastructure, you've got that covered. When you want to fine-tune models on your specific domain, you control the entire stack.

Getting Started

If you can write queries, you can build AI apps using Postgres. All you have to do is load your content (which you'd have to do anyway), vectorize your tables with a single function call, and start querying. It doesn’t matter whether you're building a customer service system that knows your documentation, a search interface for internal knowledge bases, or any other AI-powered application; it all can be contained within a few SQL functions.That's the real power of keeping AI close to your data. Not to mention, chances are if you're here, your data is already in Postgres.Thinking about watching the webinar in full? It’s recorded - find it here

Introduction to Postgres Extension Development

Thu, 11 Sep 2025 05:37:00 GMT

Possibly the most notorious benefit to running a Postgres cluster is access to the vast ecosystem of extensions, including procedural languages, foreign data wrappers, index types and storage systems, handy utility functions and much more. Extensions run the gamut from columnar table storage, efficient vector functions , or BM25 fulltext search analytics, to more trivial augmentations like UUIDv7.But what’s involved in actually developing an extension? What arcane Postgres APIs need to be invoked? How does installation work? Does it take a genius? This article will attempt to answer all of those questions and a few more besides, and hopefully by the end, you too can achieve immortality by contributing a useful feature to Postgres.

Choosing a Topic

I’ve written two relatively simple SQL-only extensions in the past. This isn’t uncommon, as packaging up a few PL/pgSQL functions into a utility library is a convenient way to distribute them. Since the possible subject matter is so vast, what might be an interesting piece of functionality to add using an extension? Some DBAs might want to prevent users from executing DDL for instance, so would that be something we could do?More astute readers probably already know that it’s possible to reject DDL using a Postgres Event Trigger like this:Ironically however, the command itself will not invoke this trigger. What if we wanted a more tightly coupled method, directly embedded in Postgres itself through an extension written in C? How might that change things?

Getting Things Under Control

According to the Postgres extension documentation, all extensions require a control file to define various bits of metadata about the extension itself.So let’s create a folder for the extension:And create a file with these contents:This is essentially the bare minimum necessary for a control file. It provides the name of the extension, its version, and the path where the library itself may be found. In this case, the library path is just the default location for a library named “noddl”.Now Postgres knows what to do with our fancy new extension when it gets installed.

Preparing the Environment

Since this will be a C extension, we need to prepare the development environment a bit more. A Postgres build environment requires a minimum of gcc, make, and the Postgres development headers. This is all completely operating system and distribution dependent, so doing this is beyond the scope of this guide.However, the Postgres Download repository usually includes development packages for all supported operating systems. If this were a Debian, Ubuntu, or Mint system for example, adding Postgres development packages is one convenient command away:Red Hat, Fedora, or Rocky systems may do this instead:Once the postgres development headers are installed, double-check the availability of the command, as this will guide the build and install process for our extension.The flag should display the location of the extension Makefile that will do most of the hard work of setting header paths, installing to the correct library directory, and so on.It’s also a good idea to have a local copy of the Postgres source itself. Postgres C extension development often depends on invoking otherwise undocumented internal functions, referencing multiple structs, and calling several interface macros. The best way to do this is to examine the source code where these things are defined, as there is often an incredibly helpful comment heading or even a README which explains things in greater detail.So go to GitHub and obtain a copy:Trust me, you’ll be very glad you did.

Making Everything Possible

Speaking of makefiles, it’s time to create one for the noddl extension. The PGXS documentation provides a sample Makefile along with an explanation of all available makefile parameters. Using that as a guide, our should look something like this:Similarly to the control file, we’re really just describing the extension. Note that the last three lines in the file are there to include the Postgres extension makefile which does all of the real work. There’s nothing better than delegating responsibilities when you don’t really know what you’re doing.

Start Your Engines

There are two critical components to a Postgres C extension.

The
PG_MODULE_MAGIC
macro. Don’t worry about this; it’s magic!

A
PGinit
function to act as an entrypoint into the extension for setting hooks, creating GUCs, launching worker processes, and so on.

Given this module will be blocking DDL, there should be a convenient way to enable and disable it so superusers can execute DDL when necessary. That means we need at least one GUC. If we take this into account, an early skeleton for the extension will look something like this:This is actually a functional extension already. The only thing it does right now is set the GUC, but this will successfully compile and install as a functional Postgres extension.The next thing we should do is make our extension useful.

Intercepting DDL Commands

Understanding how to trap DDL statements is easier said than done. We know that all statements must be parsed before they’re dispatched to whatever part of the engine that will execute them. That means we need to add some kind of function to the list of hooks Postgres calls during that process.It turns out that one of the categories the Postgres dispatcher uses is reserved for utility commands. If we check out the Postgres source, there’s even a test module which specifically leverages this functionality: test_oat_hooks. If we examine test_oat_hooks.c a bit, we’ll find this code in This code essentially checks a custom GUC to decide whether or not to block statements at all, and will only block commands for non-superusers. If both of these criteria match, it throws an error, and that’s really the extent of the function. This example demonstrates how to indiscriminately reject all utility commands, but it’s the first step to isolating DDL events.How do we isolate DDL specifically? If we look for the string “DDL” in the Postgres source code, src/backend/tcop/utility.c stands out dramatically. The function categorizes all command tags into various categories for the log_statement GUC. The signature includes a parameter which includes a field. If we pass that into the function, we only want statements identified as .If we take that into consideration, blocking DDL looks something like this:The particular hook we want to set is named . Assuming our callback function is named , we just need to add this to the end of our function:If we stopped here, our hook function would be fully operational and our work would be done. Unfortunately it’s not quite that simple.

Being a Good Neighbor

Unlike some extensible software, Postgres does not provide utility functions to register hooks in some kind of dedicated callback stack. Instead, it defines several global variables which extensions may choose to override, one of which we set in the previous section. However, these global variables may be manipulated by any extension loaded into the process environment, which could represent a user session, background worker, or some other kind of mixed context.What if is set like this:Now what happens if the noddl extension sets the hook? In this case, it would essentially deactivate the pg_stat_statements extension! What we really need to do is preserve any previously set hook and propagate the call in a kind of simulated stack. Every well-behaved extension does this, so in theory every extension’s hooks are eventually honored. That means we need another global variable:Then we need to preserve any previously defined ProcessUtility hook at the end of before overriding it:And finally, the full body of our own ProcessUtility function hook needs to physically invoke either the default Postgres ProcessUtility function, or a previously defined hook:In an ideal world, Postgres would handle hook stacks directly, if only to prevent one misbehaving extension from ruining it for everybody. It’s a bit ironic that our extension has less of its own code than calls to maintain an unbroken hook chain.

Some Assembly Required

Believe it or not, that is the entire extension. Who knew writing a C extension for Postgres would be so simple? Since we already have a makefile, the final steps are to make, install, and activate the extension. Let’s start with building it:That will deposit the extension in the appropriate library directory, and copy the control file to the directory Postgres uses to track available extensions. Next we activate the extension by adding it to shared_preload_libraries:And finally, restart Postgres. Debian-based systems would use a command like this:Red Hat family systems would use a command like this:Or you can do it the old-fashioned way:Once Postgres restarts, the noddl extension is available and active. But remember, our GUC is disabled by default, so we need to enable it. Connect to Postgres as a superuser and issue these commands:Then attempt to execute literally any DDL statement:How rude! Fortunately we used for this GUC, meaning superusers can override it at the session level:Unlike an event trigger, it’s not possible to circumvent this extension by disabling the event_triggers GUC. If we remove the GUC, the only way to enable DDL again is to unload the extension itself. There are rare environments that require structurally static deployments, and this would be one way to enforce that restriction.

The Joy of Deploying pgEdge Using Ansible

Thu, 17 Jul 2025 04:19:00 GMT

Once upon a time, the Perl programming language earned the nickname “the Swiss Army chainsaw of scripting languages” from its users. In many ways, Ansible has attained a similarly lofty reputation for orchestrating infrastructure. Many experienced Postgres DBAs figured this out a long time ago, jealously hoarding collections, roles, and playbooks and yearning for a day when they too could sit back and watch their new cluster “compile”. We at pgEdge understand this compulsion all too well, extensively leveraging Ansible for internal testing purposes. It’s hard to beat the convenience of whipping up a few VMs and building a full active-active pgEdge distributed Postgres cluster within a few minutes, bombarding it with millions of transactions to reveal operational edge cases, and then lather, rinse, and repeat. Some time during this unsanctioned VM abuse, we realized everyone else might want to join in on the fun, and so pgedge-ansible was born.

What is pgedge-ansible

Frankly, pgedge-ansible does exactly what it says on the tin: it’s an Ansible collection built specifically to deploy pgEdge clusters. It provides configurable roles for spinning up a variety of pgEdge architecture stacks, from a simple two-node cluster, all the way to our 10-node Ultra HA design featuring six Postgres nodes, independent Patroni failover management, HAProxy traffic control, and pgBackRest backup management.While the pgEdge CLI platform tool has been around for a while and fulfills a similar role when combined with a JSON cluster definition file, there are certain advantages to including Ansible. Principal among them is the ability to provide more direct control over the full hardware and software stack. Ansible is designed to be flexible, either by using roles in playbooks exactly as intended, or by integrating various customizations the collection authors may not have anticipated.In true Ansible fashion, the pgedge-ansible role provides a minimum reference implementation and most parts may be readily swapped out to better match existing infrastructure. Don’t need HAProxy and have a role for configuring a common F5 load balancer or an AWS elastic load balancer (ELB)? Be our guest! Want to use Barman instead of pgBackrest for backup management? Please feel free. We want to make using pgEdge easy and convenient, and that means meeting the customer where they are.

How does it work?

We hope you’ll love this part, because it’s incredibly easy to use. We’ve included playbooks for two sample architecture designs in the collection:

A simple cluster consisting
only
of pgEdge distributed Postgres nodes.

A “batteries included” cluster with all the trimmings we like to call “Ultra HA”.

The collection operates by assuming every pgEdge node resides in its own “zone”. Something like this:This design makes it possible to extend the zone designation to far more sophisticated clusters like this:The simple cluster relies on only four roles:

init_server - Get the server ready to run a pgEdge cluster.

install_pgedge - Install the pgedge CLI platform software.

setup_postgres - Create a Postgres instance capable of running pgEdge.

setup_pgedge - Initialize each pgEdge node and wire the zones together for Active-Active logical replication.

The Ultra HA cluster requires a substantially longer list since we also have to orchestrate Patroni, etcd, HAProxy, and pgBackRest as well:

init_server - Get the server ready to run a pgEdge cluster.

install_pgedge - Install the pgedge CLI platform software.

setup_postgres - Create a Postgres instance capable of running pgEdge.

install_etcd - Retrieve and install etcd to act as a
DCS
for Patroni.

install_patroni - Retrieve and install Patroni to manage replicas and failover per zone.

install_backrest - Retrieve and install pgBackRest for backup management.

setup_etcd - Configure etcd to create a quorum within the node zone.

setup_patroni - Configure Patroni for its local etcd and pgEdge Postgres instance. This will also bootstrap the replicas.

setup_haproxy - Retrieve, install, and configure HAProxy based on the Patroni-managed nodes in the zone.

setup_pgedge - Initialize each pgEdge node and wire the zones together for Active-Active logical replication. This has to come after setup_haproxy because each zone needs to communicate through the proxy to make sure it always reaches the current pgEdge primary node in that zone.

setup_backrest - Configure pgBackRest to back up nodes in its zone, and take an initial backup to bootstrap everything. It will only back up the current Primary node.

That’s a lot of work to do to set up a cluster! But the beauty of Ansible is that it handles all of that. The roles even recognize three group names to keep inventory files simple:

pgedge - Nodes where pgedge is being installed or managed. These are automatically sub-grouped by zone.

haproxy - Nodes where HAProxy is running. These are also sub-grouped automatically by zone because each HAProxy will only perform node health checks for routing within that zone.

backup - Dedicated backup servers. These are also sub-grouped automatically by zone so each zone has its own set of backups.

Installation and use

Using the pgedge-ansible collection is pretty straightforward; we’re doing our best to keep everything compatible with Ansible best practices and intend to eventually submit everything to Ansible Galaxy.Installation is about what you’d expect:Once the collection is installed, using it only requires activating the collection and using the roles. Consider this inventory file:That’s two nodes, each with its own logical availability zone. Then we just need a playbook to deploy the cluster:And then we just need to run the playbook:Assuming everything went as expected, we should end up with an active-active pgEdge distributed Postgres cluster with DDL replication enabled by default. But don’t take our word for it, try it yourself!The work of a DBA is never truly done, and we hope this makes your life just a bit easier. Fully active-active Postgres clusters are hardly trivial to deploy and administer, but a good Ansible collection can certainly improve the situation considerably. There’s an exciting new world of running writable Postgres nodes “on the edge”, and now diving in is easier than ever before.Now’s your chance to experiment. Let us know if pgedge-ansible helped you bring your Postgres cluster to the next level.

Living on the Edge

Mon, 02 Jun 2025 15:00:00 GMT

The PGDay Chicago 2025 Postgres conference has come and gone, and so too has my presentation on conflict management in Postgres Multi-Master clusters. For those who couldn’t attend this year, there’s no need to fret, as this blog will present all of the relevant details. By the end, you'll be well armed with the knowledge necessary to identify, mitigate, and avoid the worst scenarios which may arise in a distributed Postgres cluster.

Is Bi-Directional Logical Replication Implicitly Safe?

The release announcement for Postgres 16 included this cryptic note:Despite how that may sound on the surface, that shouldn’t be interpreted as tacit permission or a resounding recommendation to immediately spin up an Active-Active logical replication cluster using native Postgres logical replication. Among other things, there are novel sequence rules, data merging concerns, conflict management, and the dreaded potential of node divergence.None of these situations are managed by default, and a problem related to any one of these issues could result in implicit or explicit data loss. Consider a cluster distributed to three distinct geographic regions:What might happen if Bob is in Ashburn, Virginia, and opens an account at ACME using his email address bob@smith.com? Probably nothing. Unless perhaps his wife is on vacation in Frankfurt, Germany and happens to be touring the ACME global office and decides to sign Bob up for an account because she knows he’s been meaning to do so. Through some providence of fate, this happens within the round-trip latency between Frankfurt and Ashburn, and now we have a problem. Which set of account credentials and related information do we retain?The answer to that question is why we’re here, and why solutions like pgEdge are necessary to successfully navigate the perilous seas of Multi-Master Postgres clusters. Let’s discuss the various types of complications we may encounter while operating a distributed Postgres cluster so we can do so safely.

In Theory

Those familiar with database cluster architectures may already be well acquainted with CAP theorem. Unlike ACID , which describes how databases like Postgres guarantee data validity, CAP is more focused on distributed operation within clusters. It goes like this:

Partitioned (distributed) clusters have either availability or consistency, never both. What happens if two nodes are disconnected? Either data may not be consistent between the two, or writes must be rejected or paused to avoid introducing inconsistencies.But this isn’t quite right because it doesn’t account for network latency, which is a kind of built-in partition dictated by the speed of network traffic. The PACELC design principle aims to solve this conundrum. Rather than focusing purely on availability and consistency, it augments CAP by adding an extra logic branch:

Choose between
L
atency or

Loss of
C
onsistency

If you need a mnemonic for this, consider the humble Pack Elk:Most Multi-Master cluster solutions choose the latter (lost consistency) as the default. Essentially Postgres nodes in such a cluster accept local writes and transmit those changes to other participating nodes via logical replication. Until that transmission is complete and other nodes integrate that information, there is a small window where the cluster is not fully consistent. This is where problems begin for the unprepared.A single Postgres instance doesn’t suffer from this indignity. There’s a single source of truth much like a chess board, a single state of truth dictated by the board itself.In an Active-Active Postgres cluster, it’s more like a flock of starlings, where each node is generally going in the same direction most of the time. It’s a beautiful orchestration of form where no single node truly dictates the fate of the cluster.This is ultimately what we’re trying to achieve when deploying a distributed Postgres cluster.

Disagreeable Outcomes

In a fully distributed Postgres cluster, the primary source of problems is the list of potential data conflicts, usually caused by giving preference to fast local writes rather than 100% consistency across the cluster. Such asynchronous operations lead to simultaneous changes which could be incompatible. The resulting conflicts make everyone sad, including the software components mediating the dispute.When it comes to Postgres, there are basically four categories of conflict that may arise:

Naturally convergent conflicts

Resolvable conflicts

Divergent conflicts

(Bonus) Phantom conflicts

The first step to avoiding a situation is recognizing it - let’s tackle them one by one.

Convergent Conflicts

A convergent conflict is what happens when the order of operation doesn’t matter. It’s an idempotent result where, even if nodes apply changes with total disregard, they’ll all reach the same end state anyway. And remember, a “conflict” just means two different nodes performed some operation on the same row(s).The first of these is an / conflict:

Update happened first? It gets deleted

Delete happened first? Nothing to update

End state: the row is deleted

Next is a / conflict:

End state: the row is deleted

Then there’s the / conflict:

Update happened first? The table gets truncated

Truncate happened first? Nothing to update

End state: the table is truncated

Similarly, the / conflict:

Delete happened first? The table gets truncated

Truncate happened first? Nothing to delete

End state: the table is truncated

And finally comes the / conflict:

End state: the table is truncated

All in all, these conflicts aren’t very interesting; they’re all dependent on either a single row being deleted, or the whole table being emptied. It doesn’t take much consideration to realize what will happen in the end.

Resolvable Conflicts

Then there are conflicts that can be resolved automatically by some default operation such as “Last-Write-Wins”, which assumes the most recent write should “win” the exchange on all nodes. This allows all nodes to remain consistent, but may result in data loss. Whether that overwritten row matters or not tends to depend on the application.
Let’s start with / conflicts:

Node A: INSERT ... (id, col1) VALUES (2, 10)

Node B: INSERT ... (id, col1) VALUES (2, 100)

Last "update" wins (default resolution method)

Result: one INSERT is discarded / overwritten

The conflict on the “id” column must be resolved, meaning we must discard either the value of 10 or 100, depending on the timestamp of the committing transaction. Will the transaction selected for discard make a difference?This also applies to tables which rely on multiple unique constraints, such as a primary key and a unique index on an email column. Consider this scenario:

Node A: INSERT ... (id, email) VALUES (1, 'bob@smith.com')

Node B: INSERT ... (id, email) VALUES (2, 'bob@smith.com')

Last "update" wins (default resolution method)

Results:

The row will be consistent across the cluster, but which row? Will it be the email address with an ID of 1, or 2? Did the same transaction also insert dependent rows in another table with a foreign key referencing this ID? Should this even be handled automatically, or should there be some kind of custom resolution method to account for such scenarios? If Jim and Jen both entered a forest at the same time and only Jen returned, what happened to Jim?The other types of resolvable conflict are both / variants. Consider this first example:

Node A: UPDATE t SET col1=100 WHERE id=4

Node B: UPDATE t SET col1=500 WHERE id=4

Last "update" wins (default resolution method)

Result: contents of col1 are discarded / overwritten

We can only retain a single update, but which one? If you imagine this is a bank app setting your total balance, it suddenly becomes a lot more relevant. But this can also go wrong in a more subtle way:

Node A: UPDATE t SET col1=100 WHERE id=4

Node B: UPDATE t SET col2='stuff' WHERE id=4

Last "update" wins (default resolution method)

Result: contents of col1 or col2 are discarded / overwritten

This happens because logical replication doesn’t just transmit the specific modification, it sends the entire resulting tuple. This is necessary because there could be triggers or other after-effects which further mutate the change being applied, so the only way to ensure an accurate representation is to capture the entire row contents at the end of the process.Let’s examine the entire process:

Old tuple on both nodes: {id: 4, col1: 50, col2: 'wow'}

Node A: UPDATE t SET col1=100 WHERE id=4

Logical replication sees: {id: 4, col1: 100, col2: 'wow'}

Node B: UPDATE t SET col2='stuff' WHERE id=4

Logical replication sees: {id: 4, col1: 50, col2: 'stuff'}

The receiving node must pick one or the other, not both; there’s no automatic “merge” possible here. Using a Last Write Wins strategy would ensure a consistent picture across the cluster, but what about the column values we lost? Again, two people entered the forest and one vanished without a trace.What happened to Jim?

Divergent Conflicts

Then there are conflicts that actually cause data to diverge on one or more nodes in the cluster on a permanent basis. Luckily this is usually caused by some kind of race condition between at least three nodes, so they’re much less likely to occur in the wild. Still, it’s important to know how they might occur so we can avoid triggering the surrounding circumstances.Let’s start with an / conflict between three nodes:

Node A: INSERT -> Node B

Node B: UPDATE -> Node C

Node C: ignores UPDATE

Node A: INSERT -> Node C

In this case, Node B received the INSERT before Node C, and subsequently updated the value. If that UPDATE reaches Node C before the INSERT from Node A, we have a problem. If Node C discards that phantom update for a row it doesn’t have, and later processes the INSERT, that means Node A and B have the updated value, and Node C does not. This is a node divergence. Once the cluster reaches this state, the only way to fix it is through manual intervention using some kind of data comparison tool like the Active Consistency Engine (ACE) from pgEdge.This can also happen in an / scenario in a 3-node cluster:

Node A: INSERT -> Node B

Node B: DELETE -> Node C

Node C: ignores DELETE

Node A: INSERT -> Node C

Now Node C has a row that is missing on both Node A and Node B.This situation is actually slightly worse than an INSERT / UPDATE conflict. A custom conflict management routine could convert the UPDATE to an INSERT using its original timestamp on nodes where the value is missing. Then when the original INSERT arrived, it would be skipped as being an older transaction. In the case of a DELETE, this isn’t possible because there is no way to modify a DELETE so it only affects the incoming INSERT.Then there’s a / conflict in a 3-node cluster:

Node A: INSERT -> Node B

Node B: TRUNCATE -> Node C

Node C: applies TRUNCATE

Node A: INSERT -> Node C

This time both Node A and Node B have no rows in the affected table, and Node C has diverged by retaining one rogue . There’s no easy fix for this situation either, as Node C has no way to know there’s a pending when the arrives. Network latency makes fools of us all.

Phantom Conflicts

The last type of conflict is technically not a conflict at all, but an application may treat it like one. It won’t show up in logs, it’s not due to latency, and it can actually happen on a single Postgres node outside of replication of any kind. So what is it?Consider this scenario:

App: INSERT; COMMIT -> Node A

Node A: COMMIT -> WAL -> Node B

Node A: Confirm !-> App

Node A: Crash

Node B: Becomes new write target

App: Connection aborted? Retry insert!

App: INSERT -> Node B

Postgres normally sends a confirmation message to the client after every transaction commit. But what happens if the node crashes during the confirmation and the client never receives the message? A good application would notice the error and try again. But thanks to that second attempt, a duplicate record now exists in the database.How is that possible? Postgres always writes transactions to the WAL before applying them to backend data files. These WAL files are also used for crash recovery. The client never received an acknowledgement that the transaction committed successfully, but it succeeded nonetheless. Since the transaction is in the WAL, it will be retained following crash recovery of Node A and any replicas in the cluster. Yet the application layer thinks it failed and will probably try again.The easiest way to avoid this is to rely on natural keys, or in the absence of those, a surrogate key that the application itself controls. Consider calling nextval() directly and supplying sequence values manually, or even relying on an external ID generator. Alternatively, the application retry loop should confirm whether the transaction failed with a SELECT prior to simply retrying.Remember, this can affect even a non-distributed Postgres cluster. Trust, but verify.

Preventing the Inevitable

How do we mitigate or even avoid these scary conflict situations? Luckily there are a plethora of ways to address conflict potential in a distributed Postgres cluster. Let’s start with the easiest one of all!

Don’t Do That

An ounce of prevention is worth a pound of cure, after all.Don’t do what? Consider the primary cause of conflicts: simultaneous operations occurring on the same data on different nodes. That happens because data may not have distinct structural or regional boundaries, or because a single application instance is interacting with multiple nodes simultaneously without regard for transmission latency.Thus the simplest way to avoid conflicts is to control write targets

Use “sticky” sessions. Applications should only interact with a single write target at a time, and never “roam” within the cluster.

Assign app servers to specific (regional) nodes. Nodes in Mumbai shouldn’t write to databases in Chicago, and vice versa. It’s faster to write locally anyway.

nteract with specific (regional) data. Again, an account in Mumbai may physically exist in a globally distributed database, but multiple accessors increase the potential for conflicts.

Avoid unnecessary cross-node activity. Regional interaction also applies on a more local scale. If applications can silo or prefer certain data segments on specific database nodes, they should.

To solve the issue of updates on different database nodes modifying the same rows, there’s a solution for that too: use a ledger instead.This is probably a much more invasive application-level change than most, but some systems already do this and as a result, are automatically a great fit for distributed Postgres clusters.

Solving Conflicts with CRDTs

The official meaning of CRDT acronym is “Conflict-free Replicated Data Type”, but that’s cheating, so I usually refer to them as Conflict Resistant Data Types. These data types were specifically designed for distributed clusters or databases with constant asynchronous write conflicts. The latter occurs when two different writers modify a single row, but each started with an “old” copy of the row. Think of a -> pattern.As a result, CRDTs can actually be useful even in non-distributed Postgres clusters. In the context of a distributed Postgres cluster, there are basically two types of commonly available CRDT provided by the Active-Active extension:

Apply a diff between the incoming and existing values

Use a custom data type with per-node “hidden” fields

Additionally, these two approaches only work for numeric columns, like , , , , and so on. And of those two techniques, there are also two methods for providing them to Postgres: by patching how Postgres handles logical replication for numeric columns, or providing dedicated data types.To understand how this works, let’s use a CRDT in the pgEdge Spock extension (a component of pgEdge Distributed Postgres):In this case, we’ve told Postgres that we need to track the previous value of the column, and that there’s a special callback function it should use when modifying values in that column. That function will ensure updates from any node in the cluster will add the difference between the value on the two nodes rather than the absolute value, resulting in a numeric merge operation.If we used the EnterpriseDB BDR extension (a component of EDB Postgres Distributed) instead, it would look something like this:This use case may appear simpler, but the BDR extension actually provides multiple CRDTs , each with its own unique rules, behaviors, and best practices. This also causes a certain amount of vendor lock-in, as it becomes impossible to uninstall the BDR extension so long as these column types are in use.In either case, a delta-based CRDT works like this:

Both nodes start with a tuple: {id: 5, total: 100}

Either node: UPDATE account SET total = total + 100 WHERE id = 5

New resulting tuple: {id: 5, total: 200}

Old and new values are sent to the remote node

The delta function or type calculates:

Here’s a scenario where that procedure specifically avoids a data conflict and also preserves the intended value across the cluster:

Initial tuple on both nodes: {id: 5, total: 100}

Node A adds 100, sends: {id: 5, total: 200}

Node B adds 400, sends: {id: 5, total: 500}

Node A: total = 200 + (500 - 100) = 600

Node B: total = 500 + (200 - 100) = 600

And indeed, an initial balance of $100, increased by $100 and then $400 would be a total of $600. This is exactly what we want, and the overhead from managing the deltas is generally minimal.There’s also an alternative technique of maintaining separate hidden fields within the column for each node in the cluster. The benefit here is that each node only interacts with its own assigned sub-record, and the data type handles merging on display or retrieval of the stored value. There’s never a risk of conflict or unexpected merge behavior because nodes are always interacting with separate values.Here’s how that works:

Initial tuple on both nodes: {id: 5, total: (100, 0)}

Total displayed as 100

Node A adds 100

Node B subtracts 50

New tuple: {id: 5, total: (200, -50)}

Total displayed as 150

If a third node is added to the cluster later, the CRDT will simply add another hidden field to the column when rows are updated by the new node. However, there's an important limitation to this technique: values can never be set to zero without using a special function. Consider this scenario:

Initial tuple on both nodes: {id: 5, total: (200, -50)}

Total displayed as 150

Node A sets total to 0

Node B does nothing

New tuple: {id: 5, total: (0, -50)}

Total displayed as -50

Since each node only interacts with its own dedicated sub-field, it’s impossible to reset the column value to zero without having every node also reset its data. The only real way to set such CRDT columns to zero is to use a special utility function that will essentially revert the column to an initial empty state. Once that action enters the logical replication stream, every other node will also empty that column. Applications which intend to use these types of CRDT would definitely need to account for this behavior.

Key Management

Surrogate keys are probably the most commonly leveraged technique for assigning unique values to primary keys in Postgres tables. However, this approach often assumes a single source of key values, or fails to account for the presence of other nodes. If two nodes rely on a sequence for assigning values, this sequence will frequently yield the same value on both nodes, causing constant conflicts and resulting in data loss during statements.There are effectively four methods for safely generating surrogate keys which are guaranteed to be unique across every node in distributed Postgres clusters:

Sequence offsets

Globally unique keys

Global allocations

External key generator

We’ll explore the first three options here, since it’s incredibly uncommon to leverage an external service to generate IDs rather than doing it locally or algorithmically.

Sequence Offsets

Sequence offsets are the most backward compatible approach because they rely on no outside technology or extension. They work like this:For every sequence on Node 1:For every sequence on Node 2:Afterwards, the sequence on Node 1 will generate values such as 1001, 1011, 1021, and so on, while Node 2 will not conflict, as its values will be 1002, 1012, 1022, etc.. The primary difficulty here is never forgetting to apply these modifications to all sequences on a node when it joins the cluster. New sequences will also require this treatment on every node. Further, the width of the increment must account for the maximum number of expected cluster nodes, and can never really be expanded afterwards. The ongoing amount of maintenance here is incredibly high.

Globally Unique Keys

It’s also possible to algorithmically generate surrogate keys in such a way that they’re guaranteed to be unique across the cluster. The most common approach already in use in many applications is through leveraging UUIDs. Postgres has a built-in function for this:Better yet, Postgres 18 is expected to have native support for UUID v7, which should improve compatibility with BTREE indexes.Another approach is to use a Snowflake ID, an algorithmic technique that combines a timestamp, node identifier, and a millisecond-scale sequence, popularized by sites like Twitter and Instagram. The pgEdge Distributed Postgres stack includes the Snowflake extension, which provides alternative nextval() and currval() functions for generating these kinds of values.The primary drawback when using these values is that they’re always 64-bit integers. This is fully compatible with the Postgres type, but applications may require explicit modification to support such large 64-bit values, such as JavaScript and its BigInt type.

Global Allocations

The last technique we’ll discuss is the use of globally allocated sequence ranges. In this approach, each node requests a contiguous block of values from the other nodes in the cluster, and they all agree on block assignments. As a result, Node 1 may have two 1-million row chunks from 1-2,000,000, Node 2 uses 2,000,001 to 4,000,000, and Node 3 has 4,000,001 to 6,000,000.Nodes retain two blocks so they can request another block once they start consuming the second. INTs may default to smaller chunks such as 1-million, while s can reasonably assign larger 1-billion row chunks.This process is something a distributed cluster extension can do by hooking into internal Postgres extension hooks. But it also means there’s a maintenance process and a consensus aspect for distributing and using assigned blocks. On the other hand, unlike Snowflake IDs, there’s no potential for number length incompatibility, and unlike sequence offsets, there’s no need to manually modify each sequence on every node.

Sequence Safety

While we’re on the topic of sequences, we need to look at how Postgres uses them internally. We’ve all been told not to do this:Why? Because there’s a lot of hidden behind the scenes magic that doesn’t mesh well with current SQL standards. This is actually what happens when using SERIAL or BIGSERIAL in Postgres:Postgres creates an implicit sequence and associates it with the table. Usage grants to the table do not cascade to the sequence, and the value is supplied by a weak “default” clause. As a result, many new applications and schemas use IDENTITY syntax instead:Now there’s no more magical nextval shenanigans, the word ALWAYS means what it says, and it’s standard SQL. There’s even extra syntax to set the starting point, increment size, and other attributes. It’s great, right? Well maybeWhat happens in Active-Active clusters? We can no longer explicitly substitute our own nextval function, so we may be stuck with the monotonically advancing sequential values provided by the identity. It means we can’t use an alternative ID generator such as snowflake, timeshard, galloc, or anything else unless the distributed cluster extension can override the appropriate identity calls. The application definitely can’t do it with something as simple as the Snowflake extension.Ironically the new and recommended approach which is perfect for a replicated Postgres cluster is actually too inflexible for a distributed cluster. It may even be necessary for application schemas using IDENITITY to reverse migrate to , , or using explicitly.

Fixing Divergent INSERT / UPDATE Conflicts

When we discussed the topic of / conflicts, we noted that Node C would ignore an UPDATE if it arrived prior to the dependent . One way to fix that would be for the distributed Postgres extension to convert the to an . Then when the original arrives, its earlier timestamp means it loses the Last Write Wins battle, and the risk of divergence vanishes.The only caveat here is that updates don’t normally include TOAST data unless the toasted value itself is modified. This means the distributed Postgres extension must account for this and manually detoast the column value while performing the logical decoding step. If the extension doesn’t perform this translation, the only other option is to declare the entire row as a replica identity like this:This forces Postgres to retrieve the toasted value and add it to the logical replication stream automatically. This may affect logical replication efficiency if there are a lot of large text fields, but it facilitates the to conversion necessary to avoid this type of conflict.

Historical Relevance

Finally, there’s a technique for addressing divergent conflicts involving statements which no current distributed Postgres solution provides. Postgres Multi Version Concurrency Control works by retaining old row records in the table heap. We ran into a divergent conflict earlier in this article because a node has no way to reject an INSERT for a previously row. But what if we did?Consider how the hot_standby_feedback configuration parameter works. It allows a physical standby node to inform the primary of the lowest visible XID for any active transactions. The primary will then refrain from performing destructive maintenance on these visible tuples. A distributed Postgres cluster should know the timestamp from the oldest transaction from any other node in the cluster. What if it used that value to prevent similar cleanup across the cluster?This “tombstone” record would follow the same rules as Postgres transaction visibility normally would, but would allow the distributed Postgres extension to physically examine the older record if present. If a removes a record, but there are pending transactions which might need to see that delete, the tombstone would make that possible.What if the arrived before any ? Rather than discarding the event, the distributed Postgres extension could insert it purely as a tombstone record. Once all transactions older than the have replayed, the tombstone snapshot would move to the next outstanding XID the same way as hot_standby_feedback.Barring this kind of advanced automatic handling, the easiest way to avoid divergent conflicts caused by or is to avoid executing either. It’s actually fairly common for databases to never delete data, electing instead to use a technique called a “soft” delete. A soft delete is simply something like this:In the event of a system audit, the record is still there, its old contents are still available, and there’s no risk of a divergent conflict because the existing record also acts as a tombstone. Some DBAs and data scientists actually strongly recommend always doing this even in non-distributed Postgres clusters. Having a reliable audit-trail is sometimes more important than having a pristine database with no “unnecessary” records.

Final Words

Managing a distributed Postgres cluster is hardly a trivial commitment. But it’s certainly less daunting when armed with the right product such as pgEdge Distributed PostgreSQL, knowledge of the possible conflicts and related edge cases, and a clear understanding of the best mitigation techniques. Any application stack can excel in a distributed cluster environment by following certain precautions.As always, the power of Postgres makes it possible!