pgEdge Posts from Andrei Lepikhov

EXPLAIN Prettier, or Post-Processing Query Plans in Postgres

Tue, 26 May 2026 11:22:00 GMT

This story started with a book gifted by a colleague. Reading Jimmy Angelakos' «PostgreSQL Mistakes and How to Avoid Them», I realised something that had been bugging me - in Postgres, the EXPLAIN command produces far too much information. The examples that authors typically present when discussing various aspects of database systems make it harder to analyse the problem at hand and distract the reader. That's how the idea of a post-processing for EXPLAIN output was born - to make query plans more readable and problem-focused.

Information Overload

Anyone who has worked with PostgreSQL knows the command, or more precisely . It is typically used to investigate query performance issues or demonstrate optimisation techniques. But there is one problem: its output is packed with highly specific information. For instance, the parameter is rarely needed when analysing EXPLAIN output. Some fields, such as , take up a lot of visual space, are system-dependent, and frequently unnecessary - yet if we want to see , we have to run EXPLAIN with , and inevitably comes along for the ride.So, is it really a big deal to study an EXPLAIN with slightly more information? Ha! Let’s look at a typical query plan from my investigations - here, for example, are two plans for the same query: one is the bad plan, and the other is the good one.Finding problems in such a large plan takes time, and every extraneous detail makes it harder to spot the problematic decision. Sure, with the rise of AI agents, I can just ask Claude to compare a pair of plans, highlight the differences, and analyse what’s wrong. But this doesn’t always work - either there are too many details, or automation is needed across a large stream of queries - so the problem remains.

Regression Test Stability

EXPLAIN output changes between Postgres versions, and if your extension supports 4-5 recent versions, then your tests must pass on each of them. This means we need to filter EXPLAIN output to guarantee stable test runs across different hardware and software configurations. Maintaining alternative expected output for tests can quickly turn into a nightmare.For example:But on a different machine or version, this EXPLAIN looks slightly different:The CI/CD pipeline fails. Not because the plan changed, but because internal differences between software systems broke the exact string match.Here’s another simple case. Take a trivial query with sorting:The value is platform-dependent. On a different machine with a different allocator or memory alignment, you’ll get or . And there is no option in to suppress this field - it is always printed when a Sort node executes in memory. The same goes for Hash nodes, which display , , . You simply cannot ask PostgreSQL to “show me the plan without memory details” - no such setting exists.

Naming Phantom Objects in Query Plans

Another example of cross-version instability is SubPlan naming. In PostgreSQL 17, the display format changed for SubPlan and InitPlan nodes in EXPLAIN output. Previously (PG 16 and earlier), InitPlan was displayed with a suffix, and references to its result looked like :Starting with PG 17, the suffix was removed, and references changed to the format :The situation with SubPlan is similar - the suffix was also removed, and parameter references were changed to . But SubPlan has an additional change: in PG 16, the filter line showed just without comparison details, while in PG 17 the full expression became visible - the operator, the ALL/ANY keyword, and the specific column:The changes are sensible - it’s now clearer what’s happening in the plan. But for tests, it’s still a disaster: the same query on PG 16 and PG 17 produces different textual output. And in fact, such fluctuations extend to all kinds of phantom objects - entities that don’t exist in the database or in the original SQL, yet appear as data sources after query plan transformations - see, for example, .EXPLAIN post-processing can stabilise the external representation of a query plan: normalise both forms to a single representation - strip phantom object names from InitPlan/SubPlan lines, bring references to such objects to a single stable format, and unify the display in filters so that the result is version-independent.

Query Plans for Articles and Documentation

When demonstrating the advantages of BitmapScan, for example, there’s no need to burden your reader with extraneous information such as or . Moreover, space in a book is physically limited by the page formatting - you end up shrinking the font just to fit a moderately complex plan.Imagine: you’re writing an article about a new PostgreSQL query optimiser feature and want to show before-and-after plans:Memory sizes, buffers, and cost estimates - this clutters the overall picture and will differ on the reader’s machine anyway. What actually matters here is one thing: how the plan structure changed. It should be clean and compact. The reader should see the essence of the optimisation, free of noise.You could turn to AI to generate “clean” plans for publication, but there’s no absolute trust in its output, nor any guarantee that it will verify each plan against the correct DBMS version and configuration.That’s how explain prettier came to be. The original trigger for its creation was the need to stabilise tests for the pg_track_optimizer extension. The main purpose of pg_track_optimizer is to examine query plans, so it naturally includes numerous regression tests that compare EXPLAIN listings. This feature was integrated directly into the extension’s interface and allowed us to reduce the number of alternative test outputs and prettify tracking query plans.

How It Works

This PL/pgSQL script provides two functions. The first - - executes a query and post-processes its EXPLAIN output. It is primarily intended for test stabilisation. The second - - accepts and processes an existing EXPLAIN output in text format, primarily intended for incident investigation when there is no access to the server or the data to run the query.By default, everything platform-dependent is hidden: memory allocations ( becomes ), worker counts ( becomes ), trailing decimal places in row counts ( is simplified to ), and actual time values are rounded to integers. Buffers, hash memory allocation, and other implementation details are removed. And so on. At the same time, everything important is preserved: the plan structure - nodes, joins, scans; filters and conditions; row count estimates. Phantom object standardisation is not yet implemented.

Controlling the Output

The function allows you to define EXPLAIN parameters by passing your own options string in the argument. The main settings are:

- a flag that hides all platform-dependent data from the output

- hides lines describing execution details of each node. Such data may or may not depend on configuration. The main idea here is to remove supplementary information while keeping only the principal structure of plan nodes.

,
,
- flags that hide specific details from the main node description line in
output.

All flags default to - for maximum conciseness and stability. But if more detail is needed, it’s easy to adjust:

Example

Let’s see what post-processing gives us. We’ll filter the EXPLAIN ANALYZE example from above - the two plans with HashJoin and NestLoop. The prettier result is:Looks a lot more readable, simpler, and more compact, doesn’t it?

Quick Start

You can find the code for pretty_explain in my GitHub repo. It is designed as a plpgsql script that installs interface functions into your Postgres database: , , and two utility functions: and .After installing the functions, you can use them in a regression test:For copy-pasting from psql:Or, for plan comparison:

The State of the Community

This post would be incomplete without addressing the vanilla PostgreSQL approach to this problem. With a fairly large regression test suite, the community has repeatedly had to partially mask the EXPLAIN output. A simple code search reveals functions like , , and that help manage output. These functions are scattered across tests as ad-hoc solutions. As a result, developers of new features - and even more often, extension developers - have to invent their own masking functions, which feels unsystematic. I would prefer a single-core function that handles all such tasks.

Binary, ternary, or is it actually quaternary logic in Postgres functions

Mon, 18 May 2026 06:22:16 GMT

How an attempt to introduce a new optimisation ran into the problem of preserving ERROR-freedom of PostgreSQL functions.

TL;DR

A fundamental principle of query optimisation is that the result must not depend on the plan. PostgreSQL enforces this in part by tracking function volatility: volatile expressions restrict where the optimiser may push predicates and sort keys. But volatility isn't the only way an expression can be plan-sensitive - a stable function can also throw a runtime error on inputs that a different plan would have filtered out.In this post, we explore the issue through the lens of a concrete optimisation - outer join sort pushdown - which enables pushing Sort nodes below Joins to benefit queries. The problem it exposed, however, is more general and already latent in the core planner. We outline a solution by extending PostgreSQL's existing planner support function (prosupport) machinery with a new request type that allows functions to declare themselves unsafe for early evaluation. We'd welcome feedback and criticism from the community on this approach.

Plan Independence and Function Volatility

A query's result must not depend on how the planner chooses to execute it. PostgreSQL's primary tool for guaranteeing this is function volatility. Every function is classified as , , or : Volatility controls what the optimiser is allowed to do. For functions, the key rule is: the optimiser must not move the expression to a place that changes how many times it gets evaluated. A call to random() must run once per row - collapsing it into a single pre-evaluated constant or feeding it into an index condition would change the result. and expressions don't have this restriction: since their value doesn't change (within a statement or ever), the optimiser can safely evaluate them once and reuse the result, push them into index scans, or hoist them out of loops.A quick example makes the difference visible. Say we have a table with a timestamp index: A filter using (a STABLE function) can be evaluated once and fed straight to the index - the optimiser knows it won't change mid-statement:But throw in (a VOLATILE function) and the picture changes. Now the expression is different for every row, so the index is useless - the optimiser has to scan everything and check the filter at runtime:This works well for what it's designed to do: stop the optimiser from reusing values that change between calls. But it has a blind spot.

The Blind Spot: Stable Functions that ERROR

Take a type cast like . It's IMMUTABLE - the conversion from text to integer is deterministic, with no dependencies on the environment. Not volatile, not set-returning, parallel-safe. Every optimiser safety gate says "go ahead."But this cast is what we'll call a partial function (borrowing from mathematics): it throws a runtime error when the input isn't a valid integer (e.g., ). Volatility says nothing about this. The optimiser treats the cast as freely movable - and can push it somewhere it finds inputs the original plan would have filtered out.This isn't a theoretical worry. It already happens in vanilla PostgreSQL via plain-old predicate pushdown. Here's an example:The subquery joins raw_data with numbers, producing only . The outer query applies to the join result. Reading the query as written, you'd expect the cast to never see - the join should filter it out first. (The SQL standard doesn't actually guarantee this - evaluation order is implementation-defined - but the query text strongly suggests it.)But the optimiser pushes the predicate into the raw_data scan:That filter hits every raw_data row - including . Boom:ERROR: invalid input syntax for type integer: "42.1"This is the classical relational algebra equivalence σ_p(R ⋈ S) = σ_p(R) ⋈ S. It works when p only touches columns of R and is a "total" predicate - defined for all inputs. When p is "partial" - when it can error on some inputs - the equivalence breaks: the left side errors on rows that the right side would have quietly filtered away.The optimiser has no way to know this. The cast is IMMUTABLE, the predicate only references raw_data columns, and the pushdown is perfectly valid under standard relational algebra. The trouble is that the algebra assumes all expressions are "total".This isn't just a PostgreSQL quirk. The SQL Server team documented the same problem back in 2006 in their blog post "Predicate ordering is not guaranteed": a view that converts varchar to int only for certain row categories blows up when the optimiser evaluates the conversion before the category filter. Their recommended workaround - use CASE expressions to guard the conversion - is essentially a manual way for users to enforce "totality" at the query level. The problem is well known across database engines; what's missing is a systematic solution.

A New Optimisation that Widened the Gap

We ran into this while implementing and testing a new optimiser feature (branch: https://github.com/danolivo/pgdev/tree/enforce-presorted-scan-on-query-pathkeys) that expands the plan space for joins. It targets queries with and on the outer side of a join: sometimes it's cheaper to pre-sort the outer side and let the join produce at most N rows. This pattern is reportedly common in queries generated by ERP or CRM platforms, and SQL Server already plans it efficiently.The idea: Perform the sort before the join.Using the same tables, consider an over a join:Normally, PostgreSQL sorts the entire result, then grabs 10 rows:Our optimisation pushes the Sort below the Join. If the inner side is small or efficiently indexed, a NestLoop with a pre-sorted outer wins - and the can propagate down, enabling a top-N heapsort:When uses a plain column like , this is perfectly safe - the Sort just reorders rows the Scan was already producing. Faster query, same semantics. Everybody wins.

Where it Breaks

But what happens when uses a "partial" expression?Without the optimisation, this works fine: the INNER JOIN drops (no match in numbers), and the Sort only sees . With the optimisation, the Sort gets pushed below the Join - now it evaluates on every row, including , before the Join gets a chance to filter it out:ERROR: invalid input syntax for type integer: "42.1"Same "partial"-function problem as with predicate pushdown, just triggered by a different transformation. The original plan would have eliminated the bad row before the cast ran, but the new optimization broke that by evaluating it earlier.

Why this Matters in Practice

These examples might look contrived, but they point to a real risk for production systems. A query that works today can start throwing errors after a PostgreSQL upgrade - not because anyone changed the query or the data, but because the optimiser got smarter.Every major release brings new optimisations: join reordering heuristics, incremental sort, memoise, and partitionwise joins. Each one gives the optimiser more freedom to rearrange the plan tree. A query that survived for years because the old planner always put the Sort above the Join might blow up under a newer planner that's clever enough to push it down. The user sees a stable query break on upgrade, with zero schema or data changes - and no obvious explanation.That's not a great look for a DBMS. Whether a query succeeds or fails shouldn't depend on which optimisations the planner uses. When it does, users lose trust in upgrades and start pinning plans or disabling features, which defeats the whole point of having an optimiser.To be fair, the SQL standard explicitly permits implementations to differ on whether a query errors or succeeds, depending on evaluation order. That's a remarkably honest admission - but from a user's perspective, it's cold comfort. The gap is real enough that even major DBMSes have to publish workaround guides to help users protect themselves from optimiser-induced errors.

A Missing Dimension in Function Classification

The predicate pushdown and sort pushdown examples are really the same problem in different clothes. The optimiser applies transformations that are valid under standard relational algebra, which assumes expressions are "total" - but breaks when expressions can error.PostgreSQL classifies functions along one axis: volatility. But there's a second, independent axis: "totality".In mathematics, a "total" function is defined for every element of its domain. A "partial" function may be undefined - i.e., it errors - for some inputs:You might wonder: isn't "partial" just "more volatile than VOLATILE" - a fourth step on the volatility ladder? It's not. Volatility is about value stability: IMMUTABLE means "never changes," STABLE means "constant within a statement," VOLATILE means "might differ on every call." Each step tells the optimiser when it may evaluate the expression - how aggressively it can cache and reuse."Totality" asks a different question: can the evaluation blow up? And these two questions are genuinely independent:

If "partial" were just "super-volatile," the optimiser would refuse to touch these expressions, treating like . But that's way too conservative. The cast is deterministic; the optimiser should cache it, use it in index conditions, and pre-evaluate it with constants. We just don't want it moved somewhere it sees unfiltered data.Bottom line: volatility controls when to evaluate (once, per row, or per call?). "Totality" controls where to evaluate (above which filters?). They're orthogonal dimensions, not points on the same scale.

What a Solution Might Look Like

PostgreSQL already has a mechanism designed for exactly this kind of problem: planner support functions (prosupport). Every entry can optionally point to a support function that the optimiser calls at planning time to ask function-specific questions. Today, support functions handle tasks such as custom selectivity estimates, cost overrides, row-count estimates for set-returning functions, and simplifying function calls.The infrastructure is already there. A new request type, , would let a function tell the optimiser: "don't evaluate me on unfiltered data."The natural place to call it is in - the function that already serves as the central gatekeeper for "can this expression be evaluated at a lower plan level?" It already checks for volatility, set-returning functions, and parallel safety. A call would slot in as a fourth check:What makes this elegant is that is already called from all the right places:- Sort pushdown () - our optimisation- Gather Merge paths () - parallel plans- FDW/custom-scan paths ()So a single check in one function automatically protects every code path. And the same walker could be called from to protect predicate pushdown too - closing the gap we demonstrated with the vanilla PostgreSQL example.The approach has several advantages over a static catalogue column:

No schema change - no new
pg_proc
column, no catalogue version bump.

Incremental adoption - you add support functions to known "partial" functions one at a time. Functions without one keep the current behaviour.

Extensibility - extensions can register support functions for their own "partial" functions.

Custom Properties for PostgreSQL Database Objects Without Core Patches

Wed, 07 Jan 2026 06:20:45 GMT

Working in development, there is a common challenge: how to attach custom metadata to database objects without modifying Postgres's core code. In this article, I briefly demonstrate a practical solution using Postgres's SECURITY LABELS mechanism to implement custom properties that are transactional, properly linked to database objects, and work with standard Postgres operations.

The Problem: Managing Replication Conflicts

Consider a two-node setup where two active nodes replicate data to a third node. Such configurations are prone to UPDATE/UPDATE conflicts. While resolving replication conflicts is always a complex and generally unresolved task, certain columns lend themselves to simpler approaches.For example, imagine a banking application with account balances. If operations on the balance column are limited to increments and decrements, you can use a delta-based approach: instead of choosing between conflicting absolute values, apply the delta (change amount) from both updates. This requires only basic validation (overflow checks, ensuring balance ≥ 0) and guarantees all updates are accounted for.The challenge is: how do you mark specific columns to use this delta-based conflict resolution if Postgres doesn't support it? Ideally, we'd want something like:Such a level of integration with Postgres's SQL dialect is probably difficult to achieve and hardly necessary for portability. Still, it would be nice to have the ability to set a parameter using a call like:Proposals for customising properties of particular objects have been discussed in the community several times (see, for example, here and here), and core patches have even been proposed. However, their code is quite substantial, and the use case is narrow (or at least appears to be at the moment), which calls into question the practicality of merging them into core. Moreover, sometimes you want to attach a property not to a table but, say, to an index, large object, or data type - and that's a matter of time, code complexity, and the fact that an accepted patch would only change the behaviour of future versions of the DBMS core. But what if you need this functionality today?

Requirements

To understand how to do this, let's first define the requirements for the functionality:

- Lifecycle properties must have an internal dependency on the database object. For example, when you delete the parent object, your object should be deleted along with it when you invoke a DROP... CASCADE command.

Object properties must have some association with our extension so the DBMS can correctly handle the situation where the extension has been removed from the system.

- Object properties must behave transactionally and follow visibility rules. That is, when modified simultaneously in parallel transactions, the new property value should be visible only within the transaction until commit and should revert to its original state on ROLLBACK.

and dump/restore should correctly transfer properties along with the database.

Of course, we'd like not just transactional but also session-level behaviour, similar to Postgres GUCs - but implementing that would be much more difficult.

Critics

Using a simple global hash table with the object's OID as the key doesn't suit us, since the parameter value can be variable-length (e.g., a string), and implementing the link between the object and the property is quite complex. But most importantly, how do we ensure transactional behaviour and MVCC?The only simple way to satisfy our 3rd requirement is a database table. Perhaps we should create a table as part of the extension and add properties to it, with the extension extracting the value from this table? That's an option! However, having extensive experience using tables in extensions (see, for example, AQO and sr_plan), I can say that this approach is quite dodgy - it causes various problems. And that's not just the challenge of upgrades, dump/restore, and replication, but also the constant need to check that the object exists in the database. Plus, there's the overhead of lookups…

Implementation Details

My specific task was to create a 'delta_apply' property for a column of an arbitrary table. This property is intended for managing logical replication and should define the conflict-resolution mechanism for UPDATE/UPDATE conflicts involving numeric values. If this property is set, the subscriber, instead of using various conflict resolution strategies, calculates the delta between the new and old column values and adds it to the current value on the subscriber.For this parameter, all the above requirements are relevant; to satisfy these requirements in PostgreSQL, we find only one mechanism - SECURITY LABELS. While we may call them SECURITY labels, nothing restricts us from using them for non-security metadata. Just be aware:

Security-focused tools may inspect or validate labels.

We need to use a distinctive provider name to avoid conflicts.

How does it work? Let's look at the picture:Implementation of Postgres DB object properties based on the SECURITY LABELs mechanism.The solution uses the pg_seclabel system table, whose records are, by design, linked to the database objects. Adding, updating, and deleting records in this table is done via the 'SECURITY LABEL ...' UTILITY command, which allows monitoring this process in the extension using a utility hook.The SECURITY LABEL tool allows marking objects of many types, including VIEWs and even functions, and this should be sufficient for most tasks. The label record contains a reference to the object's OID and the object class - table, attribute, function, etc. - which is quite convenient. The label's text field lets us attach any data we want to the given object, and the 'provider' field lets us distinguish labels created by a specific module.To minimise overhead (going to the table every time is expensive, and a system cache for the pg_seclabel table hasn't been added yet), we add a cache in the form of a local hash table:Invalidation of entries in this table are managed by registering a RelcacheCallback, whose job is to mark the entry as invalid in case of any ALTER TABLE call to our object.The logic for adding entries to the cache can be implemented differently depending on the expected use cases. For my task, it's sufficient to add an entry to this cache when accessing the object in a hook (if the call comes from core) or in the extension's UI function (if the action is user-initiated).

Nuances of Implementing Property Addition and Deletion

To keep local caches of our properties consistent, we need to send invalidation messages for relevant objects on each change, and in each backend, receive these messages and clear the corresponding cache entry. If the change happens to the object itself, then, as mentioned earlier, we handle this event in the RelcacheCallback and mark the entry as invalid. But what if we're changing the property itself, which is actually just DML in the table? How do we notify all other processes in time?Postgres has a tool for sending invalidation messages to other objects: CacheInvalidateRelcacheByRelid. It's convenient, but limited to objects from . If the property is assigned to, say, a data type, this function won't help. So in practice, when adding or deleting a security label, I trigger an update of the object itself without actually changing anything in it, and the corresponding invalidation callback marks the entry in the extension's local cache.The extension provides the user with an interface via the function, which assigns a SECURITY LABEL to the specified object. The label description specifies the object property value, for example: . The callback, implemented in the extension for each label creation, controls the correctness of the object and the property being set.Text description of the label leaves a wide range of possibilities; for example, it allows you to include complex property logic stored in a JSON description inside the field.Thus, the client and extension gain a new communication tool that's fairly native, even though it's implemented outside the core. The extension now has the ability to check whether the user has assigned a property, and change behaviour accordingly.In the delta_apply feature code, this is used as follows (see, for example, commits 67bf955, 09d5d7e). The logical replication subscriber process receives an UPDATE record containing the old and new values of a table row. When opening the table, we simultaneously access the delta_apply property cache. If the table includes one or more such 'incremental' attributes, for each such attribute, the delta between the old and new incoming values is calculated and added to the value in the subscriber's database. Even if, according to the conflict resolution strategy (e.g., last-update-wins), a decision is made to reject the incoming update, changes to the delta_apply columns will nevertheless be applied. This reduces the likelihood of conflicts between records, and ensures that all updates to the incremental column are accounted for.

External interference

Remember, pg_seclabel is still a catalogue table. As a result, any user with sufficient rights (DBA) can change the table, and attempt to create/alter/drop a security label manually.To avoid this issue, I propose introducing an internal GUC in the extension, for example . The GUC would be set to at the start of any extension UI routine and reset to when the routine completes. The extension can then check the GUC value before and after execution and take corrective action if the state does not match the expected lifecycle.There is a possibility that when a superuser creates a routine, they might access this parameter and break the guard manually. Of course, we can resolve the issue by assigning a hook on this parameter, but it already looks like we're paranoid and overengineering…