The answer I got was disarming: 'Postgres is just good enough for our purposes.'

Think about what that means. Open source, no vendor lock-in, and you can always find someone who knows the codebase and can reliably fix a problem or scale the load. Good enough beats brilliant. Significant part of database systems market doesn't care about superiority — they want simplicity, maintainability, predictability, and reliability.

That sounds like a trivial observation. It isn't — not if you are the one trying to get a new, pioneering, sometimes provocative feature into Postgres core.

The uncomfortable logic

Postgres holds mission-critical data, so efficiency and scalability are only second- or third-order concerns. And no code is free of bugs: every line you add is a new bug and a maintenance liability the whole community inherits. So the bar is brutal, and it applies in this order — your feature must first prove safety and the absence of regressions, then provide documentation, and only after all that does the gist of the idea finally get to take the stage. A feature survives that gauntlet only if users are already hurting for it: if you can show they genuinely struggle with a problem it solves.

If your instincts were formed between 2000 and 2015, when people boldly shipped massive, breakthrough features, this comes as a revelation. And if you just want an easy commit to prove your database technology, Postgres core is the wrong venue — pick a project where stability matters less, probably something from the OLAP area.

So what do you do with a pioneering idea? Prove it in a fork first.

A fork lets you see your code actually working — under real load, in a real DBMS, with real users hitting real edge cases. That is also, incidentally, a well-defined role for the Postgres enterprise companies I was sceptical about not long ago. Let me show you what that path actually looks like, because I have walked it.

Parallel queries and temporary tables

The brightest example for me is the parallel execution of queries involving temporary tables — something that, for a long time, simply hasn't been possible in core. Temporary tables are heavily used in CRM systems that need to hold uncommitted changes on the server side, and I have frequently watched a parallel plan collapse the moment a temp table appears in it.

It took three attempts to design parallel scan on temp tables.

Attempt one — the Postgres Professional fork. Formally, the table's definition, all necessary temp types, indexes and functions - even its data pages on disk - are visible to any backend, since the catalogue entries reside in the shared system catalogues and the files are stored on shared storage. Only the dirty pages that haven't been flushed yet are private, sitting in the owning backend's local buffers. So the trivial idea was to flush those buffers and then let background workers scan the table safely as part of a provably read-only query. I implemented exactly that: when the planner spotted a temp-table entry in the plan, it forced the executor's Gather node to flush the local buffers belonging to that table before scanning.

It worked — until sophisticated benchmarks in a pre-production test environment revealed that TOAST tables, indexes, and more are also touched, and their buffers need to be flushed. Worse, a temporary table can be referenced from many parts of a query: an expression, a subplan, or a LIMIT clause are just a few trivial examples. It looks obvious in hindsight, but at the time the problem was buried under the complexity and uncertainty of such a massive task. That fundamental flaw broke the approach.

I don’t know exactly what implementation they run in production. I hope that, with all their users and testing facilities, they eventually found a much better approach.

Attempt two — the Tantor Labs fork took a completely different route. First, a series of experiments showed that flushing a temporary buffer is not that expensive — only about 20% more than reading the page back from disk. Since the memory allocated for local buffers is usually much smaller than for shared buffers, it makes sense (and safe) to flush all dirty buffers at once. Next, the core parallel-safety machinery was extended with a new category: 'safe as long as you flush local buffers.' We then used it in the parallel-hazard detection code to pinpoint the exact node where the temp table is used, added the flushing cost to the Gather estimate, and let the optimiser decide on the best strategy. The first Gather node to execute flushes all local buffers on its first run and enables a safe scan. Such a lazy approach adds overhead only when strictly necessary.

Then real life exposed the design flaws yet again. Even with the buffers provably flushed, a read-only query can still change hint bits or prune dead tuples on a page — making it dirty and forcing another flush. So I added a safety net and disabled pruning and hint-bit updates whenever a temporary table is scanned in parallel.

That patch-on-a-patch is the real lesson. Scanning the same file in parallel is still fundamentally unsafe; if we commit such an approach into core, we will keep paying rent in code complexity and stability every time we touch it. It is doable and usable — but the honest conclusion is that the better fix is to stop fighting local buffers altogether and move temporary-table pages into shared buffers — the direction Konstantin Knizhnik explored years ago in his Global temporary tables proposal.

In theory, building your feature as a pure extension sounds ideal. In reality, though, it often works out poorly. Users tend to avoid custom extensions that lack a track record for stability and safety. Cloud providers usually stick to their own internal code, since they pay developers to handle upgrades, error reporting, replication, and other common issues. AI agents help by automating many routine production features, but they still aren't trustworthy enough.

The part I still can't crack

The fork approach works well enough. I've now seen a new idea run under real load, fail in useful ways, and clearly show promising long-term design. That's step one, and it works.

Step two is harder: getting a proven idea back into the community's field of view.

The main challenge is simply capturing attention. The community's pulse lives on the mailing list — that is where people engage to move a feature forward. But with the sheer volume of topics and emails, the long discussion history, and insider terminology, joining those conversations — and getting others genuinely interested in your feature — can feel like a full-time job. It is not really a path for researchers seeking to showcase an algorithm or a data management technique: such proposals often receive no feedback at all. It almost looks like an unspoken strategy for keeping the project's entropy in check — the barrier stays high enough for the approach to survive, but clearing it takes real, sustained effort.

That's not an issue if a committer or company backs your project, or if you can attend the PGConf conferences and pitch your idea at an unconference session, face-to-face with the community. But I'm talking about the typical, fully remote case — where you've never met the other contributors in person and know one another only by their footprint on the Internet.

My current approach is to publish the proposal early on the pgsql-hackers mailing list. From my experience, there is no reason to expect deep discussion or even a reply, but this way it's visible to search engines, AI tools, and the next developer who faces the same problem. Mention it everywhere - in your posts and talks. And of course, continue to improve your patch, reporting again and agin in the thread. The hope is that by keeping it visible, you'll eventually get real feedback, and someone else stuck on the same issue will find your thread and move it forward instead of reinventing it alone.

But I can't prove that part works yet. So here's my real question for you: if you've managed to turn a quiet mailing-list proposal into real community momentum, in a world this spread out and remote where you never meet most of the people you want to reach, how did you do it? I'd really like to know.

THE END.
Melbourne, Australia. July 26, 2026.

My hunch is that, as usual, we are held captive by our own experience and use AI in the ways we're accustomed to. Text chat, for example, remains the primary way of interacting with AI in software development — even though voice commands are often quite sufficient, and the necessity of a keyboard is no longer all that obvious. It's also unclear what will happen to websites and other digital interfaces: picking out a bike or a laptop is clearly easier in a chat with an AI assistant, and for the purchase itself, a banking app will do.

Yet some fundamental shifts can already be glimpsed. To my mind, the two most significant are the growing role of public sources of information and the shift of human effort away from processing and presenting information toward producing new knowledge.

Openness as a condition of existence

People now include AI in their decision-making chain, in various forms. Even in the most conservative variant (like PostgreSQL core development), it is present as a mechanism of critique — hunting for weak spots in a nearly finished product. But if a relevant source of information is not openly accessible, an LLM simply doesn't know about it. Which means it won't use it in decisions, won't combine it with other sources, and won’t check it for correctness. For the model, such knowledge does not exist.

Imagine: with limited time and money, you present your breakthrough idea not at a top international conference but at a university seminar in your hometown. Or at a department meeting, even. Twenty years ago, it would simply have sunk without a trace. Today, when I ask an AI agent to analyse a problem and identify known attempts to solve it, it draws on two sources of knowledge: whatever was available during training and whatever it finds by searching the web. So what matters is no longer so much where your results are published as the mere fact of their presence online, their accessibility to indexing crawlers, and the completeness of the write-up.

Hence, a corollary: communities and social networks built on open access and freedom of criticism gain a notable advantage — a collective search for solutions emerges within them, in the spirit of the Borg from Star Trek (minus the assimilation). If you've made something, it is critically important to publish it, present it, and get it widely discussed — that is what makes a result visible and weighty for AI.

From organising information to producing knowledge

The second aspect is the shift toward the production of new knowledge. As I see it, the job of organising and curating information — what newspapers and television traditionally did — is losing its former value as a standalone mass profession. A personal newspaper or video feed can now be assembled with a simple prompt, no development skills or knowledge of ranking algorithms required. I, for example, put together a personal digest tuned to my interests and easy to reconfigure. Writing a survey on a given topic is now fairly easy, while thirty years ago it was one of the most labour-intensive and respected kinds of work in science.

The requirements for the publication itself are changing, too. Republishing the same material on multiple platforms is just redundant noise to an AI agent. How readable the write-up is, and even what language it's written in, no longer matters. What the agent cares about is a clear and complete problem statement, the method, and the verification details.

Curiously, the very notion of authority is changing along the way. An LLM has no explicit metric of an author's reputation — what operates during training is rather a corpus consensus, the corpus being the entire body of text the model was trained on: a claim independently confirmed by many open sources is absorbed with greater confidence. An author's name carries weight only once it has itself become a visible part of that corpus: the model will "trust" a well-known contributor to a public mailing list more than an anonymous blogger — but only because it has already seen that name next to verified results. Authority is turning from a status attribute ("professor at such-and-such university", "editor of such-and-such journal") into an accumulated trail of verifiable publications in the open. You cannot show your reputation to an AI — you can only accumulate it, and accumulate it in public.

This argument is partly borne out by reality: ACM and the Royal Society opened their archives in 2026, while Wiley sells access to its content to some LLM vendors. My guess is that Elsevier, Springer, and IEEE are staying with hybrid, transitional models out of sheer conservatism and will, sooner or later, move in the same direction.

Will this affect the PostgreSQL community?

Our community, it seems, turned out to be ready for the new reality long before it arrived — and unintentionally at that.

For decades, all the substantive work in PostgreSQL has been carried out on the public pgsql-hackers mailing list: design discussions, patch reviews, arguments and rejected ideas — everything is archived, indexed, and available to any crawler. Commits come with detailed messages linking back to the discussions. By the standards of my thesis, this is a nearly ideal corpus: a complete problem statement, the method, the counterarguments — all in the open. Small wonder that LLMs today reason fairly confidently about the planner's internals or MVCC: they have "read" not just the code but thirty years of arguments around it, the history of commits and reverts.

The reputation mechanism works here literally as described. The names of long-time contributors appear in the corpus next to thousands of accepted decisions — and the model reproduces that trust. An amusing consequence: the weight of the argument "well, so-and-so objected to this approach in such-and-such thread" now extends beyond the mailing list — to anyone who asks an AI about the same design.

But there is a flip side. Knowledge that the community passes around outside the archives — in conference hallways, in chats, in committers' heads — does not exist for AI. For instance, unspoken criteria like "this patch won't make it because it complicates maintenance" are often never stated explicitly anywhere. The gap between the community's written and oral traditions is becoming palpable: the former grows stronger, the latter is likely depreciating. I suspect this will nudge us to write down the unwritten rules more often — which is no bad thing in itself. Or perhaps to make conferences more transparent for online participants?

That said, factors as crucial to DBMS code as reliability, simplicity, and the cost of long-term maintenance across platforms remain poorly codified, which casts doubt on how deeply AI agents will infiltrate PostgreSQL development in the near future.

One last thing: commercial forks and closed-source products built on PostgreSQL end up at a disadvantage. Their solutions, however elegant, take no part in the "collective mind": an LLM will not assimilate their work during training, will not analyse their code, will not include it in its "memory" — so AI will pay off noticeably less in enterprise projects. It looks like openness is ceasing to be merely an ideology or a development model — it is becoming a condition of influence.

What follows

If both trends are real, the formula for successful intellectual work is changing. It used to read: "do good work, publish it in a prestigious venue, and retell it at a few conferences, each time with a different sauce." Now it reads more like: "produce new knowledge, lay it out fully and verifiably, put it in the open, spark a broad discussion — and do it systematically, because your reputation for AI is precisely the accumulated trail of such publications."

Paradoxically, the technology many see as a "killer of thinking" actually devalues the routine processing of information — and raises the price of what a machine cannot do: framing a new problem, running the experiment, taking responsibility for the result.

What about you — what have you noticed? Which changes in your reality already look irreversible?

THE END.
July 19, 2026 — Brisbane, Australia.

I get it: the way humans interact and consume information doesn't really change, our analog bandwidth is limited, and the format — refined over centuries — is probably close to optimal. But why do we limit ourselves artificially and use almost none of what information technology has to offer?

Concretely, here's what I want to discuss.

Delays

For some mysterious reason, we still wait days, weeks, sometimes months for the slides and the video of a talk that matters to us. What stops organisers from publishing the slides before the talk starts, and the audio/video right after it ends? Editing isn't what makes a technical talk valuable — I can rewind to the right moment myself.

Access

Sure, people pay for tickets — but mostly for the chance to meet in person, argue, and soak up the atmosphere. The commercial upside of events like PGConf.dev or PGConf.EU is hardly the main point: they lean heavily on sponsors. And sponsors care about reach, not box office, don't they? So what stops us from streaming at least an audio feed from the nearest smartphone in real time and attaching the recording to the talk page the moment it ends? The community is international, and travel costs and visa formalities keep plenty of professionals away.

Interactivity

An on-site attendee can ask a question after the talk — or in the hallway track, over coffee, at an afterparty jogging. Spontaneous voice conversation is familiar, convenient, and important. But there are also people who don't think that fast on their feet, aren't that strong in face-to-face conversation, don't speak the local language, or simply happen to be far away. So why not attach a group chat, a forum thread, or a mailing list thread to every talk and accumulate the discussion there? Give the speaker one small duty: for a week after the talk, they check the thread and answer questions — they're already deep in the topic, and they prepared for Q&A anyway.

Discussion lifetime

I mean not just keeping the materials around (we already have that), but how long the discussion itself stays alive and is scrollable back and forth, like mailing list threads. Who said an important topic may only be discussed while the event is running? If there's a public venue with the recorded talk, the ongoing chat, and maybe some extra arguments collected along the way, why can't the conference be just the opening move? Technology lets us gather everyone in one online place, open to all, instead of splintering into private micro-groups — we just have to actually do it. The deferred-participation effect matters enormously here: someone might code up the proposed method and run experiments a month after the talk goes online — and the natural place to discuss those results is exactly where the theory was argued.

Language accessibility

This one is genuinely new. In Spanish, I can order coffee or chat about the weather, but certainly not pitch an idea. Yet in the age of live translation, I can — and do — walk into a Spanish-speaking meetup, follow what's said on stage and in the hallway, and even ask a question, provided a neighbour has a spare pair of translating earbuds. For conference content, it works even better: auto-generated captions can be machine-translated on the fly into any language you like. Imperfect, yes — but quite understandable, and it rarely distorts the substance. The same goes for chat discussions. Suddenly, you can consume information in your native language and share ideas in it, too.

Post-processing

With AI, you can summarise a talk, extract only the aspects you care about, fact-check the claims, and follow specific threads of a discussion — all it takes is getting the information into a text format that's easy to parse. Organise this in real time, and the listener gets a powerful context-enrichment tool: no more waiting for the talk to end to clarify an unfamiliar term, find the project's website, or get a quick survey of similar projects.

And yes, all of this should be freely indexable by search engines, so that anyone — human or AI — can find the material and build on it.

The objections

First: "If everything is streamed for free, who buys a ticket?" But nobody travels to a conference for raw content anymore — content is abundant. People come for the hallway track, personal connections, and the atmosphere, none of which fit through a screen: the online formats of the pandemic years never replaced meeting in person. And sponsors, as noted above, care about reach, not the till.

Second: a speaker may not want their raw, unedited talk — stumbles and all — to live online forever. Fair, and solvable: a per-speaker opt-out, plus a community understanding that a live talk is not a studio recording. After all, PostgreSQL itself has been developed for decades on public mailing lists with a permanent archive — and the community is better for it.

Third: entropy. A reasonable fear that bots and flame wars — the natural fauna of group chats — will bury the signal in noise. Fair again. But nothing stops us from requiring a community-account sign-in for active participation: registration stays open to everyone, and per-account spam filtering is a solved problem. Slack-style threads keep long bikeshedding debates contained, and AI tools will distil the takeaways for anyone unwilling to read 1000+ messages in a general channel.

So what are we waiting for?

None of the above is complicated or resource-hungry: all the building blocks already exist and are used in everyday life. So why not include them in our conferences? A minimal experiment for the next meetup:

• a smartphone on a tripod — live stream and recording of the talk

• a public chat or forum thread per talk — with the speaker promising to drop by for a week

• an auto-transcript — a text version open to indexing, translation, and AI processing

If something breaks along the way — say, the video-to-text pipeline — the community will survive. What matters is momentum: actually trying the capabilities the digital era keeps handing us.

It's not that nobody is moving in this direction. POSETTE is a fully virtual conference — yet (as I see it) it still follows the traditional format, simply delivering content in a convenient digital form. A pre-recorded stream is one of the strangest setups I've come across. The schedule offers a live Q&A with the speaker during the broadcast — sure — but if the recording already exists, why not publish it in advance? Then you could experiment with the format — have the speaker walk through the talk live, hitting the key points, interview-style, and let interested attendees join the call one by one or all at once; moderation tools are flexible enough to handle a real discussion if one breaks out. Hard to say what would come of it, but the cost — for organisers and attendees alike — is minimal. Someone just has to go first.

Then again, maybe I'm missing something important that breaks this whole chain of reasoning. Tell me in the comments!

THE END
Sydney, Australia. July 2026.

In systems programming, reliability, maintainability, and extensibility usually matter more than new features. That makes AI agents genuinely hard to apply well. If I'm being honest, I can't claim the technology has meaningfully moved the needle on our productivity — the bottleneck still sits squarely in review and QA. So when frontend colleagues share their excitement and start planning which industry to pick up next, I can only look on with envy.

That said, research, analysis, and internet scanning have become dramatically easier — I can now read Chinese colleagues' work without friction. Exploring the source code of adjacent OSS projects is where it genuinely shines: in five minutes, in a language I barely know, without any prior knowledge of the project structure, I can get answers to real questions, steal a few useful patterns, and learn from both the smart and the questionable architectural decisions. The pity is that work like this isn't my main job.

There is, though, one area where AI has become genuinely indispensable — saving me time and sanity every day: news filtering.

The technical infosphere — Planet PostgreSQL being a prime example — has increasingly filled with two kinds of content: posts that re-explain familiar database concepts for the thousandth time, and thinly veiled marketing whose real goal is to put a product in front of you. I have nothing against good marketing, but those of us who are here for the technology rather than the buying decisions find that it generates a persistent low-level background noise.

There is another dimension: pgsql-hackers has quietly started accumulating AI-assisted content. I’ve seen review feedback containing arguments that felt suspiciously familiar — the kind of well-structured points that turn up when someone asks a language model to review a patch, and that I had already seen in Claude's review and set aside at the development stage. Nobody can read everything; what you actually want is to surface the interesting threads, the unexpected turns in a discussion, the non-obvious solutions.

So the obvious question arose: why not hand this filtering work to something actually built for text analysis?

Take YouTube, which curates your video feed and serves up what its algorithm thinks you want. Or dev.to and similar aggregators, which persistently recommend things that don't quite match your interests. With an AI agent, you can write your own prompt and build a genuinely personal feed — without depending on someone else's black-box algorithm. You can even have separate work and personal flavours.

No sooner said than done.

Claude is now instructed to crawl the Internet for technical blogs covering Postgres, SQL Server, and other relational databases, check each author's profile, and keep only engineers with a verifiable track record — committers, core contributors, and recognised community members. From those sources, scan new posts and surface only those that contain genuinely new ideas (or challenge known techniques), novel database techniques, concrete production experience, or real debugging stories. Fact-checking and author-topic consistency checks are built in. A separate prompt watches for hidden commercial intent — is the article's real goal to sell the reader something?

Claude also scans non-English sources and summarises in English — Chinese content in particular has been a pleasant surprise. Teaching Claude to search non-English websites isn't entirely straightforward, though: the built-in web_search tool returns near-empty results for non-Latin queries. The workaround was to use Chrome MCP to locate RSS feeds, then parse and translate them.

Another valuable input: tracking changes on CommitFest and the mailing lists. Here, I let Claude use its own judgment about what might interest me and what counts as a "meaningful turn" in the discussion of a given feature. The history of our communication underpins this part - I think, Claude knows me better than anyone else ;).

Collecting CFP announcements and upcoming talk previews for relevant conferences and meetups turned out to be surprisingly useful, too — you never know when time and budget will align to catch an interesting speaker. And as a cherry on top, I asked Claude to scan academic paper aggregators and put together a weekly selection of research relevant to my work.

The output is an auto-generated weekly digest, tuned to one person's interests. The prompts and all supporting files live on GitHub — Claude has full write access. GitHub Pages hosts the digest in a readable format, with an RSS feed for mobile devices. The whole thing is driven by an auto-run routine in the Claude Desktop App: when you open the project, it checks whether a fresh issue is due and, if so, generates it and pushes the changes.

Here's what a typical entry from the latest issue looks like:

[hackers] Conflict log history table for Logical Replication
— A PG20-targeted patch that lets a subscription write apply-worker conflicts to a system-managed table instead of the server log hit a real bug: a concurrent ALTER SUBSCRIPTION changing the log destination drops the table while the apply worker still holds a cached OID — the worker crashes, and the conflict record disappears entirely. (vignesh C, Amit Kapila et al.)

After four weeks, the material has been genuinely useful. The occasional irrelevant item slips through — that's unavoidable if you want to keep the risk of missing something important low. But the signal-to-noise ratio is dramatically better than raw Planet PostgreSQL or a generic news aggregator.

This experiment raises a bigger question: what happens if personalised content filtering like this becomes common? Not long ago, we all shared the same TV channels, music, magazines, books, and a few must-read blogs. That overlap made conversations easier: you could cite a film and expect others to catch the context. If everyone now uses their own custom filter, what happens to our shared context? How can we have productive debates or build consensus when each of us is reading a different part of the same world?

I don't have an answer. But I think it's worth asking.

THE END
June 25, 2026 · Madrid.

That earlier article answered the question 'how widespread is this pattern?' After all, if you're going to improve something, it helps to know how useful the improvement will be, right? Here, I want to give a sense of how this pattern leads to performance regressions and point out directions in the PostgreSQL optimiser that could make the situation easier.

Spoiler: not much yet — but a few things are moving on pgsql-hackers. Three patches, discussed across 2024–2026, target three different sources of regression. Each is covered below.

Where the problems come from

As a reminder, here is what the query looks like:

SELECT
    ol.id,
    COALESCE(p.name, g.name, s.name) AS item_name
FROM order_lines ol
  LEFT JOIN products p
    ON ol.type = 'A' AND ol.item_id = p.id
  LEFT JOIN gift_cards g
    ON ol.type = 'B' AND ol.item_id = g.id
  LEFT JOIN subscriptions s
    ON ol.type = 'C' AND ol.item_id = s.id
WHERE EXISTS (
  SELECT 1 FROM orders o
  WHERE o.id = ol.order_id AND o.placed_at >= DATE '2024-01-01')
ORDER BY ol.popularity
LIMIT 100;

At the moment, the Postgres optimiser implements fairly primitive logic here. For every row of the base table (order_lines), the query probes each of the N subtype tables through a LEFT JOIN. Only one of those joins ever returns a match — the one whose discriminator matches the type value in that row. The remaining N−1 joins are guaranteed to come up empty: their ON predicate carries a different discriminator value (see the schema below).

The optimiser picks one scan order for the inner side of the LEFT JOIN and applies it to every row — it cannot route each outer row to the right inner table based on the discriminator value. The baseline cost of the pattern is O(M × N) probes, of which only O(M) are useful. And you genuinely can't avoid touching the inner side of each OUTER JOIN: there is no constraint that would guarantee that key values across the inner tables are non-overlapping. So we have to look for a compromise…

Having gathered some statistics on problematic cases, I settled on the following main causes of query-performance degradation that show up in production:

1. Size of the base table. Doubling the row count of the base table doubles the number of fruitless inner-side join probes.

2. Aggregation of the result (usually ORDER BY + LIMIT). A sort often stalls the pipeline. As a consequence, the LIMIT operator can't reduce the read from the base table and cut the scan of the join tree short. Imagine that out of 100k kinds of items in the order_lines table, the query only needs the TOP-10 most popular — a full scan looks plainly excessive.

3. An EXISTS subquery that the optimiser pulls up into a join. A somewhat non-obvious but painful — and therefore noticeable — case: once the number of tables in the join tree exceeds the join_collapse_limit GUC, the pulled-up SEMI JOIN drops out of the join-order search problem and is moved into a separate one. The upshot is that this EXISTS subquery, which originally served as a row filter on the base-table scan, floats up to the top of the query tree and ends up filtering base-table rows at a far later stage of execution.

4. Accumulating cardinality-estimation error. As the number of joins grows, the estimate quickly bottoms out at the minimum value and doesn't change higher up the plan tree. For our cascade of LEFT JOIN operators, this isn't a huge deal, but it's worth a mention.

Let's go through the ways out.

1. Result Filter, or one-sided gating

In December 2024, a thread was opened on pgsql-hackers titled "Do not scan index in right table if condition for left join evaluates to false using columns in left table". Tom Lane proposed splitting the JOIN ... ON ... conditions into two groups — those that depend only on the outer side and those that depend on the inner side. The first group can be evaluated before starting the inner-side scan of a NestLoop: if they're already false, the scan can be skipped entirely. Andres Freund suggested simplifying the implementation by moving the evaluation of the outer clause into a so-called gating operator — a separate Result plan node.

The need to look at the inner side for each outer row remains — there's still no constraint in the database that would guarantee that, if a row was found for a given item_id in gift_cards, no such row exists in subscriptions. However, since each join clause has the form:

... ON ol.type = 'subscription' AND ol.id = s.id

We can cap the overhead of scanning the inner side by checking the condition on ol.type: if it evaluates to false, there's no reason to reach into the inner side because the inner row would be replaced by a set of NULLs anyway. We end up with something like the following plan shape:

So we add a single gating-expression operator to the plan. It is evaluated for every row of the outer side of that join. That operation is usually cheap enough — it won't touch shared buffers, indexes, or disk. So while formally the reach into the inner side remains, the overhead is driven down to almost nothing. This is especially effective when the inner side isn't a plain table scan but a complex subtree with additional joins and aggregation.

The join tree of our query's plan would then look something like this:

The benefits of this approach are clear:

1. Fewer reaches into shared buffers and disk.

2. Bounded performance impact as the list of categories grows (i.e. the number of joined tables).

3. Protection against regression when the inner-side scan isn't index-based.

2. Sort Pushdown, or pre-sorting

The second scaling problem with such queries is tied to the ORDER BY + LIMIT construct. We can't ignore it because picking the TOP-N most popular items is a core pattern in online stores.

What exactly is the problem? If the base table order_lines has a suitable index and the data already arrives sorted, then the join pipeline runs without a hitch: we pull only the minimum number of rows from order_lines needed to satisfy the LIMIT. But if the data isn't sorted, then every row of order_lines flows through the join tree, and the sort happens higher up — see the figure below:

So why not just build an index and be done with it? First, the need for fast DML and maintenance operations on the database often won't allow for many indexes — each new index is frequently an added overhead. Second, such an index can contain many duplicates, which reduces its efficiency. In our case, for example, if popularity is discrete and spread over some fixed integer scale, having that index may not be worthwhile.

The most effective way out is to insert a sort operation (heapsort, TOP-N) directly above the base-table scan — yes, we read every row from it once, but at least we don't push them through the entire join tree. That is, such a sort reduces the regression as the number of joins in the tree grows (see the figure below).

So why doesn't PostgreSQL do this already? The issue is that query planning happens bottom-up: at the moment, it chooses the optimal scan method, yet the optimiser knows nothing about the structure of the join tree or whether it's worth sorting the scan result on any particular field. And the join above doesn't know it either, since whether the technique pays off depends on the plan's overall structure. Generating plan variants "just in case" is a poor strategy because it enlarges the search space and, therefore, increases memory consumption and planning time.

In April 2026, a patch was proposed on the hackers mailing list: try to build a pre-sorted path for the base table of a NestLoop if that table is mentioned in a prefix of query_pathkeys. The compromise on offer: if a table's columns appear in some prefix of the ORDER BY or GROUP BY operator, there's a fair chance it will be efficient to have a pre-sorted set of rows on that prefix. And that can pay off not only in our query, but in MergeJoin operators too, can make more use of IncrementalSort, and generally give the optimiser a bit more room to choose — after all, we'll have to sort on that prefix anyway, won't we?

3. SubLink relocation, or filtering as early as possible

This problem is harder to grasp. When a query has (say) an EXISTS subquery that can't be transformed into a JOIN, it acts as a filter, reducing the number of rows the table-scan operator produces (see the picture below). Consequently, fewer rows flow through the join tree. If only 1% of order_lines rows pass the EXISTS subquery, then exactly 1% of rows should enter the LEFT JOIN tree, and the remaining work shrinks by a factor of 100.

If, on the other hand, such an EXISTS subquery is transformed into a JOIN, then on one hand this changes how many times the EXISTS is evaluated — instead of evaluating the subquery per base-table row, the number of passes is determined by the join type (for HashJoin and MergeJoin it's usually once). On the other hand, this join is inserted at the top of the join tree — i.e. as far as possible from the base table order_lines (see the figure below). And when join_collapse_limit is smaller than the number of joins, this means the optimiser will never consider the orders ⋈ order_lines variant. That's a potential performance cliff — rows that used to be filtered out early in execution can, after an upgrade or as the join tree grows, suddenly flow higher up the JOIN tree and be filtered much later.

To take the edge off the problem, in May 2026, a patch was proposed on hackers that changes the subquery-to-JOIN transformation logic (the so-called pull-up):

After converting EXISTS into a SEMI JOIN, find a suitable place for it: don’t insert it at the top of the join tree. Instead, push the SEMI JOIN down into the smallest subtree that contains the relations referenced inside that subquery.

In our example, there's only one reference to tables outside the EXISTS subquery — o.id = ol.order_id, which refers only to order_lines. So the newly created SEMI JOIN is inserted into the join tree right next to order_lines.

The resulting join tree looks like this:

And the optimiser gets the chance to choose the join order — including making it roughly what it was before the Subquery → JOIN transformation (if that's efficient).

4. Join statistics

This text wouldn't be complete without mentioning one feature under active development in the community — JOIN selectivity statistics.

Alexandra Wang is developing a working prototype of "join statistics": she's trying to solve one aspect of the long-standing optimiser problem, where optimisers estimate join selectivity from each table’s statistics in isolation and therefore miss skew in the joint distribution of values across the two tables. That error grows up the plan, pushing the planner to pick a nested loop where a hash/merge is needed. The idea of the feature is to collect statistics on the join result itself and feed them into the selectivity estimate.

It's proposed as a declarative thing: the user declares the desired join via CREATE STATISTICS (modelled on ordinary extended statistics). For now, a narrow but useful slice actually works: MCV statistics for an INNER JOIN of two tables on an equality expression. This is still a proof of concept, and the community broadly agrees that the direction is useful. The open questions are the fundamental ones — how to store such statistics in the catalogue so they generalise to joins of more than two tables, and how to collect them honestly.

Wrapping up

The polymorphic query pattern is a natural consequence of mapping object-oriented abstractions onto the relational model. Banning it "from above" is extremely hard: it's generated automatically by ORMs and platforms, and retroactive schema changes in large production systems are prohibitively expensive. So the only practical short-term direction is to have the planner handle it better.

The three directions listed above aim to reduce the non-linear effects that can arise as the database grows or the schema becomes more complex. None of them “closes” the problem entirely — but each removes one specific pain point:

• Result Filter / one-sided gating removes the M × (N−1) fruitless probes.

• Sort Pushdown lets LIMIT localise the disk access when there’s no index on the sort condition.

• SubLink relocation reduces the fallout from the non-cost-based transformation of subqueries (EXISTS, ANY, etc.).

All three are being discussed on pgsql-hackers, and none are committed yet. If you have a workload with the polymorphic pattern — especially with N > 10 LEFT JOINs, or with subqueries in the WHERE clause whose transformation leads to painful effects — I'd be very interested to take a look. Drop a comment, or write straight to pgsql-hackers.

THE END.
June 20, 2026, Madrid.

Just for the introduction

Think of the homepage of a typical online store — say, Amazon. Personally, on my homepage, I see ads for several products:

• An air fryer

• Magnesium Citrate + B6

• Protein brownies

• and so on.

Thinking like a database developer, I can roughly imagine that this page is built from the results of a query like:

SELECT name, description FROM products p
  LEFT JOIN kitchen_appliances ka ON (p.id = ka.id)
    LEFT JOIN pharmacy f ON (p.id = f.id)
      LEFT JOIN sports_nutrition sf ON (p.id = sf.id)
      ...
ORDER BY p.popularity DESC
LIMIT N

Planning such a query efficiently is no easy task — and in my experience, this is confirmed by user reports from the 1C world, since PostgreSQL is currently not rich in LEFT JOIN optimisations. At the same time, the properties of this pattern enable the development of various techniques to improve execution efficiency. I've managed to implement several straightforward optimisations of this template. But first, let's understand what polymorphic references actually are, where they come from, and how common they really are. That's the gap I'm trying to fill with this post.

The Pattern

Many real-world data models contain references that can point to one of several target entity types. An order line may reference a physical product, a digital download, a gift card, or a subscription. An activity record in a CRM system may be linked to a contact, a company, or a deal. An audit log entry may refer to any entity in the system.

Relational schemas have no built-in mechanism for such references. The most common encoding uses a discriminated foreign key — a pair of columns that jointly identify the target table and the row within it:

CREATE TABLE order_lines (
    id          SERIAL PRIMARY KEY,
    order_id    INTEGER NOT NULL REFERENCES orders(id),
    item_type   VARCHAR(20) NOT NULL,   -- discriminator
    item_id     INTEGER NOT NULL,        -- polymorphic FK
    quantity    INTEGER NOT NULL
);

Here item_type may contain 'product', 'gift_card', or 'subscription', and item_id stores the primary key of the corresponding table. No single FOREIGN KEY constraint can enforce referential integrity across all three target tables, so the database engine treats item_id as a plain integer column.

To resolve the reference — say, to get the human-readable name of the ordered item — the query must join against each possible target table, gated by the discriminator:

SELECT
    ol.id,
    COALESCE(p.name, g.name, s.name) AS item_name
FROM order_lines ol
LEFT JOIN products      p ON ol.item_type = 'product'
                          AND ol.item_id = p.id
LEFT JOIN gift_cards    g ON ol.item_type = 'gift_card'
                          AND ol.item_id = g.id
LEFT JOIN subscriptions s ON ol.item_type = 'subscription'
                          AND ol.item_id = s.id;

For each row in order_lines, at most one of the three LEFT JOINs finds a match; the other two return NULLs. As the number of target types grows, so does the fan of LEFT JOINs. This query form can be named the polymorphic reference resolution pattern.

Structural invariants

The pattern has a precise structure that distinguishes it from an arbitrary collection of outer joins:

1. Mutual exclusion. The discriminator predicates in the join conditions are pairwise disjoint: for any row of the base table, the discriminator predicate of at most one join evaluates to true. In the example above, item_type = 'product', item_type = 'gift_card', and item_type = 'subscription' cannot simultaneously be true for the same row. This guarantees that at most one LEFT JOIN produces a match for each base table row. Note that if a row's discriminator value isn't represented by any join in the query (e.g., item_type = 'coupon' with no corresponding join) or the discriminator is NULL, then no join matches — the row passes through all joins with NULLs in all target table columns. The invariant is "at most one," not "exactly one."

2. Inner-side key uniqueness. The join key on each target table's side is its primary key (or at least a unique key). This is verifiable by the query planner via pg_catalog metadata. Combined with mutual exclusion, this guarantees that each LEFT JOIN produces at most one match per base table row — no join duplicates rows.

3. Column usage constraint. Every reference to a target table T_i's column in the SELECT list sits inside a COALESCE or CASE expression that evaluates to NULL when the discriminator doesn't match T_i. No target table column is used outside such a folding expression. Moreover, each folding expression must contain a contributing column from every target table participating in a LEFT JOIN in the query: if a COALESCE includes p.name and g.name, it must also include s.name — unless the LEFT JOIN contributes no column to that particular folding expression.

Thus, given invariants 1–3, for a subset of base table rows filtered by discriminator = c_k, the LEFT JOINs to all other auxiliary tables can be safely removed from the plan.
This claim covers the flat SELECT case. When a polymorphic resolution query is embedded in a larger query with GROUP BY or aggregate functions, the interaction between join elimination and aggregate semantics requires additional analysis — that's a direction for further research.

Where this pattern appears

The pattern is widespread across application domains and frameworks. It appears in two distinct forms that coincide at the query level but differ in schema properties.

Polymorphic associations (no referential integrity). Ruby on Rails popularised the term "polymorphic association," implemented as a pair of _type / _id columns in the referencing table. A foreign key declaration is impossible because the target table differs for each row. When a query needs to resolve the reference, the ORM generates discriminator-gated LEFT JOINs to all candidate tables. Django's django-polymorphic library uses the same approach and documents the resulting query overhead. GitLab's engineering guidelines outright forbid this form, citing loss of referential integrity and query performance degradation observed in production.

Joined table inheritance (with referential integrity). Hibernate's @Inheritance(strategy = JOINED) places each subclass in a separate table whose primary key is also a foreign key to the parent table. The discriminator is stored in the parent table. When loading a base entity, Hibernate generates LEFT OUTER JOINs to all subclass tables in the hierarchy. Community forum threads document queries with 30–40 LEFT OUTER JOINs from hierarchies of moderate depth. Django's multi-table inheritance produces an analogous structure. Unlike polymorphic associations, joined inheritance provides referential integrity constraints between parent and child tables — but the resolution query shape remains identical.

PostgreSQL's table inheritance mechanism (INHERITS) is a partial schema-level solution to the same problem: queries against the parent table automatically include rows from child tables without explicit joins. However, INHERITS has significant limitations — no foreign key constraints on child tables, no unique constraints spanning the hierarchy, and poor interaction with some planner optimisations — which limit its applicability here.

CRM platforms. Salesforce provides polymorphic lookup fields (WhoId, WhatId) for the Activity object. A single WhatId field can reference Account, Opportunity, Campaign, or any of dozens of custom objects. Salesforce offers a TYPEOF operator in SOQL to handle polymorphic resolution without explicit joins to each target type.

Evidence of cost in real systems

Quantitative benchmarks that isolate the overhead of the polymorphic resolution pattern are rare in the peer-reviewed literature. However, developer communities and framework documentation provide substantial qualitative evidence that the cost is real.

Hibernate community forums feature discussions where developers report that loading a single entity from a JOINED inheritance hierarchy generates queries with dozens of LEFT OUTER JOINs, and that these queries dominate response time under read-heavy workloads. The django-polymorphic documentation dedicates a separate section to performance, recommending that developers use .non_polymorphic() when the specific type isn't needed. GitLab's warning on polymorphic associations is explicitly motivated by query performance degradation observed in production.

A related body of work shows that multi-join queries are inherently prone to plan-quality degradation. Leis et al. (2015) showed that cardinality estimation errors accumulate multiplicatively as they propagate through a join chain, often reaching several orders of magnitude even on well-analysed tables. The polymorphic resolution pattern with its N outer joins is susceptible to exactly this kind of error accumulation.

Schema-level alternatives

The database literature proposes several types of hierarchy modelling strategies that avoid the discriminated foreign key. Fowler (2002) catalogued three practically oriented patterns:

• Class Table Inheritance — a common parent table plus one per subtype, with subtype primary keys serving as foreign keys to the parent. Preserves referential integrity but adds one join per query and requires two writes per insert (one to the parent, one to the subtype table).

• Single Table Inheritance — all subtypes collapsed into one wide table. Avoids joins entirely but produces a wide, sparse table: columns from other subtypes are NULL for a given row, wasting disk space and complicating indexing.

• Concrete Table Inheritance — completely independent tables for each subtype. Eliminates both joins and NULLs, but requires UNION ALL across all subtype tables for polymorphic queries, and schema changes to common columns must be propagated to each table independently.

Karwin (2010) devoted a chapter in SQL Antipatterns to the discriminated foreign key, calling it "Polymorphic Associations," arguing that the approach sacrifices referential integrity and query performance for schema simplicity. He recommends Class Table Inheritance ("Common Super-Table") as the preferred alternative.

Despite these alternatives, the discriminated foreign key remains dominant in practice. ORM frameworks generate it by default, and retroactive schema changes in large production systems are prohibitively expensive. Therefore, practically meaningful optimisations must accept the schema as given and work at the query execution level.

Interaction with EXISTS subqueries

The polymorphic resolution query as presented above has no WHERE clause: the base table is scanned in full, and every row passes through N LEFT JOINs. In practice, the query often includes an EXISTS subquery filtering the base table — for example, restricting order lines to orders placed within a certain date range:

SELECT
    ol.id,
    COALESCE(p.name, g.name, s.name) AS item_name
FROM order_lines ol
LEFT JOIN products      p ON ol.item_type = 'product'      AND ol.item_id = p.id
LEFT JOIN gift_cards    g ON ol.item_type = 'gift_card'     AND ol.item_id = g.id
LEFT JOIN subscriptions s ON ol.item_type = 'subscription'  AND ol.item_id = s.id
WHERE EXISTS (
    SELECT 1 FROM orders o
    WHERE o.id = ol.order_id
      AND o.placed_at >= '2024-01-01'
);

The EXISTS subquery here references only columns of the outer table — a common case in practice. How the planner handles it has a significant impact on performance, and the result depends on whether the subquery is converted to a semi-join and on the join_collapse_limit / from_collapse_limit settings.

Case 1: EXISTS remains a SubPlan (no pull-up). When the planner doesn't convert EXISTS into a semi-join — for example, because the subquery contains constructs that prevent pull-up — it's typically evaluated as a filter at or near the base table scan level. Each order_lines row is checked against the SubPlan before entering the join tree. Rows that fail the EXISTS check are discarded immediately, so only filtered rows reach the N LEFT JOINs. This is the favourable case: the base table filter prunes the fan early.

Case 2: EXISTS is converted to a semi-join (within join_collapse_limit). PostgreSQL normally converts a simple EXISTS subquery into a semi-join (see the picture below).

After pull-up, the semi-join's relation (orders) becomes one of the base relations that the planner considers when enumerating join orders via dynamic programming (or GEQO). The planner can consider all valid join orderings. Since the semi-join clause references only order_lines columns, and the LEFT JOINs also require order_lines an on the outer side, the planner can, in principle, place the semi-join early — joining orders with order_lines before any of the LEFT JOINs fire. With accurate selectivity estimates, the planner typically chooses exactly this early placement, preserving the filtering effect.

Case 3: join_collapse_limit exceeded. Adding the pulled-up semi-join brings more tables to the join tree. When this exceeds join_collapse_limit (default 8 in PostgreSQL), the planner preserves the syntactic JOIN nesting from the parser instead of flattening all relations into a single list for exhaustive enumeration. The limit exists because the number of join orderings grows super-exponentially with the relation count, making exhaustive search impractical for moderately large relation counts.

The LEFT JOINs form a left-deep JoinExpr tree that the planner treats as a single block, executing joins in their syntactic order. The semi-join relation, added to the top-level FromExpr During pull-up, sits outside this nested block. The planner can't wedge the semi-join into the unflattened LEFT JOIN chain — it can only attach the semi-join to the block as a whole. Result: the semi-join is evaluated against the full output of the LEFT JOIN chain. Every base table row passes through all N LEFT JOINs, and only then does the EXISTS filter check whether the order was placed this year. For large base tables with a selective EXISTS predicate, this can degrade performance by orders of magnitude compared to Case 1.

This is felt especially acutely during upgrades, when a new type of pull-up blocks early filtering of base table rows and leads to noticeable query degradation that can only be overcome by rewriting the query.

Summary

This query pattern turns out to be quite widespread. And there's no point blaming ORM systems for excessive complexity — that's just the nature of the domain and the limitations of the relational model. Performance-wise, it will execute more or less efficiently if the right indexes are in place (especially on the base table), and will immediately start dragging if the base table filter isn't covered by an index. So there's optimisation work to be done, and in future posts I'll try to describe a set of hacks for the PostgreSQL optimiser that target this pattern.

What do you think about this pattern? Do you agree with the analysis?

THE END.
May 18, 2026, Madrid, Spain.

A production story

This story started in July 2021. At the time, I was finishing a sharding solution built on top of postgres_fdw. During testing, our engineers sent me an example query that would crash periodically with a SEGFAULT. One look at the plan told me something was very off.

 Append
   ->  Nested Loop
         Output: data_1.b
         Join Filter: (g1.a = g2.a)
         ...
   ->  Materialize
         Output: g2.a, data_2.b
         ->  Hash Join
               Output: g2.a, data_2.b
               Hash Cond: (data_2.b = g2.a)
               ...

The first obvious question: how did a Materialize node end up as a direct child of an Append? The second, more interesting one: how can one Append combine two sources with different tuple widths? No wonder the query was crashing — and to make it worse, the failure was intermittent; sometimes the very same query produced a perfectly reasonable plan.

On paper, the bug shouldn't have been possible: the optimiser doesn't work that way. A few days of debugging pointed the finger at dangling pointers. While building one of the alternative Append paths, the optimiser adds a cheaper path to a child RelOptInfo’s pathlist and evicts the one that was there before. But the previously constructed Append still holds a pointer to that now-freed slot. A step or two later, the allocator hands the exact same chunk back out for a new Path higher up the tree, for, say, an enclosing JOIN. The result is a plan that makes no semantic sense at all.

Where dangling pointers come from

PostgreSQL builds each relation's pathlist incrementally through add_path(). When a newly arrived path dominates an existing one — cheaper across all relevant dimensions (startup cost, total cost, pathkeys, parallel-safety) — the dominated path can be released immediately.

Because the optimiser builds the plan bottom-up, the pathlists of lower operations (scans, for instance) are completed first, and then the pathlists of upper operations (Append and friends) are assembled with references to specific entries in the lower nodes' pathlists.

This works fine right up until the optimiser, while building a Path for an upper operation, decides that the plan can be improved by adding something to a lower pathlist. At that point, a path may be evicted from the lower pathlist — one that is already referenced from higher up the plan tree. When that happens, we have a dangling pointer.

There is also an example — an upper rel evicting a path that a lower rel still references — in the Postgres core itself. A minimal reproducer looks roughly like this:

DROP TYPE IF EXISTS insenum CASCADE;
CREATE TYPE insenum AS enum ('L1', 'L2');

EXPLAIN (COSTS OFF)
SELECT enumlabel,
  CASE WHEN enumsortorder > 20 THEN NULL ELSE enumsortorder END AS so
FROM pg_enum
WHERE enumtypid = 'insenum'::regtype
ORDER BY enumsortorder;

With pg_pathcheck loaded, you will see:

WARNING:  pg_pathcheck: invalid NodeTag T_SeqScan in pathlist, rel {pg_enum}
DETAIL:  pathlist contents: [0] T_ProjectionPath; [1] T_SeqScan INVALID

 Sort
   Sort Key: enumsortorder
   ->  Seq Scan on pg_enum
         Filter: (enumtypid = '16590'::oid)

What's going on: the path representing the scan of pg_enum gets shared under certain conditions with ordered_rel, the rel that represents the sorted query result. Later, when a new path PP3 arrives in ordered_rel, the old shared PP2 is evicted and freed by add_path() — but input_rel->pathlist still holds a pointer to the freed chunk:

 input_rel {pg_enum}                    ordered_rel
 ┌──────────────────┐                   ┌──────────────────┐
 │ pathlist:        │                   │ pathlist:        │
 │   [0] → PP1      │                   │   [0] → PP2 ◄────┐
 │   [1] → PP2 ─────────── SHARED ──────────────────────── ┘
 └──────────────────┘                   └──────────────────┘


 input_rel {pg_enum}                    ordered_rel
 ┌──────────────────┐                   ┌──────────────────┐
 │ pathlist:        │                   │ pathlist:        │
 │   [0] → PP1      │                   │   [0] → PP3      │
 │   [1] → ??? ◄─── dangling ───╳╳╳     │                  │
 └──────────────────┘      pfree'd      └──────────────────┘
                           chunk

Why does it still work in practice

In vanilla Postgres, this example causes no visible problems, thanks to a subtle invariant: Postgres never walks input_rel's pathlist once ordered_rel is under construction — it uses direct references to the cheapest_* paths instead. The dangling pointer is created, but nobody dereferences it before the per-query memory context is reset at the end of the statement.

Extensions, however, often carry their own internal logic and may need to traverse the full pathlist, which can trip the bug. Separately, when building complex plan transformations along the lines of the example above, Postgres forks can produce dangling-pointer situations of their own — and there is no guarantee that all of them will be found and fixed before shipping to production.

Features that manipulate the query plan at runtime — disable_node, pg_plan_advice, pg_hint_plan and the like — can also accidentally trigger latent optimiser issues and crash the system with a SEGFAULT.

A small walker

pg_pathcheck is around 600 lines of C. It registers two planner hooks: create_upper_paths_hook to remember the top PlannerInfo and planner_shutdown_hook to do the work.

The walker visits every Path reachable from the top PlannerInfo. That means the upper_rels[] arrays, every simple_rel_array entry with its optional subquery subroot, every join rel collected during dynamic programming, the various parallel RelOptInfos (unique_rel, grouped_rel, part_rels), and — within compound Path nodes — every embedded sub-path field (outerjoinpath, innerjoinpath, subpath, subpaths, bitmapqual, …). A visited-pointer hash keeps the traversal linear.

At every pointer, two checks run. The first is a NodeTag whitelist: if path->type is not one of the known Path-family tags, the memory has either been filled with 0x7F (freed, not yet reused) or re-allocated as some other kind of node. The second, used for base and join rels, is a parent-match check: path->parent on a path in rel->pathlist must equal rel. A mismatch catches same-size-class aliasing — a freed chunk that has been recycled into another valid Path belonging to a different rel entirely. The tag check passes in that case, but the ownership is wrong.

When a check fires, the extension emits a report at a configurable elevel (WARNING / ERROR / PANIC, controlled by pg_pathcheck.elevel). The report names the rel, the slot where the stray pointer sits, the full contents of the containing list (with each element annotated by its node kind), and — via debug_query_string in the hint — the query that triggered it.

The allocator wrinkle

PostgreSQL's aset.c uses power-of-two size classes. A Path is 80 bytes, which lands in the 128-byte class. So do:

• ProjectionPath (96), SortPath (88), MaterialPath (88)

• NestPath, AppendPath (112).

• Among Plan nodes, SeqScan (112), BitmapHeapScan (120), NestLoop (128), Hash (128), Result (128), Gather (128), and roughly a dozen more scans.

When a Path is freed, its slot returns to the 128-byte freelist. The next makeNode(<Something>) inside the planner picks up that exact address. An old dangling pointer that had been invisible during planning now references a perfectly live Path or Plan node — with a valid but wrong NodeTag, and with fields shaped for an entirely different kind of object. This is also why CLOBBER_FREED_MEMORY on its own is not enough to detect the problem: the clobber pattern is overwritten by the re-allocation before any walker gets a chance to see it. Out of about 4,000 findings in my full-suite run, the number of pointers found carrying the raw 0x7F7F7F7F fill was zero. It looks like memory is fully reused.

Valgrind catches a use-after-free at the moment of the dereference — and in this case, nothing ever dereferences the stale pointer during normal execution. So, a structural walker that verifies the pathlist's semantic invariants — ‘this pointer must reference a live Path owned by this rel’ — is the right tool for a use-after-free that the rest of the program is disciplined enough not to trigger — byte-level tools catch only the dereference, not the dangling reference.

Who may need this

The three audiences may find this code useful.

PostgreSQL core developers. The hackers' threads discuss at least three solution shapes — reference-counted paths, a used flag, and local memory contexts — and have been going back and forth on whether the unwritten contract is actually worth tightening. This dataset grounds the discussion.

Extension authors — especially those who write custom-scan providers, FDWs, optimisation features, or plan-inspection tooling — benefit from the tool as a sanity check.

Fork maintainers have the largest blast radius. Forks tend to modify the planner more aggressively than extensions can, and they ship on schedules that are not always in lockstep with the PostgreSQL master. Running pg_pathcheck against a fork’s test suite will tell you whether your modifications preserve the invariants that core happens to rely on.

Continuous coverage

The repository ships a GitHub Actions workflow that runs make —k check-world against a freshly cloned PostgreSQL master with pg_pathcheck enabled. It runs on every push, every pull request, on manual dispatch, and nightly. The artefacts include the full server logs and a deduplicated summary rendered into the step-summary panel. The wiki hosts the raw reports and written analyses of each run.

If upstream master introduces a new source of dangling pointers, the nightly will flag it the morning after the commit lands. If it closes one, the counts will drop. In either direction, the workflow provides a real-time pulse on the contract's state.

Running it yourself

git clone https://github.com/danolivo/pg_pathcheck
cd /path/to/pg_pathcheck
USE_PGXS=1 PG_CONFIG=/path/to/install/bin/pg_config make install

echo "shared_preload_libraries = 'pg_pathcheck'" > /tmp/ppc.conf
TEMP_CONFIG=/tmp/ppc.conf make check-world

After the run, look into the warnings from every tmp_check/log/*.log, log/postmaster.log, results/*.out, and regression.diffs. For interactive debugging, use SET g_patcheck.elevel = error or panic which makes it easy to correlate a specific finding with a specific test query.

The extension targets PostgreSQL master specifically — it uses PG_MODULE_MAGIC_EXT and the extension_state slot API, both recent additions. It registers no SQL objects. CREATE EXTENSION pg_pathcheck does nothing useful. All effects are routed through the planner hooks it installs at library load.

If you try it and find something — on an unmodified master, on a fork, in your extension's test suite — I would like to read about it in comments.

THE END.
April 20, 2026, Madrid, Spain.

Disclosure

Most of pg_pathcheck's code and this post were drafted with the help of a large language model (Claude). Every change was reviewed by a human before being committed, but the prose and structure are largely machine-produced.

This question was recently raised on the pgsql-performance mailing list, and the investigation revealed a somewhat surprising culprit: the column statistics stored in PostgreSQL's pg_statistic table.

The Context

In PostgreSQL, query optimisation relies on various statistical measures, such as MCV, histograms, distinct values, and others - all stored in the pg_statistic table. By default, these statistics are based on samples of up to 100 elements. For larger tables, however, we typically need significantly more samples to ensure reliable estimates. A thousand to 5000 elements might not seem like much when representing billions of rows, but this raises an important question: could large statistical arrays, particularly MCVs on variable-sized columns, seriously impact query planning performance, even if query execution itself is nearly instantaneous?

Investigating the Problem

We're examining a typical auto-generated 1C system query. '1C' is a typical object-relational mapping framework for accounting applications. PostgreSQL version is 17.5. Notably, the default_statistics_target value is set to only 500 elements, even below the recommended value for 1C systems (2500). The query contains 12 joins, but 9 are spread across subplans, and the join search space is limited by three JOINs, which is quite manageable. Looking at the EXPLAIN output, the planner touches only 5 buffer pages during planning - not much.

Interestingly, the alternative PostgreSQL fork (such forks have become increasingly popular these days) executed this query with nearly identical execution plans, and the planning time is considerably shorter - around 80 milliseconds. Let's use this as our control sample.

The Hunt for Root Cause

The first suspicion was obvious: perhaps the developers expanded the optimiser's search space, and it's simply passing through multiple extra paths. A flamegraph comparison between the slow planning case and the alternative fork showed remarkably similar patterns. Both exhibited search space expansion from features standard in 1C-related PostgreSQL forks (Joinsel and 'Append of IndexScans'), but nothing surprising beyond that.

However, the detailed analysis of the flamegraph revealed something more telling: a performance bottleneck in the byteaeq() comparison operation, triggered by the cost_index() function's cost estimation and toast_raw_datum_size() calls. The optimiser invokes this repeatedly while evaluating all possible index combinations across various expressions - not just those explicitly mentioned in the query, but also derived ones through 'equivalence classes' created by equality operations.

The query references just three columns: inforg10621::fld10622rref, inforg10621::fld15131rref, and inforg8199::fld8200_rrref. Yet these are involved in 20 different expressions, 15 of which are join clauses. When you factor in the number of indexes on these tables - eight between the two - it becomes clear that the number of possible combinations can explode. But how can we confirm this suspicion? How many times does the optimiser actually consult table statistics?

Unfortunately, standard PostgreSQL doesn't provide this information directly. So I turned to my own project - pg_index_stats, which uses PostgreSQL's internal hooks (relation_stats_hook and get_index_stats_hook) to collect precisely this data and display it in EXPLAIN output.

Here's what we found (1c and alternative):

The statistics for four columns are being accessed more than 100 times each. Remarkably, for the fld10622rref column, the optimiser fetches, decompresses, and uses the statistics 217 times! While this is less critical for the fld809 column (which has no histogram or MCV due to its nearly unique nature), other columns require repeatedly decompressing substantial arrays. The alternative fork accesses statistics roughly twice as frequently - a significant improvement, though not quite enough to fully explain the planning time difference.

Digging Deeper

What statistics do we actually have, and in what volume? Comparing statistics dumps from both PostgreSQL versions (here and there) shows that our tables indeed contain MCV and histogram arrays with up to 500 elements for several columns. Their uncompressed size reaches tens of kilobytes (compressed, over 2KB), and extracting them requires decompression before use. Surely we don't need to fetch and decompress these large arrays repeatedly?

After all, PostgreSQL does have caching that should calculate selectivity for a given expression only once …

We have two obvious suspects: columns fld10622rref and fld8201rref. Let's test our hypothesis by mechanically zeroing out their statistics and seeing what happens:

UPDATE pg_statistic
SET stanumbers1 = CASE WHEN stakind1 = 1 THEN NULL ELSE stanumbers1 END,
    Stavalues1 = CASE WHEN stakind1 = 1 THEN NULL ELSE stavalues1 END,
    Stakind1 = CASE WHEN stakind1 = 1 THEN 0 ELSE stakind1 END,
    Stanumbers2 = CASE WHEN stakind2 = 1 THEN NULL ELSE stanumbers2 END,
    Stavalues2 = CASE WHEN stakind2 = 1 THEN NULL ELSE stavalues2 END,
    Stakind2 = CASE WHEN stakind2 = 1 THEN 0 ELSE stakind2 END
WHERE (starelid = ‘_inforg10621’::regclass AND staattnum = (
    SELECT attnum FROM pg_attribute
    WHERE (attrelid = ‘inforg10621’::regclass AND attname = ‘fld10622rref’)))
OR (starelid = ‘_inforg8199’::regclass AND staattnum = (
    SELECT attnum FROM pg_attribute
    WHERE (attrelid = ‘_inforg8199’::regclass AND attname = ‘_fld8201rref’)));

The result? EXPLAIN now shows planning time at around 30ms:

Planning: Buffers: shared hit=5 Memory: used=4030kB allocated=4096kB
Planning Time: 31.347 ms
Execution Time: 0.237 ms

If we delete all statistics entirely with:

DELETE FROM pg_statistic;

We get the theoretical minimum planning time for this query:

Planning: Buffers: shared hit=5 Memory: used=3932kB allocated=4096kB
Planning Time: 18.477 ms
Execution Time: 0.421 ms

This aligns perfectly with the alternative fork's planning time.

But in the current master branch, since commit 057012b, Postgres employs a hashing technique to reduce the N^2 overhead of long MCV array passes. Ok, let's backpatch our case and check the explain:

 Planning:
   Buffers: shared hit=5
   Memory: used=3984kB  allocated=4096kB
 Planning Time: 64.603 ms
 Execution Time: 0.197 ms

It is definitely better than before, but we still see overhead that may grow with larger statistical arrays and repeated detoasting/decompression attempts.

The Verdict

Statistics indeed causes the excessive planning time, but the question remains: is it the overhead of decompressing statistics, or the overhead of repeatedly iterating through long MCV and histogram arrays? The answer is likely both.

We can indirectly confirm the impact of repeatedly traversing MCV arrays by noting that changing the storage type of columns in pg_statistic from EXTENDED to EXTERNAL produces no measurable difference:

DELETE FROM pg_statistic;
SET allow_system_table_mods = ‘on’;
ALTER TABLE pg_statistic ALTER COLUMN stavalues1 SET STORAGE EXTERNAL;
…
VACUUM ANALYZE;

Conclusion and Solutions

The root cause is clear: the optimiser's search space expanded due to increased index counts and statistics sizes - both entirely legitimate scenarios that can occur beyond ORM applications. The execution itself remains efficient and doesn't consume significant disk or memory resources, so it doesn't significantly impact neighbouring operations. However, the planning time can become problematic.

What Can Be Done?

First approach: Implement a caching system for frequently accessed, extensive statistics. This could even be implemented as an extension (similar to how I collected statistics access patterns in pg_index_stats). The code wouldn't be overly complex - just a standard module allocating a DSM segment for a hash table and decompressed statistics. Additionally, it's worth exploring a balance and perhaps storing MCVs in sorted order (when the data type allows), enabling fast element matching on both sides during JOIN estimation and quick lookup during filter estimation.

Second approach: You can just reduce the statistics size on problematic tables or columns:

ALTER TABLE table_name ALTER COLUMN column_name SET STATISTICS 0;

Of course, the challenge here is detecting the problematic spots (columns, clauses) inside the query. There's no universal answer - you need to EXPLAIN on suspicious queries with and without statistics, then perform the same analysis I did above. And naturally, report findings to the vendor, because there's always room for improvement!

THE END.
Istanbul, Turkey. January 26, 2026.

Introduction

Temporary tables in PostgreSQL have always been parallel restricted. From my perspective, the reasoning is straightforward: temporary tables exist primarily to compensate for the absence of relational variables, and for performance reasons, they should remain as simple as possible. Since PostgreSQL parallel workers behave like separate backends, they don't have access to the leader process's local state, where temporary tables reside. Supporting parallel operations on temporary tables would significantly increase the complexity of this machinery.

However, we now have at least two working implementations of parallel temporary table support: Postgres Pro and Tantor. One more reason: identification of temporary tables within a UTILITY command is an essential step toward auto DDL in logical replication. So, maybe it is time to propose such a feature for PostgreSQL core.

After numerous code improvements over the years, AFAICS, only one fundamental problem remains: temporary buffer pages are local to the leader process. If these pages don't match the on-disk table state, parallel workers cannot access the data.

A comment in the code (80558c1) made by Robert Haas in 2015 clarifies the state of the art:

/*
 * Currently, parallel workers can't access the leader's temporary
 * tables.  We could possibly relax this if we wrote all of its
 * local buffers at the start of the query and made no changes
 * thereafter (maybe we could allow hint bit changes), and if we
 * taught the workers to read them.  Writing a large number of
 * temporary buffers could be expensive, though, and we don't have
 * the rest of the necessary infrastructure right now anyway.  So
 * for now, bail out if we see a temporary table.
 */

The comment hints at a path forward: if we flush the leader's temporary buffers to disk before launching parallel operations, workers can safely read from the shared on-disk state. The concern, however, is cost - would the overhead of writing those buffers outweigh the benefits of parallelism?

On the path to enabling parallel temporary table scans, this cost argument is fundamental and must be addressed first. We can resolve this issue by providing the optimiser with a proper cost model. In this case, it could choose between a parallel scan with buffer flushing overhead and a sequential scan performed by parallel workers. Hence, we are looking for a constant, such as DEFAULT_SEQ_PAGE_COST, to estimate writing overhead. Let's address this question with actual data and measure the cost of flushing temporary buffers. My goal is to determine whether this overhead represents a real barrier to parallel execution or simply an overestimated concern that has kept this optimisation off the table.

Benchmarking tools

PostgreSQL currently provides no direct access to local buffers for measurement purposes. To conduct this benchmark, I extended the system with several instrumentation tools and UI functions. The temp-buffers-sandbox branch, based on the current PostgreSQL master, contains all the modifications needed for this work.

The implementation consists of two key commits:

No.1: Statistics infrastructure

This commit introduces two new internal statistics that track local buffer state:

• allocated_localbufs - tracks the total number of local buffers currently allocated in this backend (it can't be more than the temp_buffers value).

• dirtied_localbufs - counts how many local buffer pages are dirty (not flushed to disk).

I believe these statistics potentially provide the foundation for the cost model, giving the query optimiser visibility into the current state of temporary buffers before deciding whether to flush them.

No.2: UI functions

This commit adds SQL-callable functions that allow direct manipulation and inspection of local buffers:

• pg_allocated_local_buffers() - returns the count of currently allocated local buffers.

• pg_flush_local_buffers() - explicitly flushes all dirty local buffers to disk.

• pg_read_temp_relation(relname, randomize) - reads all blocks of a temporary table either sequentially or in random order.

• pg_temp_buffers_dirty(relname) - marks all pages of a temporary table as dirty in the buffer pool.

If local buffers are free or not allocated yet, reading a relation block in a random order simulates a random distribution in memory. So, the following flush operation of these pages to the disk serves as a simulation of a simple 'random-write' mode. These functions enable almost direct measurement of read and write operations.

Methodology

The complete test bench is available here.

Fortunately, local buffer operations are pretty straightforward: they don't acquire locks, don't require WAL logging, and avoid other costly manipulations. This eliminates concurrency concerns and simplifies the test logic. To build a cost estimation model, we need to measure three things: write speed, read speed, and the overhead of scanning buffers when no I/O is required.

The ratio of read to write speed will allow us to derive a write-page-cost parameter based on the DEFAULT_SEQ_PAGE_COST value used in core PostgreSQL. The optimiser can use this parameter to estimate the cost of flushing dirty local buffers before parallel operations begin.

Each test iteration follows this algorithm:

Sequential access testing:

1. Create a temp table and fill it with data that fits within the local buffer pool (all pages will be dirty in memory).

2. Call pg_flush_local_buffers() to write all dirty buffers to disk. Measure I/O.

3. Call pg_flush_local_buffers() again to measure the overhead of scanning buffers without actual flush (dry-write-run).

4. Evict the test table’s pages by creating a dummy table that fills the entire buffer pool, then drop it.

5. Call pg_read_temp_relation('test', false) to read all blocks sequentially from disk into buffers. Measure I/O.

6. Call pg_read_temp_relation('test', false) again to measure the overhead of scanning buffers without an actual read (dry-read-run).

Random access testing:

1. Evict the test table's pages again by creating and dropping a dummy table.

2. Call pg_read_temp_relation('test', true) to read blocks in random order, distributing them randomly across the buffer pool.

3. Call pg_temp_buffers_dirty('test') to mark all table pages as dirty.

4. Call pg_flush_local_buffers() to flush pages to disk. Since pages were loaded randomly into buffers, this pretends to simulate random write patterns.

All measurements are captured using EXPLAIN (ANALYZE, BUFFERS), which records execution time in milliseconds and buffer I/O statistics (local read, local written, local hit counts). Planning time is negligible (typically < 0.02ms) and excluded from analysis. While it's possible to avoid EXPLAIN and the Instrumentation overhead entirely, I believe this overhead is minimal and consistent between write and read operations. Using EXPLAIN provides a convenient way to verify execution time and confirm the actual number of blocks affected.

The tests cover buffer pool sizes at powers of 2 from 128 to 262,144 blocks (1MB to 2GB), with 30 iterations per size for statistical reliability. Each test allocates 101% of the target block count to accommodate Free Space Map and Visibility Map metadata. Higher buffer counts cause memory swapping and produce unreliable results.

Benchmark results

On my laptop, the most stable performance occurs in the 4-512 MB range:

Large datasets show higher write overhead and variability:

Scanning without I/O (Dry-Run) is minimal, around 0.002-0.240 ms.

Based on the results, the temp table write cost should account for both sequential overhead and random access patterns. The analysis shows:

Sequential write overhead: Approximately 20% slower than sequential reads.

Random access degradation: Random buffer distribution patterns show an additional 10-24% performance degradation compared to sequential access. This occurs when temporary table pages are scattered across the buffer pool due to interleaved operations or random access patterns.

Recommended cost formula:

DEFAULT_WRITE_TEMP_PAGE_COST = 1.30 × DEFAULT_SEQ_PAGE_COST

This 1.30 multiplier accounts for both the sequential write overhead (~1.20) and the random-access degradation (~0.10), providing a conservative estimate that covers realistic workload scenarios where buffer access patterns may not be purely sequential.

Write cost is relatively close to read cost because no WAL logging is required for temporary tables. I’m uncertain what storage type the current default seq_page_cost targets; my measurements were conducted on an NVMe SSD. Would the relationship differ on HDD? Further investigation into different storage types may be warranted.

Also, tests indicate that we can estimate the buffer-scanning overhead at approximately 1% of the writing cost. Hence, the whole formula for the preliminary temporary buffers flushing may look like (DEFAULT_SEQ_PAGE_COST = 1):

flush_cost = 1.30 × dirtied_localbufs + 0.01 × allocated_localbufs

What’s next?

This benchmark provides the foundational cost model needed to enable parallel query execution on temporary tables in PostgreSQL. The whole implementation requires four key development phases:

1. Add a planner flag to signal when a plan subtree contains operations on temporary objects. This allows the planner to identify when buffer flushing may be required for parallel execution. I hope that existing parallel_safe and consider_parallel flags may be modified to serve this purpose.

2. Implement the buffer flush operation in Gather and GatherMerge nodes before launching parallel workers. This ensures that all dirty temporary table pages are synchronised to disk before workers begin execution.

3. Enable parallel workers to access the leader process’s temporary table data from disk. This requires teaching workers how to locate and read temporary table files written by the leader process.

4. Integrate the cost model into the query planner. The planner can then make informed decisions about whether flushing temporary buffers for parallel execution will outperform sequential execution without parallel workers.

Conclusion

• A flush of the local buffer is approximately 30% slower than a sequential read.

• For optimisation, the default write cost may be hardcoded to 1.3*DEFAULT_SEQ_PAGE_COST.

• 360 measurements with 30 iterations per size. Medium datasets (16-512 MB) show a coefficient of variation consistently below 6%, indicating highly stable results. Large datasets (1-2 GB) show higher variability (CV >150% for writes), requiring careful interpretation.

THE END.
January 05, 2026, Madrid, Spain.

In this context, I would like to revisit the discussion about the Postgres-based multi-master problem, including its practical value, feasibility, and the technology stack that needs to be developed to address it. By narrowing the focus of the problem, we may be able to devise a solution that benefits the industry.

I spent several years developing the multi-master extension in the late 2010s until it became clear that the concept of essentially consistent multi-master replication had reached a dead end. Now, after taking a long break from working on replication, changing countries, residency, and companies, I am revisiting the Postgres-based multi-master idea to explore its practical applications.

First, I want to clarify the general use case for multi-master replication and highlight its potential benefits. Apparently, any technology must balance its capabilities with the needs it aims to address. Let's explore this balance within the context of multi-master replication.

Typically, clients consider a multi-master solution when they hit a limit in connection counts for their OLTP workloads. They often have a large number of clients, an N transaction-per-second (TPS) workload, and a single database. They envision a solution that involves adding another identical server, setting up active-active replication, and doubling their workload.

Sometimes, a client has a distributed application that requires a stable connection to the database across different geographic locations. At other times, clients simply desire a reliable automatic failover. A less common request, though still valuable, is to provide an online upgrade or to enable the detachment of a node for maintenance. Additionally, some clients may request an extra master node that has warmed-up caches and a storage state that closely resembles the production environment, allowing it to be used for testing and benchmarking. In summary, there are numerous tasks that can be requested, but what can realistically be achieved?

It's important to remember that active-active replication in PostgreSQL is currently only feasible with logical replication, which denotes network latency and additional server load from decoding, walsender, and so on. Network latency also immediately arises - we need to wait for confirmation from the remote node that the transaction was successfully applied, right? Therefore, the idea of scaling the write load for the general case of 100% replication immediately encounters the fact that each server will be required to write not only its own changes, but also changes from other instances (see the figure above). That isn't a problem for massive queries with bushy SELECTs, but a multi-master query is more likely to be used by clients with pure OLTP and very simple DML.

We face a similar challenge when trying to provide each copy of the distributed application with a nearby database instance. If the application's connection to the remote database is weak, then the connection between the two DBMS instances will also be unstable. As a result, waiting for confirmation of a successful transaction commit on the remote node can lead to significant delays.

Complex situations can also occur when a replication update tries to overwrite the same table row that has been updated locally. Such an event can happen because we have no guarantee that the snapshots of the transactions that caused these conflicting changes are consistent across DBMS instances. This raises the question: whose change should be applied, and whose should be rolled back? Does a single row change within the transaction logic need to take into account the competing change for it to be valid?

The autofailover case is relatively straightforward, but something still needs to be done to make such a configuration effective: after all, if all instances can write, then a transaction commit must be accompanied by a supplemental message ensuring that the transaction is written at each instance. Otherwise, it could happen that if node N_x crashes, some of its transactions will be written to the database on node N_y, but will not be committed to (or will be rolled back in) the database of node N_z. So, how do you fix this situation except by sending the entire configuration to recovery?

So, the concept of multi-master replication can be questionable, particularly for those seeking to accelerate OLTP workload. So, why would anyone need it? Let's begin by examining the underlying technology: logical replication.

I see two significant advantages to logical replication. First, it enables highly selective data replication, allowing you to pick only specific tables. Additionally, for each table, you can set up filters with replication conditions that let you easily skip individual records or entire transactions right at the outset of the replication process during the decoding phase. This feature provides a highly granular mechanism for selecting specific data that should be synchronised with a remote system.

The second notable advantage is the high-level nature of the mechanism. This type of replication occurs at the level of relational algebra, which means you can abstract away the complexities of physical storage.

What amazing benefits come with a high level of abstraction? Imagine the possibilities! You can customise different sets of indexes on synchronised nodes, which significantly reduces DML overhead and allows you to route queries based on where execution can be most effective. For example, you could focus on loading one instance with brief UPDATE/DELETE queries on primary keys, while reserving another instance for larger subqueries or INSERTs that usually don't conflict with updates. You could even mix it up by using a traditional Postgres heap on one instance and a column storage on another! The creativity here knows no bounds when it comes to the potential of your replication protocol.

Now that we have outlined the benefits of logical replication, let's consider a use case that can be effectively implemented using a multi-master configuration.

To begin, we will set aside concerns related to upgrades, maintenance, and failovers. The most apparent use case is for supporting a geo-distributed application. By categorising the data in the database into three types - critically important general data, general data that is changed on only one side, and purely local data - we can leverage the advantages of this setup (see figure).

Here, the red rectangle denotes data that must be reliably synchronised between instances. The green and blue denote data that doesn't require immediate synchronisation and should be accessible to the remote instance in read-only mode. The grey denotes purely local data.

By designing the database schema to categorise data by replication method, we can even reduce the database size on a specific instance by avoiding the transmission of local data to remote nodes. Furthermore, plenty of data can be replicated asynchronously in one direction, avoiding the overhead of waiting for the remote end to confirm a commit. Only critical data requires strict synchronisation, using mechanisms such as synchronous commit, 2PC, and at least REPEATABLE READ isolation level, which enormously raises the commit time of such a transaction and increases the risk of rollback due to conflicts.

What is an example of this use case? To be honest, I don't have any experience with customer installations, so I can only imagine how it might work hypothetically. I envision an international company that needs to separate employee data and fiscal metrics on servers located in each country, which seems to be a common requirement these days. For analytical purposes, this data could be made accessible externally, similar to how key values can filter replicated data.

The company's employee table could be divided, replicating names, positions, salaries, and other relevant information across all database instances. Sensitive identifiers, such as social security numbers or passport numbers, could be kept in a separate local table to maintain privacy.

In principle, if updates to local or asynchronously replicated data dominate, it may be possible to achieve the desired scalability for writing operations (sounds wicked, but who knows...).

Drawing from my experience in rocket science, I've developed the habit of qualitatively evaluating the effects of the phenomena being studied beforehand. Let's estimate the percentage of the database that can be replicated in active-active mode without potentially degrading performance. For simplicity, let's assume there are two company branches located on different continents, and consider the following configuration options: (1) one server, or (2) two servers operating in multi-master mode, where access will always be local (as illustrated in the figure below).

Let's introduce some notations. Please refer to the figure above for further clarification:

• T_l - transaction execution time in a DBMS backend, ms.

• T^c_l - network round-trip time between DBMS and local application, ms.$

• T^c_r - network round-trip time between DBMS and remote application, ms.

• T_r - extra time to ensure that the transaction is successfully committed across the DBMS cluster, ms.

• X_l - fraction of local connections.

• N - fraction of transactions that are OK with asynchronous replication guarantees.

For a single server holding all the connections we have:

Active-active replication has the following formula:

Now, let's determine appropriate numbers for our formulas. We'll assume a 50% share for local connections (X_l = 0.5). Drawing from my experience living in Asia and connecting to resources in Europe, we can use the following figures as a reference:

In this context, T^c_l and T^c_r are the time of one round-trip. In contrast, confirming a remote commit (T_r) usually requires at least two round-trips: in the 2PC protocol, the PREPARE STATEMENT command should first be executed, waiting for the changes to be successfully replicated and all the resources necessary for the commit to be reserved, and then the COMMIT command should be issued.

Based on these timing considerations, we can calculate the following:

Now, let's imagine that the number of remote connections has grown by 80%:

What if we need to ensure full synchronous 2PC synchronisation of the entire database? Let's do the math:

These numbers indicate a performance loss of approximately 2.5 times, even in the most optimistic scenario. While it's not particularly encouraging, may it be sufficient for some applications?

This rough calculation suggests that if approximately 25% of DML transactions require remote confirmation, the multi-master system has a chance to stay with the same writing performance. If the majority of traffic originates from remote regions, the proportion of reliably replicated data could increase to 40%. However, for a more conservative estimate, let's stick with N = 25%. This approach also eases some of the load on the disk subsystem, locks, and other resources, allowing them to be used for local operations such as VACUUM or read-only queries. There appears to be a grain of truth in that, doesn't it?

On the other hand, replication, even if asynchronous, must be able to keep up with the commit flow. If the total time required for local transaction execution is 15 ms, and the one-way delay to the remote server is 75 ms, then even without waiting for confirmation from the remote side, a queue of changes for replication will still accumulate in a sequential scenario.

25% of the DML in our computation is committed with remote confirmation, leaving 75% to be replicated. 75 ms * 0.75 = 56 ms. To address the disparity between the rate of local commits and the speed of data transfer to the remote server, we must utilise the bandwidth by sending and receiving data on the remote server in parallel (i.e., parallel replication is required). In our rough model, it turns out that four threads are sufficient to transfer changes. Considering the freed-up server resources (resulting from distributing connections between instances), this seems quite realistic.

So, the bottom line is that by distributing data geographically in multi-master mode, we can theoretically expect comparable transaction handling speed. This also reduces the number of backends and resource consumption on each server. These resources can be freed up for system processes and various analytics. Let's not forget the ability to optimise indexes, storage, and other attributes of physical data placement independently on each system node. An additional benefit is that in the event of a connection failure, each subnet can be temporarily maintained with the expectation that, upon recovery, a conflict resolution strategy will restore the database's integrity.

It's easy to imagine an application for such a scenario with a complete network breakdown - for example, a database for a hospital network in a region with complex terrain and climate. The medical records database must be shared, but a single client is unlikely to be served by two different hospitals within a short period of time, making conflicts in critical data quite rare.

Taking all of the above into account, a viable multi-master solution should implement the following set of technologies:

• Replication sets – to classify data for replication, providing separate synchronous and asynchronous replication.

• Replication type dependency detection - check that synchronously replicated tables don't refer to asynchronously replicated tables.

• Remote commit confirmation (similar to 2PC).

• A distributed consensus protocol for determining a healthy subset of nodes and fencing failed nodes in 3+ configurations.

• Parallel replication – parallelising both DML sending and application on the remote side.

• Automatic Conflict Resolution.

Let's not overlook the importance of automatic failover and the capability to hot-swap hardware without any downtime. Currently, the concept of alternative physical storage arrangements still sounds a little wild, so I'm excluding it from our discussion for now. However, if we gain more experience with successful multi-master implementations, this option might eventually become the preferred approach.

That's all for now. This post is meant to spark discussion, so please feel free to share your thoughts in the comments or through any other method you prefer.

THE END.
October 18, 2025, Madrid, Spain.

This index:

CREATE INDEX ON order_events ((event_payload ->> 'terminal'::text),
                              event_type,event_created); -- (1)

At first (purely formal) glance, this index should not be much better than the alternatives:

CREATE INDEX ON order_events (event_created,
                              (event_payload ->> 'terminal'::text),
                              event_type); -- (2)
CREATE INDEX ON order_events (event_created, event_type); -- (3)

However, the observed speedup is significant; in fact, the performance of index (1) surpasses index (2) by more than 50 times and exceeds index (3) by almost 25 times!

The advantages of the proposed index are evident when we consider the logic of the subject area. It is more selective and is less likely to retrieve rows that do not match the filter. For instance, if we first identify all the rows that correspond to a specific airport, we can then focus on the boundaries of the date range. At this point, all retrieved rows will already meet the filter criteria. Conversely, if we begin by determining the date range, we may encounter numerous rows related to other terminals within that range.

However, when examining the EXPLAIN output, we do not see any distinctive reasons:

-- (1)
->  Index Scan using order_events_expr_event_type_event_created_idx
      (cost=0.57..259038.66 rows=64540 width=72)
      (actual time=0.095..232.855 rows=204053.00 loops=1)
    Index Cond:
      event_payload ->> 'terminal' = ANY ('{Berlin,Hamburg,Munich}' AND
      event_type = ANY ('{Created,Departed,Delivered}') AND
      event_created >= '2024-01-01 00:00:00+00' AND
      event_created < '2024-02-01 00:00:00+00'
    Index Searches: 9
    Buffers: shared hit=204566

-- (2)
->  Index Scan using order_events_event_created_event_type_expr_idx
      (cost=0.57..614892.22 rows=64540 width=72)
      (actual time=0.499..14303.685 rows=204053.00 loops=1)
    Index Cond:
      event_created >= '2024-01-01 00:00:00+00' AND
      event_created < '2024-02-01 00:00:00+00' AND
      event_type = ANY ('{Created,Departed,Delivered}' AND
      event_payload ->> 'terminal' = ANY ('{Berlin,Hamburg,Munich}')
    Index Searches: 1
    Buffers: shared hit=279131
                        
-- (3)

->  Index Scan using idx_3
      (cost=0.57..6979008.62 rows=64540 width=72)
      (actual time=0.238..8777.846 rows=204053.00 loops=1)
    Index Cond:
      event_created >= '2024-01-01 00:00:00+00' AND
      event_created < '2024-02-01 00:00:00+00' AND
      event_type = ANY ('{Created,Departed,Delivered}')
    Filter: event_payload ->> 'terminal' = ANY ('{Berlin,Hamburg,Munich}')
    Rows Removed by Filter: 4292642
    Index Searches: 1
    Buffers: shared hit=4509185

Let's say IndexScan on (3) filters a lot of tuples and is therefore slow. However, even after eliminating 4 million rows, IndexScan on (3) is still twice as fast as IndexScan on (2). At the same time, the only difference between indexes (1) and (2) is the order of the columns.

If we compare scans (1) and (2), the only noticeable difference is a 30% difference in the number of buffer pages hit. But not 50 times! That means the EXPLAIN does not show us where the main work was done; only the cost value signals the superiority of index (1).

However, we live in the world of ORM and ad-hoc queries, where it is difficult to choose the order of columns in the index, analysing the meaning of the stored data, which means we need to find out precisely what is happening there and what data is missing for the automated detection of an [un]successful index.

If you look at the optimiser code, it becomes clear in numbers why index (1) is so pleasing: all other things being equal, it is going to go through only 39 out of 1 million index pages. Compare this with index (2), which also contains 1 million pages, and we pass through 73 thousand of them. In terms of index tuples, this is 64.5 thousand versus 14 million. It turns out that the main work is to select a row, extract the appropriate attribute and perform the comparison.

The work performed is not represented in the EXPLAIN output. Additionally, the IndexScan structure of the query plan, which is accessible to the Postgres core and its extensions after the plan has been executed, lacks valuable information necessary for assessing the quality of planning and sources of execution time grow. Consequently, developing a method for automatically identifying ineffective indexes and selecting more optimal alternatives appears to be challenging, if not impossible.

There are numerous parameters that the optimiser calculates during the index scan planning. Take a look at the GenericCosts structure, including numIndexTuples, numIndexPages, indexCorrelation, and indexSelectivity. Having all this information available at the end of execution could help detect scanning anomalies and draw the DBA's attention.

Of course, the number of installations, types of load and cases is close to infinity. Hence, extending the core code by continually adding more data from the optimisation stage to the plan seems not flexible. Moreover, sometimes we would like to have alternative paths that lost the battle but may be beneficial for analysis.

Moreover, since Postgres 18, the core already has a nicely extensible explain, where we may add options, node information, and overall plan information. So, the only step needed is a bridge between the cloud of possible paths and the final plan.

Having this capability would allow extensions to analyse the predictions against the actual outcomes of query execution. Additionally, it would help in making informed decisions for fine-tuning the query planner and developing effective indexing strategies for a table.

Please feel free to share your feedback, whether you agree or disagree with my viewpoint.

THE END.
7 De Agosto De 2025, Torrevieja, España.

In practice, queries tend to be highly selective and do not require processing extensive rows. The focus, then, shifts to the order of JOIN operations, caching intermediate results, and minimising sorting operations. In these scenarios, PostgreSQL, with its wide range of query execution strategies, can indeed have an advantage.

I wanted to explore whether Postgres could be improved by thoroughly utilising all available tools, and for this, I chose the RTABench benchmark. RTABench is a relatively recent benchmark that is described as being close to real-world scenarios and highly selective. One of its advantages is that the queries include expressions involving the JSONB type, which can be challenging to process. Additionally, the Postgres results on RTABench have not been awe-inspiring.

Ultimately, I decided to review all of the benchmark queries, and fortunately, there aren't many, to identify possible optimisations. However, already on the zero query, there were enough nuances that it was worth taking it out into a separate discussion.

My setup isn't the latest - it's a MacBook Pro from 2019 with an Intel processor—so we can't expect impressive or stable performance metrics. Instead, let's concentrate on qualitative characteristics rather than quantitative ones. For this purpose, my hardware setup should be sufficient. You can find the list of non-standard settings for the Postgres instance here.

Now, considering the zero RTABench query, which involves calculating several aggregates over a relatively small sample from the table:

EXPLAIN (ANALYZE, BUFFERS ON, TIMING ON, SETTINGS ON)
WITH hourly_stats AS (
  SELECT 
    date_trunc('hour', event_created) as hour,
    event_payload->>'terminal' AS terminal,
    count(*) AS event_count
  FROM order_events
  WHERE 
    event_created >= '2024-01-01' AND
    event_created < '2024-02-01'
    AND event_type IN ('Created', 'Departed', 'Delivered')
  GROUP BY hour, terminal
)
SELECT 
  hour,
  terminal,
  event_count,
  AVG(event_count) OVER (
    PARTITION BY terminal
    ORDER BY hour
    ROWS BETWEEN 3 PRECEDING AND CURRENT ROW
  ) AS moving_avg_events
FROM hourly_stats
WHERE terminal IN ('Berlin', 'Hamburg', 'Munich')
ORDER BY terminal, hour;

Phase 0. ‘Default behaviour‘

The query does not seem surprising. Let's run a query over the default data schema (EXPLAIN has been cleaned):

WindowAgg  (actual time=21053.119 rows=2232)
  Window: w1 AS (PARTITION BY
   order_events.event_payload ->> 'terminal'
   ORDER BY date_trunc('hour', order_events.event_created))
  Buffers: shared read=3182778
  -> Sort  (actual time=21052.476 rows=2232)
     Sort Key:
      order_events.event_payload ->> 'terminal',
      date_trunc('hour', order_events.event_created)
      Sort Method: quicksort  Memory: 184kB
   -> GroupAggregate  (actual time=21051.875 rows=2232)
       Group Key:
        date_trunc('hour', order_events.event_created),
        order_events.event_payload ->> 'terminal'
     -> Sort (actual time=21037.609..21042.766 rows=204053)
         Sort Key:
          date_trunc('hour', order_events.event_created),
          order_events.event_payload ->> 'terminal'
         Sort Method: quicksort  Memory: 12521kB
       -> Bitmap Heap Scan on order_events
            (actual time=20999.978 rows=204053)
          Recheck Cond: event_type =
           ANY ('{Created,Departed,Delivered}')
          Filter:
           event_created >= '2024-01-01 00:00:00+00' AND
           event_created < '2024-02-01 00:00:00+00' AND
           event_payload ->> 'terminal') =
             ANY ('{Berlin,Hamburg,Munich}')
          Rows Removed by Filter: 57210049
          Heap Blocks: exact=3133832
          Buffers: shared read=3182778
        -> Bitmap Index Scan (actual time=1683.357 rows=57414102)
           Index Cond: event_type = ANY ('{Created,Departed,Delivered}')
           Index Searches: 1
           Buffers: shared read=48946
Execution Time: 21060.564 ms

The execution time is 21 seconds? Really? Seems too slow. Upon examining the EXPLAIN, we realise that the main issue is that the default schema contains only one low-selectivity index, which was used during execution instead of performing a sequential scan. The EXPLAIN also indicates that a significant portion of the work involves collecting identifiers (ctid) of candidate rows, which takes 1.6 seconds. Following that, filtering through these rows results in filtering out 98% of the read rows, which takes 19 seconds.

The first problem was identified quickly: I allocated 8GB for shared_buffers; however, the DBMS limits the amount of buffer space that can be assigned to a single table. The formula for this allocation is quite complex, involving multiple factors, but the NBuffers/MaxBackends ratio applies here. Consequently, with my current settings, PostgreSQL can allocate a maximum of only 2.4GB per table. Therefore, denormalising the entire database into one wide, long table is a bad idea in PostgreSQL, at least for this reason.

Despite the data access pattern in this query plan being somewhat inefficient, let’s first try a straightforward approach to improve the performance by increasing the number of parallel workers:

A very strange graph. Obviously, suppose most of the work is reading from disk and the tuple deformation. In that case, one should not expect a significant effect from parallelism. But where does this jump between 6 and 7 parallel processes come from? Looking at the EXPLAINs, one can easily understand - there was a change in the query plan. BitmapScan was used on a small number of workers, and the optimiser picked SeqScan on a larger number.

So maybe SeqScan should have been used on a small number of workers, too? Let's see how the scanning operation is accelerated separately for BitmapScan and SeqScan, and also watch how the costs of the scanning nodes change (in relative values):

Raw numbers and EXPLAINs can be found here and here.

The thing is that for a small number of workers, BitmapScan has a smaller cost than SeqScan, which in our case does not correlate with execution time. The point here is probably in a delicate balance: 'fewer lines read / but more often went from the index to the table'. It is difficult to say more precisely, since explain does not show such details of the estimate as the expected number of tuples fetched from the disk before filtering, an estimate of the number of fetched disk pages, or an estimate of the proportion of pages that will be found in shared_buffers. On the other hand, the cost model assumes better scalability of SeqScan compared to BitmapScan, which causes the plan to switch to SeqScan. Considering that for SeqScan, a change in the cost value predicts an unreasonably significant increase in performance, this may result in selecting a SeqScan method where it should not. Thus, for now, you should be careful optimising queries by increasing the number of parallel workers.

Phase 1. ‘Typical optimisation‘

Now let's move on and see what a good index will give Postgres. A typical practice is to create an index on the most frequently used highly selective column in filters. For this query, the choice is limited to only one option:

CREATE INDEX idx_1 ON order_events (event_created);

With this index, the optimiser utilises IndexScan to access data, reducing the query execution time (without parallel workers) to 6.5 seconds. Interestingly, the previous query plan accessed buffer pages 3 million times (shared read = 3182778), while in this case, with a more selective scan, it increased to 14 million (shared hit = 14317527). Although there are more trips to the buffer now, in the previous example, each page replaced a previous one in the buffer. In contrast, both the index and the disk pages now fit into the shared buffers, which contributes to the acceleration.

Next, let's explore whether parallel workers will provide additional benefits and examine how the cost model predicts this outcome:

Raw data can be taken here.

Yes, we see some acceleration. The parallelisation effect goes so far that we have to expand the permissible number of workers to 24 to track the impact to the end. And here one of the disadvantages of Postgres showed itself. Although we set all possible hooks to large values, the optimiser hit a hard-wired limit on the number of workers (10 for this table) on the order_events table. We had to bypass it with the command:

ALTER TABLE order_events SET (parallel_workers = 32);

It is pretty sad that the number of workers implicitly depends on the table size estimate. This can be a problem, for example, in the JOIN operator, which determines the required number of workers based on the number of workers requested on the outer side of the join. It is easy to imagine a situation where a petite table, insufficient even for one parallel worker, and a very large one are joined. In this case, the entire jointree may remain non-parallelised just because of one small table!

Another fact can be extracted from the above graph - the cost model for Index Scan is very conservative: with the maximum observed speedup of 8, the model did not show even a two-fold speedup. Hence the conclusions: 1) when using indexes, do not be shy about raising worker limits, and 2) Scan nodes that are more sensitive to the number of workers (we observed this, for example, with SeqScan), can unexpectedly trigger a rebuild of the query plan for the worse.

However, the current index is not the ultimate dream. Let's take a closer look at the scanning node:

Index Scan (actual time=6555.122 rows=204053)
  Index Cond: event_created >= '2024-01-01 00:00:00+00' AND
    event_created < '2024-02-01 00:00:00+00'
  Filter: event_type = ANY ('{Created,Departed,Delivered}' AND
    event_payload ->> 'terminal' = ANY ('{Berlin,Hamburg,Munich}')
  Rows Removed by Filter: 14099758
  Index Searches: 1
  Buffers: shared hit=14317527

Lots of pages touched, lots of lines filtered. Let's see what can be achieved by minimising disk reads.

Phase 2. ‘Reinforced optimisation‘

In this section, we will confidently assume the existence of an 'index adviser' that helps analyse and automatically create composite indexes. These indexes optimise data retrieval by minimising the reading of table rows, thereby adapting the entire system to the incoming load.

In this query, we have several options to consider. We will exclude the GIN index because the event_payload column lacks selectivity. This leaves us with two alternative options:

CREATE INDEX idx_2 ON order_events (event_created, event_type)
  INCLUDE (event_payload);
CREATE INDEX idx_3 ON order_events (event_created, event_type);

The idx_2 variant does not require accessing the table at all, while the idx_3 index can cause both IndexScan and BitmapScan. You can find various EXPLAINs of these indexes here. It’s interesting to note that with the previously created idx_1 index, adding idx_2 does not result in switching to the obviously faster IndexOnlyScan. This suggests that when evaluating the cost of index access, the width of the index plays a significant role. The jsonb field likely increases the size of idx_2 considerably.

Consequently, the idx_3 index has proven to be the most optimal in terms of compactness and the number of selected records when using the BitmapScan method. By closely examining the scanning nodes, we can understand the reasons behind this conclusion:

Bitmap Heap Scan (actual time=1286.430 rows=204053
  Rows Removed by Filter: 4292642
  Heap Blocks: exact=269237
  Buffers: shared hit=313925
  Bitmap Index Scan (actual time=625.170 rows=4496695)
    Index Searches: 1
    Buffers: shared hit=44688

Index Only Scan (actual time=1586.097 rows=204053)
  Rows Removed by Filter: 4292642
  Heap Fetches: 0
  Index Searches: 1
  Buffers: shared hit=2558314

Index Scan (actual time=2847.517 rows=204053)
  Rows Removed by Filter: 4292642
  Index Searches: 1
  Buffers: shared hit=4509185

All three index scans return the same number of rows, perform a single pass through the index, and filter the same number of rows. However, IndexOnlyScan wins over Index Scan due to the fact that it does not go into the table and touches the buffer pages twice as rarely (2.6 million V/S 4.5 million); BitmapScan goes into the buffer even less often (300 thousand times) - after going through the index and collecting tid of candidate rows, it then goes pointwise to the heap, touching each potentially useful page only once.

Let's see how parallel workers now help speed up the query for each type of scanning:

It turns out that in the case of BitmapScan, there is no particular sense in using workers. Having a significant computing resource and low competition between clients, it is worth considering reducing the cost (see the parallel_setup_cost and parallel_tuple_cost parameters) of parallel execution and disabling BitmapScan.

However, the cost model again turned out to be insensitive to the effect of parallelism. And first of all, something should be done here. It was also noted that with 8+ workers, the plan was again rebuilt on SeqScan, which led to an increase in execution time from ~1c to ~21c. Therefore, in the interests of the experiment, SeqScan had to be manually disabled.

However, even with such a good index, we see that a certain number of lines have to be filtered. Let's go all the way and organise selective access to only relevant data.

Phase 3. ‘Crazy optimisation‘

Let's now try to reach the theoretical limit of optimisation of this query. Here, we can imagine having an advanced 'Disk Access Tuner' that analyses various expressions of the SQL query to find combinations of high and low selective filters on the same table, which is a good reason to consider partial indexes. Let's create the following ideal index:

CREATE INDEX idx_5 ON order_events (event_created, event_type)
INCLUDE (event_payload)
WHERE
  event_created >= '2024-01-01' and event_created < '2024-02-01' AND
  event_type IN ('Created', 'Departed', 'Delivered') AND
  (event_payload ->> 'terminal') = ANY ('{Berlin,Hamburg,Munich}');

The index is designed so that no table access is necessary, and all rows in this index are relevant to the query. Therefore, there is no need to evaluate the filter value, which also helps save CPU time during the execution. The base case (without workers) now executes in 54 ms (as shown in the EXPLAINs here).

In such a straightforward scenario, it's clear that the estimation of the cardinality for the scan operator is made with an error:

Index Only Scan (cost=0.42..4998.13 rows=70210)
                (actual time=34.718 rows=204053.00 loops=1)
    Heap Fetches: 0
    Index Searches: 1
    Buffers: shared hit=110862

Scanning does not rely on a filter; the table is, in fact, static, yet it still makes errors! In this instance, it may not be significant, but if there were a join tree above, an incorrect estimate at a leaf node could lead to a substantial error when selecting a join strategy. Why can't the optimiser refine the selectivity of the sample based on the existing indexes? Let's also explore whether scaling is effective here with parallel workers. Due to the increased share of the remaining (non-parallel) portion of the query, we will focus solely on the actual time and estimated cost of the scan nodes themselves:

Here, as in the previous case, it is clear that the IndexOnlyScan scanning effect ends at three workers. But the cost model does not even show this. Why does the cost model reflect the real numbers so poorly? Perhaps it is conservative due to the diversity of hardware parallelism models and, as a result, the different impact of parallelisation on the query execution. Or maybe there is an implicit assumption that there is a neighbouring backend nearby that will compete for the resource? In any case, I would personally like to have an explicit parameter that allows me to configure this effect, given the nature of my system's load.

Conclusion

What lessons can be drawn from this simple experiment?

1. Parallel workers significantly affect performance, so it's vital to adjust the optimiser's cost model to align with the server's capabilities, increasing the proportion of parallel plans and the number of workers.

2. The efficiency of parallelisation is highly dependent on the access technique, and preference should be given to IndexScan over BitmapScan, SeqScan and even IndexOnlyScan.

3. It appears that the cost model for parallelism in PostgreSQL has not been sufficiently polished, potentially leading to side effects such as defaulting to inefficient SeqScan operations.

4. Considering the weak points of the current PostgreSQL row storage, it lacks the ability to adjust the set of indexes to optimise data access based on the actual workload.

5. More deeply employing indexes in the planning process may provide worthwhile improvements in cardinality estimations.

THE END.
July 26, 2025, Madrid, Spain.

The obvious way out of this situation is to use prepared statements, initially intended for reusing labour-intensive parts such as parse trees and query plans. For more specifics, let's look at a simple table scan with a large number of partitions (see initialisation script):

EXPLAIN (ANALYZE, COSTS OFF, MEMORY, TIMING OFF)
SELECT * FROM test WHERE y = 127;

/*
...
   ->  Seq Scan on l256 test_256
         Filter: (y = 127)
 Planning:
   Buffers: shared hit=1536
   Memory: used=3787kB  allocated=4104kB
 Planning Time: 61.272 ms
 Execution Time: 4.929 ms
*/

In this scenario involving a selection from a table with 256 partitions, my laptop's PostgreSQL took approximately 60 milliseconds for the planning phase and only 5 milliseconds for execution. During the planning process, it allocated 4 MB of RAM and accessed 1,500 data pages. Quite substantial overhead for a production environment! In this case, PostgreSQL has generated a custom plan that is compiled anew each time the query is executed, choosing an execution strategy based on the query parameter values during optimisation. To improve efficiency, let's parameterise this query and store it in the 'Plan Cache' of the backend by executing PREPARE:

PREPARE tst (integer) AS SELECT * FROM test WHERE y = $1;
EXPLAIN (ANALYZE, COSTS OFF, MEMORY, TIMING OFF) EXECUTE tst(127);

/*
...
   ->  Seq Scan on l256 test_256
         Filter: (y = $1)
 Planning:
   Buffers: shared hit=1536
   Memory: used=3772kB  allocated=4120kB
 Planning Time: 59.525 ms
 Execution Time: 5.184 ms
*/

The planning workload remains the same since a custom plan has been used. Let's force the backend to generate and use a 'generic' plan:

SET plan_cache_mode = 'force_generic_plan';
EXPLAIN (ANALYZE, COSTS OFF, MEMORY, TIMING OFF) EXECUTE tst(127);

/*
...
  ->  Seq Scan on l256 test_256
         Filter: (y = $1)
 Planning:
   Memory: used=4kB  allocated=24kB
 Planning Time: 0.272 ms
 Execution Time: 2.810 ms
*/

The first time the query is executed, a generic execution plan is created (we are using forced mode here to keep the example straightforward). This process requires resources nearly equivalent to those needed for building a custom plan. However, when the query is executed again, the generic plan can be quickly retrieved from the cache. As a result, the time spent preparing the query plan drops to just 0.2 ms, memory usage is only 24 KB, and no data page reads are required. It seems we have a clear benefit!

However, my suggestion to use the PREPARE command has often been met with rejection and scepticism. This is primarily due to the problems that arise with generic plans in practice, particularly regarding their updating (replanning) and switching to a custom plan type. To gain a clearer understanding of how the generic plan mechanism is structured and to explore the root of these issues, I decided to investigate the history of this project. Additionally, I aimed to experiment with a new publication format, such as a mailing list review.

What’s wrong with a generic plan?

Upon examining the Git history of PostgreSQL, it appears that the concept of the plan cache was introduced in 2007 with commit b9527e9. At that time, it was decided that each prepared query in PostgreSQL should be executed exclusively using a generic plan, thereby avoiding unnecessary time spent on rebuilding the plan. Unlike Oracle, SQL Server, DB2 and other colleagues in the shop (see link, link, link, and link), PostgreSQL constructs the generic plan with the 'total uncertainty' concept, without utilising any specific 'reference' parameter value. For instance, in the example mentioned, the constant '127' is set aside during the creation of the generic plan.

Due to its limited ability to estimate scan selectivities, the optimiser often depends on default 'magic' values of certain predefined constants. Consequently, a generic plan is often of lower quality compared to a custom one. Let me provide another example to illustrate this point more clearly (see the reproduction script):

EXPLAIN
SELECT * FROM test_2
WHERE
  start_date > '2025-06-30'::timestamp - '7 days'::interval;
/*
 Index Scan using test_2_start_date_idx on test_2  (rows=739)
   Index Cond: (start_date > '2025-06-23 00:00:00'::timestamp)
*/

PREPARE tst3(timestamp) AS SELECT * FROM test_2
  WHERE start_date > $1 - '7 days'::interval;

EXPLAIN EXECUTE tst3('2025-06-30'::timestamp);
/*
 Seq Scan on test_2  (rows=333333)
   Filter: (start_date > ($1 - '7 days'::interval))
*/

Offhand, here are some key reasons to consider: the lack of a constant in the inequality operator results in a filter estimate of 33%; for range filters, the default value is set at 0.5% of the total number of rows in the table; with the equality operator, using MCV statistics is not possible, so we must rely solely on the ndistinct value. Additionally, in certain situations, it is not feasible to use partial indexes.

Let's turn to the origins

The absence of alternatives resulted in a significant decline in performance and the infrequent use of the generally practical PREPARE/EXECUTE statement construct. In 2011, a discussion began that ultimately led to the e6faf91 commit, which introduced a simple automatic technique for switching between custom and generic plan variants.

This discussion began with the pressing issue that prepared statements were executed exclusively using generic plans (Mark Mielke, link). While these plans were rebuilt each time an invalidation signal was received, such as after executing the ANALYZE or ALTER TABLE commands, the quality of the planning was noticeably inferior.

Several ideas were proposed to address this problem:

1. Periodically, replan the generic plan (Jeroen Vermeulen, link).

2. Introduce a threshold for the 'planning/execution time' ratio - If the criterion value is greater than 100, then use only the generic plan; if less than 0.01, then only the custom plan. (Bart Samwel link. Yeb Havinga opposes (link) this idea - an objective criterion should not contain the 'time' parameter). However, Jeroen Vermeulen and Greg Stark (link) supported this idea with the clause that the difference between planning and execution times should be significant, amounting to orders of magnitude.

3. Track the standard deviation (stddev) value of various parameters for executing a specific query plan, which will enable estimating the probability of how long the query will take to plan and execute next time (Greg Stark, link).

4. Build several custom and generic plans, and make a choice based on the cost ratio (Tom Lane, link).

5. Abandon generic plans altogether, while reducing the cost of replanning by preserving the PlannerInfo optimiser 'cache' and replanning only that part of the jointree / subquery where the parameters are actually used (Yeb Havinga, link).

6. Use generic plans, but introduce a replanning criterion - whether the parameter value falls within the MCV or not (Robert Haas (link, link), supported by Jeff Davis).

Interestingly, the idea of re-optimisation was already being discussed back then (Richard Huxton, link). At that time, it was more of a dream, but by the 2020s, the code infrastructure had matured enough to allow us to implement a similar concept in a short time (see replan). The approach of detecting, generalising, and caching frequently arriving statements through a simple protocol, which we implemented in sr_plan, is also explicitly described here (Robert Haas, link), along with Yeb Havinga's idea of achieving this through a method similar to the then non-existent queryId (link).

At the same time, in 2011, Simon Riggs introduced the concept of a one-shot plan. The primary idea behind this type of plan is to inform the DBMS that a query plan will be created, executed immediately, and subsequently destroyed upon completion. This approach allows for the application of additional optimisations that are not relevant when there is no connection between the planning and execution phases.

To support this idea, Simon provided an example involving the calculation of stable functions, which would enable more efficient execution of partition pruning. Additionally, Bruce Momjian highlighted another potential optimisation that could be implemented in a one-shot plan: analysing the buffer cache to assess the effectiveness of using a specific index.

Meanwhile, Tom Lane was developing a similar feature, motivated by complaints about regressions in dynamic SQL queries (link, link). However, his approach was different from Simon Riggs' original concept. Tom Lane's idea focused on unifying the mechanisms of SPI, PREPARE, and the extended protocol through the use of a plan cache. As a result, Riggs' original idea did not receive much further development, though it was discussed later on (link, link).

The concept of tracking the planning and execution time of queries did not gain traction due to objections from Tom Lane, who argued against using this time characteristic, as it is inherently unpredictable and can behave inconsistently across different systems.

In 2017, Pavel Stehule raised the need for explicit control over the type of plan selected when invoking the plan cache. This discussion led to the introduction of the plan_cache_mode parameter, which has two options: force_generic_plan and force_custom_plan. These options are designed specifically for using generic and custom plan types, respectively.

What stands out to me as a developer is the emphasis on several key concepts from the Postgres core that emerged during these discussions.

1. Tom Lane pointed out that in the absence of a general solution, we should develop heuristics. Providing users with such solutions through an additional GUC is a poor idea and ultimately a compromise.

2. Greg Stark and Pavel Stehule emphasised that the predictability of execution is more important than speed.

3. Tom Lane also noted that the ability to switch between different query plan types is valuable, provided it is controlled on a per-query basis.

Outcomes

Analysing the history of feature creation, the opinions expressed within the community, and the current knowledge base on generic plans' usage experience, I conclude that many of the current problems stem from the following issues:

1. Unstable Performance. Generic plan performance may vary significantly based on different sets of input parameter values. This suggests a need to switch to a custom plan type. However, PostgreSQL cannot automatically detect and switch plans because it lacks any statistics on the query execution. The current state of the kernel code enables a straightforward implementation to track various execution parameters, including the average and standard deviation. But, before we proceed with a community's proposal, we must address a fundamental question: should the PostgreSQL kernel have a feedback system from the executor to the optimiser?

2. Outdated custom/generic cost proportion. When a plan is invalidated, for instance, due to updated table statistics, the generic plan is rebuilt, and its cost is recalculated. However, this does not happen for the custom plan. Since the custom plan's cost is not recalculated, the value stored in the plan cache may significantly differ from reality due to gradual changes in table contents. This discrepancy can often lead to situations where a generic plan is utilised, even though the efficiency of a custom plan is apparent and could be determined by the planner during replanning.

3. Inadequate plan costs. A common issue arises when erroneous estimates make the query plan costs irrelevant to the actual workload. Consequently, the choice between custom and generic plans becomes largely a matter of chance.

What can we propose?

After many years of development and testing of the code, can we generate any new ideas? As usual, there are two separate solution designs: one is an in-core part for the community, and the other is an extensible code, which may even include a core patch that could be incorporated into a Postgres fork.

For the core version, we can consider the option of resetting the custom plan statistics on the cached plan, similar to what we do for the generic plan in the event of a plan invalidation call. This would trigger a new plan selection cycle from scratch. This approach is easily justified because statistics form the basis for calculating plan costs. When they change, it's comparable to switching to a different coordinate system, making it necessary to recalculate all costs.

The second option is somewhat more controversial: we could introduce a new 'referenced' mode for the generic plan creation process. This mode would use current constants as reference values for the planner. While it may not offer any fundamental advantages, it would provide users with a familiar tool for influencing the query plan, especially for those migrating from SQL Server.

As usual, it makes sense to implement an in-core 'plan switching hook' to leverage the plan switching method within an extension.

If we extend our coding options into the enterprise domain, we can explore more sophisticated plan-switching techniques. For instance, we could track statistics on the planning and execution time for each plan, compare their relative weight with cost values, and make decisions about replanning or even forcing a specific type of plan. An even better alternative could be to use a more stable parameter, such as the number of pages read.

To be more objective, you can check the project, which includes a draft for an automated system to manage plan types, as well as another branch outlining a draft for switching between forced modes.

Have you ever faced issues when using generic plans? Does it make sense to develop a comprehensive system for switching plans, or is it enough to implement an extension that enables each specific prepared statement to monitor its state and update it manually using SQL tools like pg_stat_statements?

References

Hackers' mailing lists threads:

1. Avoiding bad prepared-statement plans. , 2010-02

2. Restructuring plancache.c API , 2010-11

3. One-Shot Plans , 2011-06

4. Transient plans versus the SPI API , 2011-08

5. why do we need two snapshots per query? , 2011-11

6. dynamic SQL - possible performance regression in 9.2 , 2012-12

7. PoC plpgsql - possibility to force custom or generic plan , 2017-01

8. The logic behind comparing generic vs. custom plan costs , 2025-03

9. inefficient/wrong plan cache mode selection for queries with partitioned tables (postgresql 17) ,2025-05

Main commits:

• b9527e9 - first attempt to the feature's design, 2007-03

• e6faf91 custom plans introduction, 2011-09

• 94afbd5 - one-shot entries, 2013-01

• 2aac339 - more sophisticated planning cost model, 2013-09

• f7cb284 - plan_cache_mode setting, 2018-07

THE END.
June 29, 2025. Madrid, Spain.

Occasionally, you may come across queries featuring complex filters similar to the following:

SELECT * FROM table
WHERE
  date > min_date AND
  date < now() - interval '1 day' AND
  value IN Subplan AND
  id = 42';

And in practice, it happens that a simple rearrangement of the order of conditions in such an expression allows for speeding up (sometimes quite notably) the query execution time. Why? Each individual operation costs little. However, if it is performed repeatedly on each of millions of the table's rows, then the price of the operation becomes palpable. Especially if other problems, like the table blocks getting into shared buffers, are successfully solved.

This effect is particularly evident on wide tables that contain many variable-length columns. For instance, I often encounter slow IndexScans that become slow when the field used for additional filtering is located somewhere around the 20th (!) position in the table, containing many variable-width columns. Accessing this field requires calculating its offset from the beginning of the row, which takes up processor time and slows down the execution.

The PostgreSQL community has already addressed this issue, as observed in the code. In 2002, commit 3779f7f, which was added by T. Lane, reorganised the clauses by positioning all clauses containing subplans at the end of the clause list (see order_qual_clauses). This change was logical because the cost of evaluating a subplan can depend on the parameters passed to it, introducing an additional source of error.

In 2007, this approach evolved with the commit 5a7471c, which established that the sorting of clauses would be performed exclusively in ascending order based on the cost parameter. This logic has remained in place to the present day, except for a minor modification in commit 215b43c, which required controlling the order of expression evaluation in each query plan node due to changes in the Row-Level Security (RLS) code.

Now, let’s take a look at what we have in the upstream as of today:

CREATE TABLE test (
  x integer, y numeric,
  w timestamp DEFAULT CURRENT_TIMESTAMP, z integer);
INSERT INTO test (x,y)
  SELECT gs,gs FROM generate_series(1,1E3) AS gs;
VACUUM ANALYZE test;

EXPLAIN (COSTS ON)
SELECT * FROM test
WHERE
  z > 0 AND
  w > now() AND
  x < (SELECT avg(y)
    FROM generate_series(1,1E2) y WHERE y%2 = x%3) AND
  x NOT IN (SELECT avg(y)
    FROM generate_series(1,1E2) y OFFSET 0) AND
  w IS NOT NULL AND
  x = 42;

Looking into the filter of this SELECT, we see the following sequence of conditions:

Filter: ((w IS NOT NULL) AND (z > 0) AND
         (x = 42) AND (w > now()) AND
         ((x)::numeric = (InitPlan 2).col1) AND
         ((x)::numeric < (SubPlan 1)))

During the execution of the query, they will be calculated in strict sequence from left to right. The operator costs are as follows for reference:

• "z > 0" - 0.0025

• "w > now()" - 0.005

• "x < SubPlan 1" - 2.0225

• "x NOT IN SubPlan 2" - 0.005

• “w IS NOT NULL" - 0.0

• “x = 42“ - 0.0025

This order appears quite logical. However, you may be wondering what can be improved here.

There are at least two straightforward opportunities for enhancement. First, you can assign a small cost to the ordinal position of each column involved in the expression. In simple terms, the further a column is to the right in a table row, the more expensive it is to evaluate. The cost should not be excessively high; it merely needs to signal to the optimiser that the expression x=42 is cheaper to evaluate than z>0, assuming all other factors are equal.

You may argue it is related to the current Postgres row-based storage. It is true, but we use this type of storage more frequently, isn't it? Moreover, it would make sense for storage to provide its own cost model.

The second standard pattern relates to pairs of expressions with approximately the exact cost. For instance, consider x=42 and z<50. Clearly, the second expression is less selective and should be placed in the second position. Since the expression x=42 will be true in fewer cases, there will be less need to evaluate subsequent conditions further down the list.

Now, let's assess the potential impact of these optimisations. Is it worth the effort? To illustrate, we can create a table where a pair of columns has the same selectivity but is positioned far apart, while another pair is located next to each other but has different selectivity.

CREATE TEMP TABLE test_2 (x1 numeric, x2 numeric,
  x3 numeric, x4 numeric);
INSERT INTO test_2 (x1,x2,x3,x4)
  SELECT x,(x::integer)%2,(x::integer)%100,x FROM
    (SELECT random()*1E7 FROM generate_series(1,1E7) AS x) AS q(x);
ANALYZE;

Let's examine the performance impact of searching for a value in a relatively "wide" row. Columns x1 and x4 are identical in every way, except that the position of the value in the column x1 is known in advance. In contrast, the position of the value in the column x4 needs to be calculated for each row.

EXPLAIN (ANALYZE, TIMING OFF, COSTS OFF)
SELECT * FROM test_2 WHERE x1 = 42 AND x4 = 42;
EXPLAIN (ANALYZE, TIMING OFF, COSTS OFF)
SELECT * FROM test_2 WHERE x4 = 42 AND x1 = 42;

/*
 Seq Scan on test_2  (actual rows=0.00 loops=1)
   Filter: ((x1 = '42'::numeric) AND (x4 = '42'::numeric))
   Buffers: local read=94357
  Execution Time: 2372.032 ms

 Seq Scan on test_2  (actual rows=0.00 loops=1)
   Filter: ((x4 = '42'::numeric) AND (x1 = '42'::numeric))
   Buffers: local read=94357
 Execution Time: 2413.633 ms
*/

It turns out that, all other factors being equal, even a relatively short tuple can have an effect of about 2-3%. This impact is quite comparable to the typical benefits gained from using Just-In-Time (JIT) compilation. Now, let's consider the influence of selectivity. The columns x1 and x2 are positioned next to each other. The key difference is that the values in x1 are almost unique, whereas x2 contains mostly duplicated values.

EXPLAIN (ANALYZE, TIMING OFF, COSTS OFF)
SELECT * FROM test_2 WHERE x2 = 1 AND x1 = 42;
EXPLAIN (ANALYZE, TIMING OFF, COSTS OFF)
SELECT * FROM test_2 WHERE x1 = 42 AND x2 = 1;
/*
 Seq Scan on test_2  (actual rows=0.00 loops=1)
   Filter: ((x2 = '1'::numeric) AND (x1 = '42'::numeric))
   Buffers: local read=74596
 Execution Time: 2363.903 ms

 Seq Scan on test_2  (actual rows=0.00 loops=1)
   Filter: ((x1 = '42'::numeric) AND (x2 = '1'::numeric))
   Buffers: local read=74596
 Execution Time: 2034.873 ms
*/

It seems we have achieved a speedup of approximately 10%.

It turns out that if we accept that the effect can accumulate throughout the plan tree, which may contain multiple scanning operators as well as joins, each contributing a particular percentage, then this technique is worthwhile to implement overall, especially considering the minimal overhead in the planning phase.

Let's proceed with the implementation and observe its effects. Creating it as an extension doesn't seem practical, as there is currently no hook that allows for operations during the creation of the plan. As for me, the necessity of introducing a create_plan_hook within the create_plan() routine is becoming increasingly evident: We may let extensions transfer some data from the optimisation stage to the plan, as well as do some additional plan enhancements (which may fit a specific load), like proposed here. However, this topic has yet to be discussed within the PostgreSQL community.

If this feature is implemented as a patch, modifications will be needed in two areas of the code: cost_qual_eval(), where the cost of expressions is evaluated, and order_qual_clauses(), which defines the sorting rules for expressions. As usual, the code can be found on GitHub in the designated branch.

Running the aforementioned examples on this branch will demonstrate that the expressions are constructed more optimally, considering column order and selectivity. Additionally, no significant overhead is observed.

Do you think it makes sense to pursue such micro-optimisations, or should we aim for broader improvements? Have you encountered similar issues? Please share your thoughts in the comments.

THE END
April 19, 2025. Madrid, Spain.

This was a long-awaited feature for me. For an extension that influences the query planning process, providing users with notes on how the extension has affected the plan makes perfect sense. Instead of merely writing to a log file - access to which is often restricted by security policies - this information may be made available through the EXPLAIN command.

The feature introduced many entities that are not easy to figure out: an EXPLAIN option registration routine (RegisterExtensionExplainOption), an explain extension ID, per plan/node hooks, and an option handler.

The pg_overexplain extension, introduced with this feature to demonstrate how it works, seems a little messy and impractical for me, at least in its current state. So, I decided to find out how flexible this new technique is and demonstrate the opportunities opening up to developers with a more meaningful example. I have modified the freely available pg_index_stats extension and added information about the statistics used in the query planning process.

The STAT parameter was added to the list of EXPLAIN options, accepting Boolean ON/OFF values. If it is enabled, information about the statistics used is inserted at the end of the EXPLAIN: the presence of MCV, histogram, and the number of elements in them, as well as the values ​​of stadistinct, stanullfrac, and stawidth.

You might wonder why this is necessary. After all, doesn't the set of statistics directly stem from the list of expressions in the query? Isn't it possible to identify which statistics were utilised by examining the cost-model code for a particular type of expression?

While it is indeed possible, this approach is not always sufficient. We understand the algorithms, but we typically do not have access to the underlying data. As a result, we cannot accurately determine which specific statistics are present in pg_statistic for a given column, nor can we know what information was available to the backend at the time of estimation.

Let's look at the example below:

CREATE TABLE sc_a(x integer, y text);
INSERT INTO sc_a(x,y) (
  SELECT gs, 'abc' || gs%10 FROM generate_series(1,100) AS gs);
VACUUM ANALYSE sc_a;
LOAD 'pg_index_stats';

EXPLAIN (COSTS OFF, STAT ON)
SELECT * FROM sc_a s1 JOIN sc_a s2 ON true
WHERE s1.x=1 AND s2.y LIKE 'a';

Explain, boosted by the pg_index_stats extension, looks like the following:

 Nested Loop
   ->  Seq Scan on sc_a s1
         Filter: (x = 1)
   ->  Seq Scan on sc_a s2
         Filter: (y ~~ 'a'::text)
 Statistics:
   "s2.y: 1 times, stats: { MCV: 10 values, Correlation,
          ndistinct: 10.0000, nullfrac: 0.0000, width: 5 }
   "s1.x: 1 times, stats: { Histogram: 0 values, Correlation,
          ndistinct: -1.0000, nullfrac: 0.0000, width: 4 }

Here, you can see that the statistics for the s1.x and s2.y columns were used. We can't detect which statistic type was actually used and how often - it is buried too deeply in the core - but we may still detect some issues:

At first, we have only ten MCV values ​​for y, and there are no MCV statistics for the s1.x at all; the histogram seems to be there, but it is of zero length. No nulls in either column are expected.

Thus, we have some helpful information that can suggest the optimiser's plan selection logic. Considering that a client who cannot provide data can very rarely give a dump of the pg_statistic table, such relatively harmless information can be a helpful aid and reveal possible causes of problems with the query plan selection. As a bare minimum benefit, users often forget to increase the statistic_target parameter (sample size) on massive tables, and this information provides a quick insight into that issue.

The extension utilises the get_relation_stats_hook to track the statistics used. It would also be useful to know whether extended statistics are used in planning, but they are too deep in the core yet, and the current set of hooks will not help here.

Finally, I would like to know what applications you see for expanding the EXPLAIN output. Regarding the example above, how harmless is even such limited information really?

THE END

April 12, 2025. Nikola Tesla Airport, Serbia.

What is extended statistics?

The extended statistics tool allows you to tell Postgres that additional statistics should be collected for a particular set of table columns. Why is this necessary? - I will try to quickly explain using the example of an open power plant database. For example, the fuel type (primary_fuel) used by a power plant is implicitly associated with the country's name. Therefore, when executing a simple query:

SELECT count(*) FROM power_plants
WHERE country = '<XXX>' AND primary_fuel = 'Solar';

we see that this number is zero for Norway and 243 for Spain. This is apparent to us since it is defined by latitude, but the DBMS does not know this, and at the query planning stage, it incorrectly estimates the sample (row number): 93 for Norway and 253 for Spain. If the query turns out to be a little more complex and the estimated data are the input for a JOIN operator, this can lead to unfortunate consequences. The extended statistic calculates the joint distribution of values ​​in columns and allows us to detect such dependencies.

In fact, there are worse situations in ORMs. In the power plant database example, this could be the joint use of conditions on the country and country_long fields. After reading their description, anyone understands that there is a direct correlation between these fields, and when the ORM groups by both of these fields, we get a significant error:

EXPLAIN (ANALYZE, COSTS ON, TIMING OFF, BUFFERS OFF, SUMMARY OFF)
SELECT country, country_long FROM power_plants
GROUP BY country, country_long;

 HashAggregate
  (rows=3494 width=16) (actual rows=167 loops=1)
   Group Key: country, country_long
   ->  Seq Scan on power_plants
        (rows=34936 width=16) (actual rows=34936 loops=1)

A human would never write such a query, but we live in the era of AI, and automatically generated queries are not uncommon. We will have to deal with this somehow.

And what about extended statistics? It allows us to define three types of statistics on a combination of columns (or/and expressions): Most Common Values ​​(MCV), distinct and dependencies. In the case of scanning filters, MCV works best: if the combination of values ​​that a query selects from the table often appears in this table, then the optimiser will get an accurate estimate. If we are looking for a rare combination (as in the case of solar power plants in Norway), having a rough estimate of the sample ntupes/ndistinct, we can refine it by throwing out everything that got into MCV.

In the case of the need to estimate the number of groups (operators GROUP BY, DISTINCT, IncrementalSort, Memoize, Hash Join), the optimiser's decision is very well supported by the ndistinct value per column combination.

Now, to see the impact of extended statistics on the optimiser's row estimation from the table, let's apply extended statistics to our case by running the commands:

CREATE STATISTICS ON country,primary_fuel
 FROM power_plants;
ANALYZE;

You may find that the queries above estimate row numbers much more accurately when selecting and grouping by these two fields. For instance, Norway is estimated to have one power plant, while Spain has 253. Just to be sure, you can verify this result using filters such as country = 'RUS' or country = 'AUT'. Although the table is not very large, the tool seems effective.

However, I rarely see extended statistics being used in practice. One possible reason for this may be the concern that running the ANALYZE command will take a significant amount of time. Yet, I believe the main issue lies in the complexity of diagnostics - specifically, knowing when and where to create these statistics.

Looking for a suitable statistics definition

Is there an empirical rule of thumb for determining where and what statistics to create? I have forged two such rules for myself:

No. 1: By Index Definition. If a DBA takes a risk by creating an index on a specific set of columns, they likely expect the DBMS to receive queries that filter on these columns frequently. Additionally, the execution time of these queries is probably critical, which serves as another reason for improving the quality of query plans. However, there isn't always a significant estimation error for filters on multiple columns, which is a drawback of this empirical approach – statistics may be generated unnecessarily. It's also possible that a point sample of data from the table is what's expected, which may diminish the impact of misestimating on a composite filter – does it really matter whether 1 or 5 rows are returned?

Due to these shortcomings, I developed Method No. 2 using actual query filter templates. In this method, the first step is to identify candidate queries based on two factors: the query's contribution to the database load (which can be measured using the pages-read criterion) and the presence of composite filter conditions in table scans. It would also be beneficial to consider only those instances where the actual cardinality of the table scan operator significantly deviates from the planned value.

This approach is more selective in choosing potential candidates for generating statistics, allowing for a significant reduction in the statistics collected. However, it raises some important questions:

1. When it comes to creating statistics, approach No. 1 provides a clear moment for generating them - at the time of the index creation. But what about approach No. 2? In this case, you must either rely on a timer to generate statistics collecting queries in the interim or manually trigger the command. The absence of a complex query that calculates bonuses at the end of the month (for the previous 29 days) does not mean that we shouldn’t execute it within a reasonable timeframe on the thirtieth day. While such a query may contribute only a tiny amount to the overall load, the accountant may not appreciate waiting several hours for the results!

2. How to Clean Up a Set of Statistics. In the previous approach, we deleted the statistics along with the index. However, this situation is less straightforward now. For instance, if a problematic query suddenly stops occurring - perhaps because the sales season for a popular product has ended - it doesn't mean it won't return in a year. This uncertainty could create potential instability in the DBMS optimiser's operation.

Additionally, it's unclear how much the actual and planned row numbers should differ to be considered significant. Should this difference be two times, ten times, or even a hundred times?

With this in mind, I decided to first write code for the easy-to-implement approach No. 1. At the same time, for approach No. 2, I just plan to develop a recommender tool that, based on data of the pg_stat_statements extension and an analysis of the execution plans of queries, will suggest candidates for creating new statistics.

Extension Description

The concept behind this extension is straightforward (see the repository for details). First, we need a hook to collect the identifiers of objects created in the database, and I have chosen the object_access_hook for this purpose. Next, we need to determine an appropriate time to filter the list of objects, selecting only those that belong to relevant composite indexes. We can efficiently add a new statistics definition to the database using the ProcessUtility_hook, executing our code after a utility command is completed.

Extended statistics, which include distinct and dependencies types, are calculated for all possible combinations of columns. This leads to a rapid increase in computational complexity. For instance, with three columns, the number of distinct statistics is 4, and the number of dependencies is 9. However, these numbers rise dramatically with eight columns to 247 distinct statistics and 1016 dependencies. It's clear now why the PostgreSQL core strictly limits the number of statistical elements to 8.

To prevent excessive load on the database, I introduced a parameter that limits the number of index elements included in the statistics definition (the columns_limit parameter) and another parameter that determines which types of statistics to include in this definition (the stattypes parameter). When these automatic statistics are created, an extra dependency is established on the index serving as the template. Consequently, the associated statistics are removed when the index is deleted.

An open question remains: Is it necessary to create a dependency from the extension to delete all created statistics when DROP EXTENSION is executed? The answer is unclear because the extension may also function as a simple module without requiring a CREATE EXTENSION call, thus potentially impacting all databases within the cluster simultaneously.

To distinguish between automatically generated statistics and those created by users, a comment object that includes the module's name and the statistics name is created. Additionally, we have introduced the functions pg_index_stats_remove and pg_index_stats_rebuild into the extension interface. These functions allow you to delete all statistics and regenerate them, which can be helpful if the data schema was established prior to loading the module or if the database parameters have changed.

A separate issue to address is the reduction of redundant statistics. Given that a database can have many indexes, a procedure has been developed to identify duplicates, aiming to decrease the computational costs of the ANALYZE command (see the pg_index_stats.compactify parameter).

For example, if an index is already defined as t(x1, x2), creating another index as t(x2, x1) would not require the creation of new statistics. A more complex scenario arises when an index t(x2, x1) is created in the presence of another index t(x1, x2, x3). In this case, the most common value (MCV) statistics must be created, as they would not be redundant but the ndistinct, and the dependencies can be disregarded.

Benchmarking

As usual, theory should be validated through practice, and code should be tested on meaningful data. I didn't have access to a ready-made, loaded PostgreSQL instance in either a test or production environment, so I found a stale dump of a database for testing purposes. This particular dump was noteworthy because it contained a large number of tables -about 10,000 - along with roughly three times as many indexes.

Additionally, composite indexes were heavily employed, with around 20,000 indexes containing more than one column. Notably, more than 1,000 of these indexes cover five or more columns. So, this database provides a suitable case for research, although it is unfortunate that no payload is available. The ANALYZE command on this database took 22 seconds to execute. However, when I installed the extension and used the default limit of five columns, the ANALYZE time increased to 55 seconds.

The table with raw data below illustrates the ANALYZE time (in seconds) based on the limit on the number of columns and the types of statistics collected.

It's clear that storing all possible combinations of columns significantly impacts analysis time, mainly when dependencies are involved. Therefore, we can limit our analysis to 3-5 columns in the statistics or consider adopting approach No. 2. I now understand why SQL Server created a separate worker for updating such statistics: this process can be pretty costly. What about reducing redundancy? Let's conduct another experiment:

SET pg_index_stats.columns_limit = 5;
SET pg_index_stats.stattypes = 'mcv, ndistinct, dependencies';
SET pg_index_stats.compactify = 'off';
SELECT pg_index_stats_rebuild();
ANALYZE;

pg_index_stats.compactify = 'on';
SELECT pg_index_stats_rebuild();
ANALYZE;

The following two queries are sufficient to check the amount of statistical data generated by the pg_index_stats extension:

-- Total number of stat items
SELECT sum(nelems) FROM (
  SELECT array_length(stxkind,1) AS nelems
  FROM pg_statistic_ext);

-- Total number of stat items grouped by stat type
SELECT elem, count(elem) FROM (
 SELECT unnest(stxkind) elem FROM pg_statistic_ext
)
GROUP BY elem;

The first query shows the total number of extended statistics items in the database, and the second one - a breakdown by type. So, let's see what happens with and without compactifying:

The overall impact is modest—approximately a 15% improvement in processing time and slightly more in the set of statistics. However, it does provide some protection against corner cases. Interestingly, the compactifying reduced the number of MCV statistics, suggesting that a significant number of indexes differ only in the order of their columns. Additionally, expression statistics, which we haven't discussed before, are generated automatically by the PostgreSQL core if the definition of extended statistics includes an expression. Although this may not pose a significant issue, it would be beneficial to have the ability to regulate this behaviour.

It's also worth comparing the analysis time to an alternative statistics collector called joinsel, which exists in the enterprise Postgres fork, provided by Postgres Professional LLC. While it isn't a direct competitor to extended statistics, it works differently. Based on the index definition, it creates a new composite type within the database, which is then used to generate regular statistics stored in pg_statistic. The advantages of joinsel include MCV and a histogram, which allows for evaluating range filters while leveraging standard PostgreSQL clause estimation techniques. However, it does have some drawbacks, such as a lack of dependency statistics and only one ndistinct value for the entire composite type (a limitation that can be addressed).

Now, let's look at how quickly the ANALYZE command is executed with joinsel.

SET enable_compound_index_stats = 'on';
SELECT pg_index_stats_remove();
\timing on
ANALYZE;
Time: 41248.977 ms (00:41.249)

ANALYZE time has increased as expected compared to regular Postgres statistics, but only by two, which is a reasonable compromise. The main advantage here is that you don't have to worry about the number of columns in the index - the complexity will increase linearly.

Coclusion

The general conclusion regarding Approach No. 1 is that it can be viable, provided we exercise caution and carefully manage the limits.

Additionally, we should enhance the extended statistics in the core. It would be nice to have the possibility of a more significant impact on this tool, allowing us to reduce or expand the volume of generated statistical data.

As for the helper and Approach No. 2, I have decided to postpone it for now. If anyone is enthusiastic and has plenty of free time and patience, feel free to reach out. I would be happy to provide guidance!

THE END.
March 9, 2025, Madrid, Spain.

My interest in this topic began while wrapping up my previous article when I noticed something unusual: the results of executing the same Join Order Benchmark (JOB) query across a series of consecutive runs could differ by several times and even orders of magnitude - both in the value of the execution-time and in pages-read.

This was puzzling, as all variables remained constant - the test script, laptop, settings, and even the weather outside were the same. This prompted me to investigate the cause of these discrepancies… .

In my primary activity, which is highly connected to query plan optimisation, I frequently employ JOB to assess the impact of my features on the planner. At a minimum, this practice enables me to identify shortcomings and ensure that there hasn't been any degradation in the quality of the query plans produced by the optimiser. Therefore, benchmark stability is crucial, making the time spent analysing the issue worthwhile. After briefly examining the benchmark methodology, I identified the source of the instability: the ANALYZE command.

In PostgreSQL, statistics are computed using basic techniques like Random Sampling with Reservoir, calculating the number of distinct values (ndistinct), and employing HyperLogLog for streaming statistics - for instance, to compute distinct values in batches during aggregation or to decide whether to use abbreviated keys for optimisation. Given that the nature of statistics calculation is stochastic, fluctuations are expected. However, the test instability raises the following questions: How significant are these variations? How can they be minimised? And what impact do they have on query plans? Most importantly, how can we accurately compare benchmark results when such substantial deviations are present, even in the baseline case?

Is it possible to achieve query plan stability?

Well, I've thought: the tables are massive, and there are a lot of rows inside - let's just increase the value of the default_statistics_target parameter, and that will solve the problem, right? I was gradually increasing the sample size for statistics from 100 to 10000, re-running all benchmark queries and recording how the pages-read criterion behaves:

Even with the highest level of statistical detalisation, query plans still change after re-executing the ANALYZE command. While increasing the sample size from 100 to 10,000 may improve something, this does not fundamentally alter the situation. Such significant instabilities call into question the possibility of independently reproducing benchmarks and conducting comparative results analyses without examining the query plans.

Now, let's dig deeper: what generally fluctuates? Using a simple script, I conducted an experiment to collect scalar statistics (stanullfrac, stawidth, stadistinct) by rebuilding the statistics ten times. The script for this operation might look like this:

CREATE TABLE test_res(expnum integer, oid Oid, relname name,
                      attname name, stadistinct real,
                      stanullfrac real, stawidth integer);
DO $$
DECLARE
    i integer;
BEGIN
  TRUNCATE test_res;
  FOR i IN 0..9 LOOP
    INSERT INTO test_res (expnum,oid,relname,attname,stadistinct,stanullfrac,stawidth)
      SELECT i, c.oid, c.relname, a.attname, s.stadistinct, s.stanullfrac, s.stawidth
      FROM pg_statistic s, pg_class c, pg_attribute a
      WHERE c.oid >= 16385 AND c.oid = a.attrelid AND
      s.starelid = c.oid AND a.attnum = s.staattnum;
    ANALYZE;
  END LOOP;
END; $$

Afterwards, I analysed the results with a query like the following:

WITH changed AS (
  SELECT
    relname,attname,
    (abs(max - min) / avg * 100)::integer AS res, avg
  FROM (
    SELECT relname,attname,
      max(stadistinct) AS max, avg(stadistinct) AS min,
      avg(stadistinct) AS avg
    FROM test_res
    WHERE relname <> 'test_res'
    GROUP BY relname,attname
    )
  WHERE max - min > 0 AND abs(max - min) / avg > 0.01
) SELECT relname,attname, res AS "stadistinct dispersion", avg::integer
  FROM changed t
  ORDER BY relname,attname;

Something more mathematically correct could be used here, but for our purposes, such a simple criterion is enough to see the instability of statistics. Using the above scripts, let's see how ndistinct fluctuates for 100, 1000 and 10000 elements of the statistical sample:

Stochastic, right? There's a lot of strange stuff here, but we won't dig too deep, attributing it to the fact that determining the ndistinct value from a small sample does not converge uniformly to the actual value. Yes, the fluctuations subside with the sample size, but 10,000 is already the limit, and the table sizes in this benchmark are not that big - in real life, statistics can remain unstable on much larger tables.

Another observation from the results of this experiment is that statistics on fields with a large number of duplicates suffer the most. In practice, this means that grouping or joining on some harmless "Status" field can cause immense estimation errors inside the optimiser, even if the expression on this field is only part of a long list of expressions in the query.

What exactly is the origin of instability?

However, what exactly causes the query plans to fluctuate in this benchmark? To do this, we need to look at the query plans. One query consistently changes its plan from iteration to iteration - 30c.sql, which makes it convenient to analyse. Comparing the query plans, we can see that HashJoin and the parameterised NestLoop compete with very close estimates (see here and here). Using the "close look" method, I found that the discrepancies in the estimates begin already at the SeqScan stage and then diverge throughout the entire plan:

->  Parallel Seq Scan on public.cast_info
    (cost=0.00..498779.41 rows=518229 width=42)
    Output: id, person_id, movie_id, person_role_id,
            note, nr_order, role_id
    Filter: (cast_info.note = ANY ('{(writer),
          "(head writer)","(written by)",(story),"(story editor)"}'))

->  Parallel Seq Scan on public.cast_info
    (cost=0.00..498779.41 rows=520388 width=42)
    Output: id, person_id, movie_id, person_role_id,
            note, nr_order, role_id
    Filter: (ci.note = ANY ('{(writer),
          "(head writer)","(written by)",(story),"(story editor)"}'))

There is a slight difference in the estimation of the number of rows selected from the table. Let's dig deeper and see why.

It is not easy to compare histograms or MCV, so let's just study our specific query or, more precisely, the problematic scan operator. The estimation of x = ANY (...) occurs by estimating each individual expression x = Ni and then adding up the probabilities. In our case, all five Ni constants are included in the MCV statistics - which means that even the ndistinct value will not be used by Postgres. Thus, the estimation should be as accurate and stable as possible. However, if you dig into the numbers, you can see that after the ANALYZE, the frequency of each of the sample elements changes. For example, for default_statistics_target=10000:

The final estimation changes by about 1%, which is not too much in principle. However, who knows - maybe we didn't hit a horrid probability? In addition, the error accumulates when planning the higher-level operators of the query tree, ultimately changing the plan of this query.

In general, this corresponds to the theory that calculating the number of distinct in a set of values ​​with reasonable accuracy can only be obtained by analysing almost all the table rows. Moreover, the issue with statistics has already been reported to the community [here and here, for example] and an attempt to solve the problem was conducted. However, in any case, this comes up against an expensiveness of the volume of the statistical sample and is not universally applicable - and therefore not applicable in the PostgreSQL core.

However, for analytical tasks, where the size and denormalisation of the scheme impose increased requirements on the quality of statistics, one computationally expensive pass through the table's rows may make sense. In addition, the data is loaded rarely in large batches, and statistical collection is not required often. So maybe we just probe the approach involving the extension mechanism?

An extension for standard statistics

How to design it? Let's recall the article by DeWitt1998 - the author suggested computing statistics attaching to sequential table scans. Postgres has a CustomScan node mechanism that can be inserted into any part of the query plan and implemented with an arbitrary complexity of the operation. Therefore, such an idea is easy to implement. Also, no one prevents you from adding a new function to the UI using an extension that will go through the entire table and calculate at least ndistinct and MCV with maximum accuracy.

Having standard "lightweight" statistics on the number of ndistinct, you can assess how expensive and feasible it will be in terms of computing resources before deciding to launch such an analysis procedure.

Feeding such refined statistics to the optimiser can be implemented employing two hooks: get_relation_stats_hook and get_index_stats_hook. They allow to replace the standard statistics obtained from pg_statistic with an alternative, as long as it corresponds to the internal Postgres tuple format. The second hook is exciting because it can be used to implement not complete statistics but predicate statistics - that is, statistics based on data selected from a table with a particular filter - an analogue of SQL Server's CREATE STATISTICS ... WHERE ... .

How to store statistics? For the optimiser to be able to use them correctly, it is evident that their storage format must correspond to the format of storing standard statistics. Nothing is complicated about this since the extension can create its tables - so why not just create table pg_statistic_extra?

Additional bonuses from creating such an extension may emerge entirely unexpectedly. For example, while writing this post, I realised that it could help solve the dilemma of statistics on a partitioned table: no one likes to spend resources on it since it duplicates the work of calculating statistics for each individual partition. At the same time, it does not change much over time: there are many partitions, and if the data is well spread out, then the statistics for all data change insignificantly (except, perhaps, for the partition key). In addition, the table can be huge, and things like ndistinct in standard statistics can be far from reality. With the help of the extension, you can implement the creation of detailed statistics by event: attach/detach partition, at the time of massive updates, etc., which can allow you to launch a computationally expensive operation more consciously. Also, knowing that the table changes rarely, it will be possible to radically simplify the statistics calculation by using indexes or implementing other sampling algorithms (for example, this one)...

That's actually the whole roadmap. All that's left is to find a passionate student, and you can apply with a project to GSoC ;). What do you think about such an extension? Write in the comments.

THE END.

February 2, 2025, South Pattaya, Thailand.

The execution time measurements they provided differed significantly due to unclear details regarding the experimental setup and instance settings. I had often encountered research reports that were almost impossible to reproduce, and the current case led me to discover how we could compare query plans and optimisation effectiveness using dimensionless criteria.

From a practical point of view, the DBMS that produces a higher TPS is more efficient. However, sometimes, we need to design a system that does not yet exist or make a behaviour forecast for loads that have not yet arrived. In this case, we need a parameter to analyse a query plan or compare a pair of plans qualitatively. This post discusses one such parameter - the number of data pages read.

It hardly needs to be said that the 'performance evaluation' section of research is crucial for applied software developers, as it justifies the time spent reading the preceding text. This section must also ensure the repeatability of results and allow for independent analysis. For instance, a similarity theory has been developed in fields like hydrodynamics and heat engineering that enables researchers to present experimental results in dimensionless quantities, such as the Nusselt, Prandtl, and Reynolds numbers. Researchers can reasonably compare the results obtained by reproducing experiments under slightly different conditions.

I have not yet seen anything like this in ​​database systems. The section devoted to testing usually briefly describes the hardware and software parts and graphs. The main parameter under study is the query execution time or TPS (transactions-per-second).

This approach appears to be the only viable method when comparing different DBMSes and making decisions regarding what system to use in production. However, it's important to note that query execution time is influenced by multiple factors, including server settings, caching algorithms, the choice of query plan, and parallelism...

Let's consider the scenario where we are developing a new query optimisation method and want to compare its performance with a previously published method. We have graphs showing query execution times (see, for example, here or there), along with a brief description of our testing platform. However, we encounter discrepancies between our results and those from published studies due to multiple unknown factors. To address this, we need a measurable parameter that can eliminate the influence of other DBMS subsystems, making our analysis more portable and accessible. I believe that developers working, for example, on a new storage system would also appreciate the opportunity to remove the optimiser's impact from their benchmarks.

When attempting to reproduce the experiments described in articles or to compare my method with the one proposed by authors, I often find that the uncertainty of the commonly accepted measurement of execution time is too high to draw conclusive judgments. This measure primarily reflects the efficiency of the code under specific operating conditions rather than the quality of the discovered query plan. Execution time is a highly variable characteristic; even when running the same test consistently on the same machine and instance, there can be a significant variation in execution times.

For instance, I've conducted ten consecutive runs of all 113 Join Order Benchmark (JOB) tests, and I've observed a typical spread in execution time of up to 50% on my desktop (see the picture below) - even under optimal conditions with all experiment parameters are meticulously controlled. This raises a crucial question: how much deviation might an external researcher encounter if they attempt to repeat the experiment, and how should they analyse the results?

One more concern is how to compare query plans executed with varying numbers of parallel workers. Using multiple workers on a test machine can yield positive results; however, parallelism can sometimes be counterproductive in a production with hundreds of competing backends. Therefore, is it best to seek a more meaningful criterion for evaluation?

In my specific area of query optimisation, execution time often seems like a redundant metric. It may be more beneficial to adopt a more specific characteristic to compare different optimisation approaches or to assess the impact of a new transformation within the PostgreSQL optimiser. Such a metric should include only factors that the optimiser may consider during the planning process.

From the perspective of a DBMS, the primary operations involve data manipulation. Thus, it would be natural to select the number of operations performed on table rows during query execution, taking into account the number of attributes in each row. Minimising this parameter would indicate the efficiency of the chosen query plan. However, collecting such statistics can be challenging. Therefore, we should aim to identify a slightly less precise but more easily obtainable parameter.

For example, DBAs often use the number of pages read as a parameter. In this context, a page refers to a buffer cache page or a table data block stored on disk. It is unnecessary to differentiate between the pages that fit in the RAM buffer and those on disk, as this distinction provides redundant information that pertains more to the page eviction strategy and disk operation than to the optimal plan identified.

For our purposes, it is sufficient to sum these values mechanically. We also need to consider the pages from the temporary disk cache used by sorting, hashing, and other algorithms for placing rows that did not fit in memory. It is important to note that the same page may need to be counted twice. We access a page once during sequential row scanning to read its tuples. However, when rescanning—such as in an inner NestLoop join—we reread the data and must account for each page again. PostgreSQL already has the necessary infrastructure for measuring the number of pages read, provided by the pg_stat_statements extension. My approach is as follows: before executing each benchmark query, I run the command SELECT pg_stat_statements_reset() and then retrieve the statistics using the following query:

SELECT
  shared_blks_hit+shared_blks_read+local_blks_hit+local_blks_read+
  temp_blks_read AS blocks, total_exec_time::integer AS exec_time
FROM pg_stat_statements
WHERE query NOT LIKE '%pg_stat_statements_reset%';

How reliable is this metric? In the same experiment mentioned above, all ten runs of the JOB test demonstrated negligible deviation in the number of pages for each query throughout the iterations:

There is a deviation of only a few pages, and while even a minor discrepancy like this should typically be examined, it seems to be more of an artifact resulting from service operations, such as accessing statistics and interactions among parallel workers. What can we infer from this indicator? Let's conduct a simple experiment. We will use one test query (10a.sql) and sequentially increase the number of workers involved in processing this query. The graph below illustrates how the query execution time and the number of data pages read change as we adjust the number of workers.

It is evident that while the query execution time may vary, the number of read data pages remains relatively constant. The number of pages only changes once when the number of workers increases from 1 to 2, resulting in a doubling of the read pages. An examination of the EXPLAIN output for these two cases reveals the reason behind this change: with 0 and 1 worker, out of six query joins, three were of the Nested Loop type, and three were Hash Joins. However, with two or more workers, the number of Nested Loop joins increases by one while the number of Hash Joins decreases. Thus, by analysing the number of read pages, we were able to identify a change in the query plan that was not apparent when considering execution time alone. Now, let's explore the effect of the AQO (Adaptive Query Optimization) extension of the PostgreSQL optimiser on JOB test queries.

We will execute each test query with AQO ten times in 'learn' mode. In this mode, AQO functions as a planner memory, storing the cardinality of each plan node (as well as the number of groups in the corresponding operators) at the end of execution. This information is then used during the planning stage, allowing the optimiser to reject overly optimistic plans. Given that the PostgreSQL optimiser tends to underestimate join cardinalities, this approach appears quite reasonable. The figure below (shown on a logarithmic scale) illustrates how the number of pages read changed concerning the first iteration, during which the optimiser lacks information about the cardinalities of the query plan nodes, the number of distinct values in the columns, etc. By the tenth iteration, almost all queries either improved this metric or remained unchanged. This suggests that the PostgreSQL optimiser may have quickly identified the best plan in the space of potential options or that our technique did not have the desired effect in these cases.

There are still six degraded queries remaining, and the number of pages read for these queries has increased compared to the first iteration. It's possible that there weren't enough iterations to effectively filter out non-optimal query plans. Therefore, let's increase the number of execution iterations to 30 and observe the results.

The figure above illustrates that the query plans have converged toward an optimal solution. Notably, two queries (26b and 33b) show an increase in the number of pages read compared to the zero iteration. Additionally, the query execution time has improved by 15-20%.

The explanations for these observations are as follows: the number of Nested Loops in the query plan has decreased by one, and the Hash Join, when constructing a hash table, scans the entire table and consequently increases the number of pages read. In contrast, the parallel Hash Join proves to be more time-efficient, leading to better query execution times. This suggests that the number of pages read is not an absolute criterion for determining query optimality. This criterion can help establish a starting point within a single DBMS, allowing for reproduced experiments in different software and hardware environments. It can also aid in comparing various optimisation methods and identifying effects that may be masked by unstable execution times.

Therefore, it may not be advisable to disregard execution time when publishing benchmark results. However, should the number of pages read to be included as well? Ultimately, by providing a graph showing the changes in the number of pages read during query execution, along with a test run script (refer to the above) and a link to the raw data, one can independently reproduce the experiment, calibrate it against the published data, conduct additional studies, or compare it with other methods under similar conditions. Isn’t that convenient?

That's it for today. The primary goal of this post is to highlight the problem of reproducibility of results and to encourage objective analysis of new methods in the field of DBMS. Should we seek additional criteria for evaluating test results? How effective is the criterion of the number of pages read for this purpose? Can this criterion be adapted to compare different yet similar query plans regarding DBMS architecture? Is it possible to normalise this criterion relative to the average number of tuples per page? I welcome any opinions and comments on these questions.

THE END.

January 18th, 2025. Pattaya, Thailand.

Introduced with commit 9eacee2 in 2021, the Memoize node fills the performance gap between HashJoin and parameterised NestLoop: having a couple of big tables, we sometimes need to join only minor row subsets from these tables. In that case, the parameterised NestLoop algorithm does the job much faster than HashJoin. However, the outer size is critical for performance and may cause NestLoop to be rejected just because of massive repetitive scans of inner input.

When predicting multiple duplicates in the outer column that participate as a parameter in the inner side of a join, the optimiser can insert a Memoize node. This node caches the results of the inner query subtree scan for each parameter value and reuses these results if the known value from the outer side reappears later.

This feature is highly beneficial. However, user migration reports indicate that there are still some cases in PostgreSQL where this feature does not apply, leading to significant drops in query execution time. In this post, I will compare the caching methods for intermediate results in PostgreSQL and SQL Server.

Memoisation for SEMI/ANTI JOIN

Let me introduce a couple of tables:

DROP TABLE IF EXISTS t1,t2;
CREATE TABLE t1 (x integer);
INSERT INTO t1 (x)
  SELECT value % 10 FROM generate_series(1,1000) AS value;
CREATE TABLE t2 (x integer, y integer);
INSERT INTO t2 (x,y)
  SELECT value, value%100 FROM generate_series(1,100000) AS value;
CREATE INDEX t2_idx ON t2(x,y);
VACUUM ANALYZE t1,t2;

In Postgres, a simple join of these tables prefers parameterised NestLoop with memoisation:

EXPLAIN (COSTS OFF)
SELECT t1.* FROM t1 JOIN t2 ON (t1.x = t2.x);
/*
 Nested Loop
   ->  Seq Scan on t1
   ->  Memoize
         Cache Key: t1.x
         Cache Mode: logical
         ->  Index Scan using t2_idx on t2
               Index Cond: (x = t1.x)
*/

The smaller table t1 contains many duplicates in the column used for the JOIN, while the bigger one t2 contains almost unique values. It also has an index to extract necessary tuples effectively.

Ok, it works for trivial joins. What about more complex forms, like SEMI JOIN? Look at the query:

EXPLAIN (COSTS OFF)
SELECT * FROM t1 WHERE x IN (SELECT y FROM t2 WHERE t1.x = t2.x);

/*
 Nested Loop Semi Join
   ->  Seq Scan on t1
   ->  Index Only Scan using t2_idx on t2
         Index Cond: ((x = t1.x) AND (y = t1.x))
         Filter: (x = y)
*/

Postgres can pull-up the subquery and transform it into a join. But it doesn't add a Memoize node in that case. To compare, execute this query in SQL Server (Use the OPTION (LOOP JOIN) hint to prevent hash join):

SQL Server performs a similar optimisation and utilises a Spool node in the inner subtree of a join. This approach allows the results of scans on the inner side of the join to be cached. Interestingly, it doesn't have to cache individual tuples; it only needs to keep track of the existence of the NULL/NOT NULL result.

So, why hasn’t Postgres implemented Memoize for JOIN_SEMI? If you examine the code, you will find that this limit was introduced in an initial commit by David Rowley.

if (!extra->inner_unique && (jointype == JOIN_SEMI ||
                             jointype == JOIN_ANTI))
  return NULL;

In the case of a semi-join, the executor only requires the first tuple from the inner subtree to make its decision. This means that the Memoize cache will contain incomplete results, which is (I suppose) a source of concern for developers. However, the MemoizePath struct already has built-in mechanisms for situations where the inner subtree provably produces a single tuple for each scan.

It appears that much of the groundwork is already in place to implement caching for semi-joins. We need to make minor adjustments to the get_memoize_path function, revise the cost model in the cost_memoize_rescan routine, and inform users about the memoisation mode by adding relevant details to the EXPLAIN output. The code that enables memoisation for semi-joins is relatively concise and can be found in the branch of my GitHub project. With this patch applied, you can see something like this:

EXPLAIN (COSTS OFF)
SELECT x FROM t1 WHERE EXISTS (SELECT x FROM t2 WHERE t2.x=t1.x);

/*
 Nested Loop Semi Join
   ->  Seq Scan on t1
   ->  Memoize
         Cache Key: t1.x
         Cache Mode: logical
         Store Mode: singlerow
         ->  Index Only Scan using t2_idx on t2
               Index Cond: (x = t1.x)
*/

The EXPLAIN parameter 'Store Mode' appears only when the Memoize node works in the 'incomplete' mode. The same way it works for ANTI JOIN cases:

EXPLAIN (COSTS OFF)
SELECT x FROM t1 WHERE NOT EXISTS (SELECT x FROM t2 WHERE t2.x=t1.x);

/*
 Nested Loop Anti Join
   ->  Seq Scan on t1
   ->  Memoize
         Cache Key: t1.x
         Cache Mode: logical
         Store Mode: singlerow
         ->  Index Only Scan using t2_idx on t2
               Index Cond: (x = t1.x)
*/

In real-life scenarios, I frequently see on the inner side of SEMI and ANTI joins not trivial table scans but huge subtrees containing join trees, aggregates and sortings. For such queries, avoiding unnecessary rescan calls is crucial. Even more importantly, knowledge of the only single tuple needed from such a subquery may cause the choice of a more optimal fractional path.

Memoise arbitrary query subtree

Here, I want to discover if the optimiser can use Memoize to cache the result of a bushy query tree. Look at the example:

DROP TABLE IF EXISTS t1,t2,t3;
CREATE TABLE t1 (x numeric PRIMARY KEY, payload text);
CREATE TABLE t2 (x numeric, y numeric);
CREATE TABLE t3 (x numeric, payload text);
INSERT INTO t1 (x, payload)
  (SELECT value, 'long line of text'
   FROM generate_series(1,100000) AS value);
INSERT INTO t2 (x,y)
  (SELECT value % 1000, value % 1000
   FROM generate_series(1,100000) AS value);
INSERT INTO t3 (x, payload)
  (SELECT (value%10), 'long line of text'
   FROM generate_series(1,100000) AS value);
CREATE INDEX t2_idx_x ON t2 (x);
CREATE INDEX t2_idx_y ON t2 (y);
VACUUM ANALYZE t1,t2,t3;

-- Disable any extra optimisations:
SET enable_hashjoin = f;
SET enable_mergejoin = f;
SET enable_material = f;

Now, let's discover the query:

EXPLAIN (COSTS OFF)
SELECT * FROM t3 WHERE x IN (
  SELECT y FROM t2 WHERE x IN (
    SELECT x FROM t1)
);

There are three joining tables. In the absence of a hash join, it would be better to use a parameterised scan. Lots of duplicated values inside Table t3 should trigger the use of the memoisation technique:

 Nested Loop Semi Join
   ->  Seq Scan on t3
   ->  Nested Loop
         ->  Index Scan using t2_idx_y on t2
               Index Cond: (y = t3.x)
         ->  Index Only Scan using t1_pkey on t1
               Index Cond: (x = t2.x)

As you can see in the EXPLAIN above, Postgres can’t insert a Memoize node at the top of NestLoop JOIN. As far as I remember, it has not yet been implemented because it is hard to discover the query subtree and find all the lateral references and parameters that are mandatory for the memoisation technique. At the same time, SQL Server is capable of doing it:

A closer examination of this example revealed an interesting aspect. Although Postgres doesn't have the opportunity to insert Memoize over the join, it might still insert Memoize over trivial scans of tables t1 or t2, thereby avoiding repeated scans. However, it didn't do this because it predicted only one rescan operation when planning the JOIN(t1,t2). In this case, using memoisation seems unnecessary.

One million rescan cycles occur due to the upper-level join, JOIN(t3, JOIN(t1,t2)), but the bottom-up optimiser lacks the insight to identify this valuable data at the level of JOIN(t1,t2). You can observe this behaviour in our test example by populating t3.x with unique data. Interestingly, SQL Server also uses a bottom-up planning strategy and fails to recognise this situation to insert an appropriate Spool node.

Postgres planning extensibility allows passing through the query plan and doing additional work. Should we consider adding a top-down planning cycle after constructing the query plan?

Beyond the NestLoop memoisation

In previous sections, we discussed how memoisation could be enhanced by extending it to other join types and applying it to arbitrary query subtrees. But can we go further? Let me dream for a little bit...

Memoisation is a technique used for caching parameters along with their corresponding results. In many real-world scenarios, I often encounter complex situations where a heavy subplan is evaluated within an expression or a CASE statement for every incoming tuple due to references to upper-query objects.

What if the optimiser could insert a Memoize node at the top of the subplan whenever an external value parameterises it? To illustrate this idea, let me provide an example:

-- Case 1:
EXPLAIN (COSTS OFF)
SELECT oid,relname FROM pg_class c1
WHERE oid = 
  CASE WHEN (c1.oid)::integer%2=0
    THEN (SELECT oid FROM pg_class c2 WHERE c2.relname = c1.relname)
    ELSE
      (SELECT oid FROM pg_class c3 WHERE c3.relname = c1.relname)
  END;

-- Case 2:
EXPLAIN (VERBOSE, COSTS OFF)
SELECT 
  CASE WHEN (c1.oid)::integer%2=0
    THEN (SELECT oid || ' - TRUE' FROM pg_class c2 WHERE c2.relname = c1.relname)
    ELSE
      (SELECT oid || ' - FALSE' FROM pg_class c3 WHERE c3.relname = c1.relname)
  END
FROM pg_class c1;

-- Case 3:
EXPLAIN (COSTS OFF)
SELECT oid FROM pg_class c1
WHERE EXISTS (
  SELECT true FROM pg_class c2 WHERE c1.relname=c2.relname OFFSET 0);

We can't flatten subplan nodes in these examples and must evaluate them repeatedly. In my mind, the optimiser should have a chance to build a plan that looks like the below:

   SubPlan 1
     Memoize
       Cache Key: ((c1.oid)::integer % 2)
       Cache Mode: logical
     ->  Index Scan using pg_class_relname_nsp_index on pg_class c2
           Index Cond: (relname = c1.relname)

In this case, the Memoize node will re-evaluate the underlying subplan only when a new combination of parameters is provided from the upper query. While we cannot address all the issues that arise with the subplan bubbling up in an expression, we can help mitigate performance cliffs caused by such constructs.

Do you think it makes sense?

THE END.

January 2, 2025, Pattaya, Thailand.

In the presence of a LIMIT statement in the SQL query, unlike the case of plain tables, the optimiser immediately faces many questions: How many rows may be extracted from each partition? Will only a single partition be used? If so, which one will be this single one? - it is not apparent in the circumstances of potential execution-time pruning ... .

What if we scan partitions by index, and the result is obtained by merging? In that case, it is entirely unclear how to estimate the number of rows that should be extracted from the partition and, therefore, which type of partition scan operator to apply. And what if using partitionwise join, we have an intricate subtree under the Append - knowledge of the limits, in this case, should be crucial - for example, when choosing the JOIN type, isn't it?

Interim-cost query plans

Such a pack of questions about planning partitions led to a compromise solution in choosing a query plan for Append's subpaths: for picking the optimal fractional path, two plan options are considered: the minimum total cost and the minimum startup cost paths. Roughly speaking, the plan will be optimal if we have LIMIT 1 or some considerable LIMIT value in the query. But what about intermediate options? Let's look at specific examples (thanks to Alexander Pyhalov).

DROP TABLE IF EXISTS parted,plain CASCADE;
CREATE TEMP TABLE parted (x integer, y integer, payload text)
PARTITION BY HASH (payload);
CREATE TEMP TABLE parted_p1 PARTITION OF parted
  FOR VALUES WITH (MODULUS 2, REMAINDER 0);
CREATE TEMP TABLE parted_p2 PARTITION OF parted
  FOR VALUES WITH (MODULUS 2, REMAINDER 1);
INSERT INTO parted (x,y,payload)
  SELECT (random()*600)::integer,
         (random()*600)::integer, md5((gs%500)::text)
  FROM generate_series(1,1E5) AS gs;
CREATE TEMP TABLE plain (x numeric, y numeric, payload text);
INSERT INTO plain (x,y,payload) SELECT x,y,payload FROM parted;
CREATE INDEX ON parted(payload);
CREATE INDEX ON plain(payload);
VACUUM ANALYZE;
VACUUM ANALYZE parted;

In this example we executed VACUUM ANALYZE twice because by-default statistics on the partitioned table cannot be built. It is built on each partition separately. To gather statistic, combining data from all partitions, we must explicitly execute ANALYZE with the name of such table as a parameter. Now, let's see how the selection from the partitioned and regular table works with the same data:

EXPLAIN (COSTS OFF)
SELECT * FROM plain p1 JOIN plain p2 USING (payload) LIMIT 100;
EXPLAIN (COSTS OFF)
SELECT * FROM parted p1 JOIN parted p2 USING (payload) LIMIT 100;

/*
 Limit
   ->  Nested Loop
         ->  Seq Scan on plain p1
         ->  Memoize
               Cache Key: p1.payload
               Cache Mode: logical
               ->  Index Scan using plain_payload_idx on plain p2
                     Index Cond: (payload = p1.payload)

 Limit
   ->  Merge Join
         Merge Cond: (p1.payload = p2.payload)
         ->  Merge Append
               Sort Key: p1.payload
               ->  Index Scan using parted_p1_payload_idx
               ->  Index Scan using parted_p2_payload_idx
         ->  Materialize
               ->  Merge Append
                     Sort Key: p2.payload
                     ->  Index Scan using parted_p1_payload_idx
                     ->  Index Scan using parted_p2_payload_idx
*/

The query plans seem optimal: depending on the limit, only the minimum number of rows will be selected since, with a helpful index on the join attribute, we have already ordered access to the table rows. Now let's prompt the optimiser to build a complex subtree under the append by enabling partitionwise join:

SET enable_partitionwise_join = 'true';
EXPLAIN (COSTS OFF)
SELECT * FROM parted p1 JOIN parted p2 USING (payload) LIMIT 100;
/*
 Limit
   ->  Append
         ->  Nested Loop
               Join Filter: (p1_1.payload = p2_1.payload)
               ->  Seq Scan on parted_p1 p1_1
               ->  Materialize
                     ->  Seq Scan on parted_p1 p2_1
         ->  Nested Loop
               Join Filter: (p1_2.payload = p2_2.payload)
               ->  Seq Scan on parted_p2 p1_2
               ->  Materialize
                     ->  Seq Scan on parted_p2 p2_2
*/

Although everything has stayed the same in the data, an unsuccessful plan has been selected. The reason for such degradation is that when planning an Append, the optimiser chooses the cheapest plan according to the startup_cost criterion. And this is the one that contains NestLoop + SeqScan - in terms of launch speed, in the absence of the necessity to scan tables at all, such a plan slightly wins even over the obvious NestLoop + IndexScan. This is how the current Postgres works, including the dev branch.

However, this problem can be fixed quite simply by adding the appropriate logic to the optimiser code. Together with Nikita Malakhov and Alexander Pyhalov, we have prepared a patch that can be found on the current commitfest to fix this problem. In the thread with its discussion, you can find another gripping remark about the revision of the startup_cost computation logic of the sequential scan operator, the implementation of which can also alleviate the situation with the choice of non-optimal fractional paths for the case with LIMIT 1. Applying this patch, we will already get an acceptable query plan:

 Limit
   ->  Append
         ->  Nested Loop
               ->  Seq Scan on parted_p1 p1_1
               ->  Memoize
                     Cache Key: p1_1.payload
                     Cache Mode: logical
                     ->  Index Scan using parted_p1_payload_idx
                           Index Cond: (payload = p1_1.payload)
         ->  Nested Loop
               ->  Seq Scan on parted_p2 p1_2
               ->  Memoize
                     Cache Key: p1_2.payload
                     Cache Mode: logical
                     ->  Index Scan using parted_p2_payload_idx
                           Index Cond: (payload = p1_2.payload)

Now, let's look at the next problem, which does not have a simple solution yet.

Calculated limit

Consider the following query:

EXPLAIN (COSTS OFF)
SELECT * FROM parted p1 JOIN parted p2 USING (payload,y)
ORDER BY payload,y LIMIT 100;

Executing it with the patch provided above gives you an optimal plan - it uses NestLoop with a parameterized index scan that will touch only the minimum number of table rows needed to produce the result. However, by simply reducing the limit, we get the original bleak picture:

EXPLAIN (COSTS OFF)
SELECT * FROM parted p1 JOIN parted p2 USING (payload,y)
ORDER BY payload,y LIMIT 1;

/*
Limit
   ->  Merge Append
         Sort Key: p1.payload, p1.y
         ->  Merge Join
               Merge Cond: ((p1_1.payload = p2_1.payload) AND
                            (p1_1.y = p2_1.y))
               ->  Sort
                     Sort Key: p1_1.payload, p1_1.y
                     ->  Seq Scan on parted_p1 p1_1
               ->  Sort
                     Sort Key: p2_1.payload, p2_1.y
                     ->  Seq Scan on parted_p1 p2_1
         ->  Merge Join
               Merge Cond: ((p1_2.payload = p2_2.payload) AND
                            (p1_2.y = p2_2.y))
               ->  Sort
                     Sort Key: p1_2.payload, p1_2.y
                     ->  Seq Scan on parted_p2 p1_2
               ->  Sort
                     Sort Key: p2_2.payload, p2_2.y
                     ->  Seq Scan on parted_p2 p2_2
*/

A SeqScan operator again reads all rows from tables, and the query becomes tens of times slower, although we only reduced the LIMIT! At the same time, by disabling SeqScan, you can see a fast plan and incremental sorting again.

The fundamental problem is that the optimiser only knows the final limit on the number of rows in the query/subquery. In this case, at the Append planning stage, the optimiser cannot estimate how many tuples the upper Incremental Sort could request. As a result, only one row or all rows from each partition may be needed, depending on the data distribution in the 'y' column.

Even if we theoretically imagine that we have taught IncrementalSort to calculate the number of groups by the 'payload' column and, based on this, estimate the maximum required number of rows in each partition, we could not improve the plan estimation since the planning of the Append operator has already been completed, the possible options for its execution have already been fixed - after all, we are planning the query from the bottom up!

To sum it up. Partitioned tables do make the task much more difficult for the current version of Postgres, limiting the search space for optimal query plans. Switching to partitions should be thoroughly tested, focusing on cases where some limited selection of tables' tuples is required and there is no noticeable pruning of partitions at the planning stage. Although the direction is actively developing, we can expect improvements soon (especially if users report emerging issues more actively). Still, there are cases where the solution within the existing architecture is not apparent and requires additional R&D.

Do you agree with my conclusions, or did I just write nonsense? Please leave your opinion in the comments.

THE END.

December 9, 2024, Pattaya, Thailand.