Why your pioneering Postgres feature should start in a fork

Post-meetup thoughts on Postgres contribution

At a Postgres meetup in Melbourne, I finally got a straight answer to a question that had nagged me for quite a long time: why do people actually choose Postgres? It isn't an idle question for me — I spend my days reading field reports and designing planner features, and keep seeing how much smarter some other databases are. The gap has narrowed drastically over the last ten years, but even now some SQL Server techniques still execute certain queries much faster.

The answer I got was disarming: 'Postgres is just good enough for our purposes.'

Think about what that means. Open source, no vendor lock-in, and you can always find someone who knows the codebase and can reliably fix a problem or scale the load. Good enough beats brilliant. Significant part of database systems market doesn't care about superiority — they want simplicity, maintainability, predictability, and reliability.

That sounds like a trivial observation. It isn't — not if you are the one trying to get a new, pioneering, sometimes provocative feature into Postgres core.

The uncomfortable logic

Postgres holds mission-critical data, so efficiency and scalability are only second- or third-order concerns. And no code is free of bugs: every line you add is a new bug and a maintenance liability the whole community inherits. So the bar is brutal, and it applies in this order — your feature must first prove safety and the absence of regressions, then provide documentation, and only after all that does the gist of the idea finally get to take the stage. A feature survives that gauntlet only if users are already hurting for it: if you can show they genuinely struggle with a problem it solves.

If your instincts were formed between 2000 and 2015, when people boldly shipped massive, breakthrough features, this comes as a revelation. And if you just want an easy commit to prove your database technology, Postgres core is the wrong venue — pick a project where stability matters less, probably something from the OLAP area.

So what do you do with a pioneering idea? Prove it in a fork first.

A fork lets you see your code actually working — under real load, in a real DBMS, with real users hitting real edge cases. That is also, incidentally, a well-defined role for the Postgres enterprise companies I was sceptical about not long ago. Let me show you what that path actually looks like, because I have walked it.

Parallel queries and temporary tables

The brightest example for me is the parallel execution of queries involving temporary tables — something that, for a long time, simply hasn't been possible in core. Temporary tables are heavily used in CRM systems that need to hold uncommitted changes on the server side, and I have frequently watched a parallel plan collapse the moment a temp table appears in it.

It took three attempts to design parallel scan on temp tables.

Attempt one — the Postgres Professional fork. Formally, the table's definition, all necessary temp types, indexes and functions - even its data pages on disk - are visible to any backend, since the catalogue entries reside in the shared system catalogues and the files are stored on shared storage. Only the dirty pages that haven't been flushed yet are private, sitting in the owning backend's local buffers. So the trivial idea was to flush those buffers and then let background workers scan the table safely as part of a provably read-only query. I implemented exactly that: when the planner spotted a temp-table entry in the plan, it forced the executor's Gather node to flush the local buffers belonging to that table before scanning.

It worked — until sophisticated benchmarks in a pre-production test environment revealed that TOAST tables, indexes, and more are also touched, and their buffers need to be flushed. Worse, a temporary table can be referenced from many parts of a query: an expression, a subplan, or a LIMIT clause are just a few trivial examples. It looks obvious in hindsight, but at the time the problem was buried under the complexity and uncertainty of such a massive task. That fundamental flaw broke the approach.

I don’t know exactly what implementation they run in production. I hope that, with all their users and testing facilities, they eventually found a much better approach.

Attempt two — the Tantor Labs fork took a completely different route. First, a series of experiments showed that flushing a temporary buffer is not that expensive — only about 20% more than reading the page back from disk. Since the memory allocated for local buffers is usually much smaller than for shared buffers, it makes sense (and safe) to flush all dirty buffers at once. Next, the core parallel-safety machinery was extended with a new category: 'safe as long as you flush local buffers.' We then used it in the parallel-hazard detection code to pinpoint the exact node where the temp table is used, added the flushing cost to the Gather estimate, and let the optimiser decide on the best strategy. The first Gather node to execute flushes all local buffers on its first run and enables a safe scan. Such a lazy approach adds overhead only when strictly necessary.

Then real life exposed the design flaws yet again. Even with the buffers provably flushed, a read-only query can still change hint bits or prune dead tuples on a page — making it dirty and forcing another flush. So I added a safety net and disabled pruning and hint-bit updates whenever a temporary table is scanned in parallel.

That patch-on-a-patch is the real lesson. Scanning the same file in parallel is still fundamentally unsafe; if we commit such an approach into core, we will keep paying rent in code complexity and stability every time we touch it. It is doable and usable — but the honest conclusion is that the better fix is to stop fighting local buffers altogether and move temporary-table pages into shared buffers — the direction Konstantin Knizhnik explored years ago in his Global temporary tables proposal.

In theory, building your feature as a pure extension sounds ideal. In reality, though, it often works out poorly. Users tend to avoid custom extensions that lack a track record for stability and safety. Cloud providers usually stick to their own internal code, since they pay developers to handle upgrades, error reporting, replication, and other common issues. AI agents help by automating many routine production features, but they still aren't trustworthy enough.

The part I still can't crack

The fork approach works well enough. I've now seen a new idea run under real load, fail in useful ways, and clearly show promising long-term design. That's step one, and it works.

Step two is harder: getting a proven idea back into the community's field of view.

The main challenge is simply capturing attention. The community's pulse lives on the mailing list — that is where people engage to move a feature forward. But with the sheer volume of topics and emails, the long discussion history, and insider terminology, joining those conversations — and getting others genuinely interested in your feature — can feel like a full-time job. It is not really a path for researchers seeking to showcase an algorithm or a data management technique: such proposals often receive no feedback at all. It almost looks like an unspoken strategy for keeping the project's entropy in check — the barrier stays high enough for the approach to survive, but clearing it takes real, sustained effort.

That's not an issue if a committer or company backs your project, or if you can attend the PGConf conferences and pitch your idea at an unconference session, face-to-face with the community. But I'm talking about the typical, fully remote case — where you've never met the other contributors in person and know one another only by their footprint on the Internet.

My current approach is to publish the proposal early on the pgsql-hackers mailing list. From my experience, there is no reason to expect deep discussion or even a reply, but this way it's visible to search engines, AI tools, and the next developer who faces the same problem. Mention it everywhere - in your posts and talks. And of course, continue to improve your patch, reporting again and agin in the thread. The hope is that by keeping it visible, you'll eventually get real feedback, and someone else stuck on the same issue will find your thread and move it forward instead of reinventing it alone.

But I can't prove that part works yet. So here's my real question for you: if you've managed to turn a quiet mailing-list proposal into real community momentum, in a world this spread out and remote where you never meet most of the people you want to reach, how did you do it? I'd really like to know.

THE END.
Melbourne, Australia. July 26, 2026.