Postgres LISTEN/NOTIFY Actually Scales | DBOS

Postgres LISTEN/NOTIFY has a bad reputation thanks in part to a popular blog post asserting it does not scale. If that were true, it would be a shame, because LISTEN/NOTIFY is a powerful tool, allowing you to use your Postgres database for low-latency durable notifications, streams, and pub/sub. The accusations aren’t wrong: NOTIFY has unintuitive and undocumented performance characteristics arising from its use of a global lock. But “unintuitive behavior” is not the same as “not scalable.” In this blog post, we’ll show how we optimized LISTEN/NOTIFY-backed streams at scale, achieving 60K writes per second on a single Postgres server with millisecond-scale latency.

Low-Latency Streaming with LISTEN/NOTIFY

The basic design of Postgres-backed streams is simple: create a streams table where each stream chunk (for example, an LLM response token) is a new row, then write to streams by inserting into the table.

‍

The tricky part is reading from the stream because you don’t know when the next chunk will arrive. One solution is polling: have each reader poll the end of the stream for new chunks. However, polling scales poorly. If the polling interval is set too high, latency is too high for interactive use-cases (e.g., online chats). But if the polling interval is set too low, concurrent pollers overwhelm the database.

The better solution is LISTEN/NOTIFY. This allows readers to block waiting for a notification from a writer that a new chunk has been published to the stream. That way, readers don’t waste resources polling, but wake up immediately when a new stream chunk arrives.

In our initial implementation of LISTEN/NOTIFY-based streams, a trigger on the streams table fired a function that sent a notification every time a new stream chunk was written. Readers waited for these notifications and woke up to a new stream chunk.

This implementation was correct and delivered low latency, but at scale its throughput was poor. Even using a large Postgres database, it could not sustain more than 2.9K stream writes per second. Interestingly, it bottlenecked without visibly consuming any Postgres resource (CPU, memory, or IOPS). As you may have guessed, the root cause was the original “LISTEN/NOTIFY is not scalable” issue: a global lock Postgres takes during NOTIFY. But why does Postgres do that, and how can we optimize it without losing the benefits of Postgres notifications?

The LISTEN/NOTIFY Exclusive Lock

To understand the problem, we’ll need to examine how Postgres LISTEN/NOTIFY actually works.

The root cause of the poor performance is that in Postgres, committing a transaction that calls NOTIFY requires taking a global exclusive lock. This lock is taken as the transaction begins to commit, and is not released until the transaction is fully committed and its contents have been flushed to disk with fsync().

This lock is necessary because Postgres guarantees that notifications are sent in transaction commit order. To enforce this, it stores all outgoing notifications in a global internal queue whose order must exactly match the commit order of the transactions sending those notifications. Adding notifications to this queue must be done transactionally as part of the commit. However, Postgres doesn’t assign transactions a commit order until those transactions are done committing, as committing can take a variable amount of time.

This creates an ordering problem: transactions containing notifications must add themselves to the queue in commit order, but commit order isn’t defined until the commit is complete. The solution is the global lock, which serializes commits of transactions containing notifications, so their commit order is defined ahead of time and they can correctly order themselves in the internal notifications queue. 

This exclusive lock explains the poor performance we observed. Because we call NOTIFY from a trigger on the streams table, every stream write includes a call to NOTIFY. In order to commit, each stream write needs to take the global lock and hold it for the entire duration of its commit, including the flush to disk. This means that stream writes need to commit sequentially, precluding Postgres’s usual optimizations like group commit (which commits many transactions together in a single fsync()). As a result, stream writes can complete no faster than Postgres can commit transactions, which leads to this bottleneck. This also explains why we did not see significant consumption of any Postgres resource such as CPU or disk: there wasn’t any, because all transactions were serialized by a global lock.

As an aside, there’s been some online discussion of a Postgres patch related to this issue. This patch (to be released in Postgres 19) does not remove the global lock or fix the bottleneck we observed. Instead, it optimizes the narrower case where there are many notification channels and each listener is waiting only on a specific channel.

Optimizing LISTEN/NOTIFY

To make LISTEN/NOTIFY-backed streams faster, we have to work around this bottleneck. The key observation is that for streams, and for many other applications of LISTEN/NOTIFY, the notifications aren’t themselves a source of truth. Instead, they just ping a reader to check a database table (the real source of truth) for new data. As a result, notifications don’t have to be globally ordered or perfectly durable, so we can optimize NOTIFY by buffering notifications in memory and periodically flushing them in a single batch transaction, significantly reducing contention on the global lock.

Buffering and batching NOTIFYs avoids the bottleneck because the global lock only needs to be taken when the buffer is flushed, not for each individual stream write. This means that individual stream writes can proceed quickly, taking advantage of Postgres optimizations like group commit to obtain high throughput, while the buffer flushes in the background.

Adopting a buffer introduces a new complication, which is that a process crash while notifications are buffered leads to those notifications never being delivered. To solve this issue, we add a fallback to stream readers: in addition to waiting for notifications, they also periodically poll the database to check if the stream was written to without a notification. The frequency of this polling can be low (because it is only a fallback for undelivered notifications), so it does not significantly affect performance.

Benchmarking this optimized solution, we see massively improved performance: in the presence of concurrent readers, we can perform up to 60K stream writes per second (20x more than before) while still obtaining 15-100ms latency. At maximum throughput, Postgres CPU is fully utilized, showing the database is actually saturated instead of bottlenecked on contention.

Learn More

All benchmark code is available on GitHub: github.com/dbos-inc/dbos-postgres-benchmark

If you like building scalable, reliable systems, we’d love to hear from you. At DBOS, our goal is to make Postgres-backed durable execution as simple and performant as possible. Check it out: