Safeguard
Application Security

The Permission You Just Revoked, According to a Replica That Has Not Heard Yet

A user is removed from a project. The write succeeds on the primary immediately. For the next several hundred milliseconds, a read replica still shows them as a member, and if any authorization check reads from it, they can still act.

Priya Raman
Staff Security Engineer
6 min read

An administrator removes a user from a project. The removal writes to your primary database and returns success immediately. For the next several hundred milliseconds, possibly longer under load, a read replica somewhere in your fleet still shows that user as a member, because the write has not propagated yet.

If any authorization check reads from that replica, and many are routed there deliberately for performance, the removed user's next request in that window is evaluated against permission state that no longer exists anywhere except in a replica's slightly outdated copy.

This post is about the gap between when a permission changes and when every reader agrees it changed. For whoever routed reads to replicas without asking which reads those should be.

Why this is not a rare edge case

Read replicas exist specifically because reads vastly outnumber writes in most applications, and routing reads away from the primary is one of the most common and effective scaling techniques available. It is also, by construction, an admission that replicas are sometimes behind, because if they were never behind there would be no reason to distinguish them from the primary at all.

Most of the time, this is invisible and harmless. A slightly stale product listing or a comment count that is off by one for a few hundred milliseconds affects nothing that matters. The failure mode specific to this post is narrower and sharper: the small set of reads that decide what a user is allowed to do, where staleness is not a display glitch but a security decision made on outdated information.

The specific windows where this bites

Revocation. A permission removed, a role downgraded, a user removed from a team or a tenant. The write succeeds, the UI confirms it, and an authorization check reading from a lagging replica continues to grant access for however long the lag lasts. This is the most consequential version, because revocation is precisely the action that matters most to get right immediately.

Session invalidation. A logout, a forced session termination after a password reset, a device removal. If session validity is checked against a replica, a token you believe you just killed can continue authenticating for the lag window, which is exactly the window an attacker who triggered the reset is trying to exploit.

Tenant boundary changes. A user moved between tenants, or removed from one entirely, in a multi-tenant system. A stale read can serve them content from a tenant they were just removed from, which is a cross-tenant data exposure with a root cause that has nothing to do with your authorization logic being wrong and everything to do with which database connection answered the query.

Feature and billing entitlement changes, where a downgrade should immediately restrict access to a paid feature. Less severe than the above, and still a real correctness problem when a cancelled subscription continues to grant access for longer than your billing terms describe.

Why this is easy to miss in testing

Local development runs against a single database instance, so there is no replica lag to observe, ever. Staging environments are frequently small enough that replication is near-instantaneous under the light load a few engineers generate. The problem is a function of production write volume and geographic distribution, which means it is exactly the kind of issue that does not appear until the system is operating at a scale where the consequences of missing it are largest.

What to actually do

Route authorization-relevant reads to the primary, deliberately and explicitly, as a rule rather than a case-by-case decision. The read/write split should distinguish "reads where staleness is acceptable" from "reads that gate an action," and the second category goes to the primary regardless of the performance cost, because the performance cost of an extra query to the primary is bounded and small, while the cost of a stale authorization decision is not.

For revocation and session invalidation specifically, do not rely on read routing alone. Maintain a fast, primary-backed or cache-backed denylist for recently revoked sessions and permissions, checked on every request regardless of which database the rest of the request's data comes from. This decouples the one check that must never be stale from the general read path, which can stay on replicas for everything else.

Measure your actual replication lag, in production, under real load, not the number from a vendor's documentation. Lag is variable and grows under write pressure, exactly the conditions present during an incident, a mass permission change, or a coordinated offboarding, which is precisely when you most need revocation to be immediate.

Treat a synchronous consistency requirement as a deliberate architectural choice for these specific paths, not a performance regression to be optimised away later. A slower revocation check that is always correct is a better trade than a fast one that is occasionally wrong, and the two need to be evaluated against different standards rather than the same latency budget.

Test it directly

Under a load test that generates realistic write volume, revoke a permission and immediately issue a request that depends on it, in a tight loop, and check how many requests succeed after the revocation before they consistently fail. Any non-zero, non-trivial count is your actual exposure window, measured rather than assumed, and it is very likely larger than whatever number would have been guessed from documentation alone.

The concession

Routing every authorization check to the primary database removes a real performance optimisation, and for a system with generous replica capacity specifically to reduce primary load, sending high-volume authorization traffic back to the primary can reintroduce the bottleneck the replicas existed to solve.

The narrower version, a denylist or cache checked on every request regardless of read routing elsewhere, gets most of the safety at a much smaller cost than routing everything to the primary, and it is the version worth building first if the full architectural change is not immediately practical.

The implication

A permission system can be entirely correct in its logic and still grant access it should not, for reasons that have nothing to do with the logic and everything to do with which physical database answered a particular query at a particular moment.

If you route any reads to replicas, and most systems at any meaningful scale do, ask specifically which of those reads gate an action a user should no longer be able to take. That list is usually short, and it is the one that needs a different answer than "whichever replica was fastest to respond."

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.