Two failures share a root cause that almost nobody has on a dependency list. Clocks disagree, and things expire.
Both produce outages that look like security failures, and occasionally security failures that look like outages. Both are entirely predictable and are usually discovered by a customer.
This post is the specific places time breaks things, and how to make expiry visible. For whoever gets paged when authentication stops working for no apparent reason.
Where clock skew breaks security controls
Anything that validates a time window fails when two machines disagree about the time.
Token validation. A JWT carries exp, nbf and iat. If the issuing service's clock runs ahead of the validating service's, a freshly issued token has a nbf in the validator's future and is rejected as not yet valid. Users see intermittent authentication failures that correlate with which instance they hit.
The convention is a small tolerance, typically a minute or two, applied to these checks. Most libraries support it and many default to zero.
Time-based one-time passwords. TOTP codes are derived from a thirty-second window. A phone and a server more than a window apart produce codes that never match, and the user concludes your MFA is broken. Servers should accept the adjacent windows.
Certificate validation. A client whose clock is wrong rejects a valid certificate as not yet valid or expired. This is the common cause of TLS failures on devices that lost their clock, in containers built with a bad base image time, and in CI runners that drifted.
Signature freshness windows. Webhook receivers and request-signing schemes reject anything outside a few minutes. Skew on either side produces rejections that look like an integration bug.
Audit log ordering. Records from different hosts interleave incorrectly if their clocks differ, and during an incident the sequence of events is what you are trying to establish. Skew of a few seconds is enough to make a timeline wrong in the part that matters.
The fix for all of these is the same: synchronise clocks, monitor the offset, and allow a bounded tolerance in validation. The monitoring is the part people skip, and drift is silent until it crosses a threshold somewhere.
# Is this host actually synchronised, and by how much is it off?
chronyc tracking 2>/dev/null | grep -E 'Leap|System time' \
|| timedatectl show -p NTPSynchronized -p TimeUSec
Alert on offset rather than on the daemon running. A time service that is running and not synchronising is the state that produces the outage.
The expiry side
Everything with an expiry date will reach it, on a date nobody has in a calendar, usually outside working hours.
The list worth enumerating for your own estate:
- TLS certificates, including internal ones. Public certificates are usually automated; the ones that expire are internal, on a load balancer, a message broker, or a mutual-TLS link between services.
- Signing keys and their certificates: code signing, JWT signing keys, SAML certificates. SAML certificate expiry breaks single sign-on for everyone at once, and the renewal requires coordination with the identity provider.
- API credentials and OAuth client secrets, several of which now have maximum lifetimes.
- Domain registrations. Rare and catastrophic.
- Third-party integration tokens, especially the ones created by a person who has since left.
- Subscriptions and licences where the software stops working rather than degrading.
Make expiry visible
The reason these fail is not that renewal is hard. It is that nobody is watching the date.
Inventory everything with an expiry, with the date and an owner. If it is not written down, it is not being watched.
Monitor from outside. For anything reachable over the network, check the certificate the way a client sees it, rather than trusting what a configuration file says is installed:
echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null \
| openssl x509 -noout -enddate
Run that against every endpoint, including internal ones, and alert at thirty days and again at seven. The two-stage alert matters because the first one gets acknowledged and forgotten.
Automate renewal where possible, and then monitor the automation. An automated renewal that silently stopped working is the most common version of this failure, because the alerting was removed once it was automated.
Give every expiring thing an owner. An unowned certificate is one nobody renews, which is the same pattern as unowned assets generally.
The failure modes are asymmetric
Worth knowing which way each one breaks.
Skew usually causes false rejection: valid things are refused. Annoying, visible, and it fails safe.
Expiry usually causes complete unavailability of whatever depended on it, and it fails at a moment determined years earlier.
The dangerous exception is a system that handles skew by disabling validation. A webhook receiver that stops checking timestamps because they were causing rejections has removed its replay protection, and a client that skips certificate date validation to work around a clock problem has removed the expiry check entirely. Those workarounds are applied under pressure and never revisited.
The concession
Perfect clock synchronisation is not achievable, which is why tolerances exist, and a tolerance that is too generous weakens the control it is part of. A signature freshness window of an hour, chosen to stop support tickets, is a replay window of an hour.
The balance worth holding: keep tolerances small, a minute or two rather than ten, and fix the skew rather than widening the window. If you find yourself increasing a tolerance to make failures stop, you are treating the symptom, and the control is quietly getting weaker each time.
The implication
Time is an input to a surprising number of security decisions, and it is the only dependency nobody declares.
Two things make most of it go away: monitor clock offset rather than whether the daemon is running, and keep a list of every expiry date with an owner. Neither is difficult, and both are usually missing.
