Getting your network time run like a Swiss clock

The recent Telstra outage was not DNS. It was time. Good analysis and an excellent LinkedIn post by Vaughan cover what broke: a time source jumped years into the past and kept serving that wrong time with confidence.

Here is the design point. NTP is built to compare sources and throw out odd ones. That only works if you give it something to compare. Telstra does not look like they ran it that way. Either one source owned the network, or several of the same boxes failed the same way at the same time. Same vendor, same firmware, same GPS week wrap: that is not diversity. That is one failure wearing many badges. Good design would have discarded the rogue clock. Shared fate cannot.

I learned this as a network engineer at the University of Geneva. Local time source, messy campus: BNC to fibre, radio, 80+ sites. Time still had to be one shared truth, without one box owning everyone.

I'm Swiss. We obsess over clocks. Precision is not a hobby. It is a way of life.

Why a few minutes matter

A few minutes of skew sounds trivial until you list what depends on "now":

TLS certificates can look expired or not yet valid. -> This hit Telstra: the appliance jumped back about 20 years, so current certificates appeared not yet valid.
JWT access and ID tokens rely on time claims such as exp and nbf. A bad clock can reject valid tokens or accept ones that should have expired.
Time-based MFA codes stop lining up.
DNSSEC signatures can fall outside their validity window.
Logs from two hosts stop telling one story.
Leases, caches, billing, and compliance records become suspect.

An attacker who can move a clock may reopen an old validity window, break authentication, or make incident timelines useless. That is integrity and availability wearing a watch.

Stratum is a map, not a trust score

NTP arranges clocks in layers called strata.

NTP stratum hierarchy from reference hardware down to clients — Stratum counts hops from a reference clock, not trustworthiness.

Stratum 0 is the reference itself: GPS, an atomic clock, or a national radio signal. It is not an NTP server. A stratum 1 server is directly attached to that reference. A server synchronised from it becomes stratum 2, and so on.

Lower is not automatically better. Stratum describes distance from a reference clock, not accuracy, honesty, hardware quality, or independence. Using one broken stratum 1 appliance can be worse than using several well-run stratum 2 or 3 servers.

How NTP chooses a time

For each source, NTP exchanges four timestamps. From those it estimates the round-trip delay and the offset between the two clocks. It keeps several recent samples and favours the cleaner ones.

The clever part happens when several sources are available. NTP's selection process roughly does this:

Reject unusable sources, including unreachable servers and candidates with excessive distance or obvious protocol errors.
Give each candidate a correctness interval around its measured offset. The interval grows with network delay, clock dispersion, and uncertainty back to the reference clock. -> This should have elmiminated Telstra's failing appliance.
Find where most intervals overlap. Candidates outside that intersection are marked as falsetickers and discarded.
Cluster the survivors to remove weaker outliers.
Combine the remaining truechimers, weighting better sources more heavily, to calculate the system offset.

Then the clock-discipline code decides how to apply that offset. Small corrections are normally slewed in gradually. A giant jump should hit a configured limit, raise an alarm, or require intervention rather than casually sending production back to 2006.

But this is not magic consensus: If there is only one source, there is nothing to compare. If two disagree, there is no independent majority. If five identical appliances share the same firmware bug and GPS input, their answers overlap beautifull so NTP won't see a problem if these are the only time sources, or if there are less than five other correct sources.

That appears to be the missing safety net at Telstra. Public reporting has not exposed the full design, so we cannot say whether there was one effective source or several nodes with shared fate. We can say the bad time propagated instead of being rejected. Either design defeats the protocol's best defence.

But what is certain is that their design was not good enough. If they'd use more sources, potentially external sources, they would have been just fine, even despite their lack of care for their existing appliance.

Where time comes from

There are many available time servers.

National and institutional services offer clear ownership and are often nearby:

Australia: NMI / timedata.nmi.gov.au (check access rules)
New Zealand: MSL (pool.msltime.measurement.govt.nz, NZ only)
and in Switzerland: METAS, if you're interested.

The NTP Pool uses DNS to distribute clients across volunteer servers, with regional zones such as au.pool.ntp.org, nz.pool.ntp.org, and ch.pool.ntp.org. A pool-aware client can maintain associations with several returned servers. Useful, but several addresses from one pool are not proof of different operators, networks, reference clocks, or software.

For a business network, run multiple internal NTP servers and point clients at those. Feed the internal servers from a deliberate mix: perhaps a local GNSS reference, a national service, an institutional server, and a public pool. Different vendors and network paths matter too.

NTP has it's security challenges

An attacker can target NTP itself:

spoof or modify unauthenticated replies to move a client's clock
replay an old valid-looking response
delay packets asymmetrically, which biases NTP's offset calculation without changing packet contents
compromise a trusted time server or its reference
abuse exposed NTP servers for reflection and amplification DDoS
use NTP to exfil data (just like it's possible using DNS or ping, for example)

Network Time Security (NTS) adds server authentication, packet integrity, and replay protection. Use it where available. It still cannot make a compromised clock truthful, detect five identical boxes failing together, or fully stop an on-path delay attack. Cryptography proves who said the time. Diverse sources help decide whether that time makes sense.

Baseline

Not a telco? Keep it boring:

multiple internal and external NTP servers
enough independent upstreams to identify an outlier
different operators, paths, hardware, firmware, and reference types
NTS or another supported authentication method where possible
alerts for lost sources, rising dispersion, source changes, and large offsets
a hard policy for implausible clock steps
patched, inventoried time appliances with tested failover

Ten copies of the same clock are not ten sources.

Swiss clocks are not magic. Measure, compare, distrust a single gear.