The five-minute file that took down a large part of the Internet

On November 18, 2025, users saw X, ChatGPT, Canva and other unrelated services fail together. The easy story was "Cloudflare is down." The more useful story is how a routine database-permissions change became a global traffic failure in 23 minutes, why the symptoms initially looked like an attack, and what your monitoring should have told you before the vendor status page did.

Cloudflare called it its worst outage since 2019. The company said the incident caused the majority of its core traffic to stop flowing, with the main impact lasting from 11:28 to 14:30 UTC and full recovery at 17:06 UTC. It was not a cyberattack. It was a configuration pipeline doing exactly what it had been built to do: distribute a file quickly and globally. The file was wrong. (Cloudflare postmortem)

What broke, in plain terms

Cloudflare's Bot Management system scores requests using a machine-learning feature file. That file is rebuilt every few minutes from a ClickHouse database and pushed across Cloudflare's network. At 11:05 UTC, Cloudflare changed database permissions so users could see metadata for underlying tables as well as distributed tables. A query used to build the Bot Management file did not filter by database name, so it began returning duplicate rows. A file that normally represented about 60 features grew past 200. (Cloudflare postmortem)

The proxy had a hard limit of 200 features. When the oversized file arrived, the Bot Management module panicked. Requests that depended on that module returned HTTP 5xx errors before they reached customer origins. The same core proxy also sat in the path for Workers KV and Access, so the failure spread into authentication and other Cloudflare services. Turnstile failed to load, and many customers could not log into the Cloudflare dashboard because the login page itself used Turnstile. Existing Access sessions, however, continued to work. Email delivery also continued, though some security functions were degraded. (Cloudflare postmortem)

This was not a single clean drop. During the ClickHouse permissions rollout, the file generator ran every five minutes and could hit an updated node or an older one. Good and bad files alternated, so services recovered and failed again. Cloudflare initially suspected a large DDoS attack. A coincidental failure of its separately hosted status page made that theory look more plausible. (Cloudflare postmortem)

That detail matters. Intermittent recovery can make a deterministic deployment failure look like hostile traffic or random infrastructure instability.

The timeline that matters

Cloudflare's first automated test detected the problem at 11:31 UTC, three minutes after customer impact began. Manual investigation started one minute later, and an incident call opened at 11:35. At 13:05, Cloudflare reduced impact by routing Workers KV and Access around the newer proxy path. By 13:37, the team had isolated the Bot Management file as the trigger. At 14:24, it stopped new files from being generated and distributed; at 14:30, a known-good file had been deployed globally and the main impact was over. Downstream recovery and load cleanup continued until 17:06. (Cloudflare postmortem)

Independent measurements line up with that account. ThousandEyes saw a sharp rise in HTTP 500 errors around 11:30 while network paths to Cloudflare showed no corresponding increase in packet loss or latency. It also observed some customers moving DNS away from Cloudflare after Cloudflare publicly acknowledged the incident. (ThousandEyes outage analysis, ThousandEyes incident report)

What your monitoring should have shown in five minutes

The first question in an incident is not "who is down?" It is "where does the request stop?"

For this outage, four views would have narrowed the fault domain quickly:

1. Outside-in checks from several regions. A simultaneous jump in 5xx responses across regions, with the same Cloudflare-branded failure, points away from one application deployment or one cloud region.
2. Origin health beside edge health. If direct origin checks remain healthy while public endpoints fail, your application may be fine and the proxy path is suspect. Keep the direct-origin probe private and authenticated; do not turn it into a public bypass.
3. Network path beside HTTP results. ThousandEyes observed reachable paths without elevated packet loss while application requests returned HTTP 500. That combination says the network can reach the provider, but something inside its request-processing layer is failing. (ThousandEyes outage analysis)
4. Dependency-level error tags. Separate DNS resolution, TLS negotiation, CDN/proxy response, origin response and authentication. "Availability down" is an alarm. "Cloudflare edge returns 500 while origin is healthy" is a diagnosis.

The vendor status page should enrich that evidence, not replace it. Cloudflare's own status page briefly failed even though it was hosted outside Cloudflare, a coincidence that confused the response team. (Cloudflare postmortem) Monitoring only your app and a provider's status page gives you two red lights and no explanation.

Could customers have stayed online?

Some did, by changing DNS to send traffic directly to their origins. That restored reachability but removed Cloudflare's edge caching, bot controls and other proxy services. ThousandEyes observed X become reachable after moving away from Cloudflare, while latency rose sharply for users farther from its origin infrastructure. (ThousandEyes outage analysis)

So "just bypass the CDN" is not a runbook. It is a business decision with security, capacity and performance costs. A usable fallback needs an origin that can absorb uncached traffic, independent protection, certificates that work on the alternate path, low-enough DNS TTLs, and a tested way back. Decide in advance how long you will tolerate an edge failure before accepting degraded service. During this incident, ThousandEyes first observed DNS changes around Cloudflare's 11:48 status acknowledgement, suggesting many teams waited for provider confirmation before switching. (ThousandEyes outage analysis)

For many lean teams, the right answer may be to wait rather than run a second edge. But that should be an explicit choice supported by measured recovery objectives, not a discovery made during an outage.

The recurring pattern

The trigger was ordinary change, not exotic failure. Cloudflare's June 2022 outage also began with an internal network-configuration rollout; a routing-policy change took 19 high-traffic data centres offline and affected 50% of requests. (Cloudflare's 2022 postmortem) The shared lesson is not "never change production." It is that global distribution turns a small mistake into a large blast radius unless validation, staged rollout, kill switches and rollback speed grow with it.

Cloudflare's promised fixes reflect that: treat internally generated configuration like untrusted input, add more global kill switches, prevent diagnostic systems from exhausting resources, and review failure handling across proxy modules. (Cloudflare postmortem)