A major outage at Cloudflare caused widespread disruption across the internet on Tuesday, leaving users unable to access several popular services, including X (formerly Twitter), ChatGPT, Spotify, YouTube, and Uber. The cybersecurity company has since released a detailed explanation of what happened and how it fixed the issue.
In a blog post published late Tuesday, Cloudflare co-founder and CEO Matthew Prince apologized for the outage, calling it the worst the company has faced since 2019. “In the last 6+ years we’ve not had another outage that has caused the majority of core traffic to stop flowing through our network,” Prince said. “On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.”
The Cause of the Outage
The disruption stemmed from an issue within Cloudflare’s Bot Management system, a key cybersecurity service designed to protect websites from DDoS (Distributed Denial of Service) attacks, content scraping, and credential stuffing attacks. The system uses an AI model to score incoming traffic requests, determining whether they are likely to be from a bot or a legitimate user.
The AI generates scores based on a set of features held in a “feature file,” which refreshes every five minutes to stay updated with evolving bot behaviors. The problem occurred when a change was made to the underlying query that generates this feature file. This change caused the file to duplicate data multiple times, making it larger than normal. As a result, the Bot Management system experienced a failure when it tried to process the unusually large feature file.
Cloudflare’s network began to experience significant failures about 15 minutes after the update was implemented, which led to widespread errors when users tried to access websites protected by Cloudflare’s system.
Initial Suspicions and Confirmation
Initially, Cloudflare suspected that the outage might have been caused by a malicious cyber attack, especially since its status page also went down, despite being independent of the company’s infrastructure. However, Prince clarified that this was a coincidence. “The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind,” he explained.
After further investigation, Cloudflare correctly identified the root cause—the corrupted feature file—and stopped its propagation. The company then replaced the faulty file with an earlier, unaffected version, restoring its services.
Recovery and Future Prevention
Cloudflare’s services were largely restored within three hours, with full recovery taking approximately five hours. Prince assured users that the company is already planning measures to prevent similar issues from occurring in the future, including strengthening error-reporting systems to avoid overwhelming its infrastructure.
Despite the disruption, Cloudflare’s quick resolution highlights the company’s resilience in managing such large-scale incidents. Going forward, the company is committed to ensuring the stability of its network and preventing similar outages.
