Post Mortem

November 04, 2023 6:18 AM

Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. Here are the details...

By 
• Matthew Prince

November 01, 2023 4:39 PM

Multiple Cloudflare services were unavailable for 37 minutes on October 30, 2023, due to the misconfiguration of a deployment tool used by Workers KV....

By 
• Matt Silverlock
• , Kris Evans

October 20, 2023 9:39 PM

On Wednesday, October 18, 2023, we discovered attacks on our system that we were able to trace back to Okta. We have verified that no Cloudflare customer information or systems were impacted by this event because of our rapid response. ...

By 
• Sourov Zaman
• , Lucas Ferreira
• , Kimberly Hall
• , Grant Bourzikas

October 04, 2023 7:40 PM

On 4 October 2023, Cloudflare experienced DNS resolution problems. Some users may have received SERVFAIL DNS responses to valid queries. In this blog, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again...

By 
• Ólafur Guðmundsson

August 02, 2023 1:05 PM

A deep dive into the recent incidents relating to Workers KV, and how we’re going to fix them...

By 
• Matt Silverlock
• , Charles Burnett
• , Rob Sutter
• , Kris Evans

July 11, 2023 8:48 AM

On July 9, 2023, users in the Asia Pacific region experienced connection errors due to origin DNS resolution failures to .com and .net TLD nameservers...

By 
• Christian Elmerot
• , Alex Fattouche
• , Hannes Gerhart

January 25, 2023 3:47 AM

Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products...

By 
• Kenny Johnson
• , Sam Rhea

October 26, 2022 3:02 AM

Today, a change to our Tiered Cache system caused some requests to fail for users with status code 530...

By 
• John Graham-Cumming

August 09, 2022 3:56 PM

Yesterday, August 8, 2022, Twilio shared that they’d been compromised by a targeted phishing attack. Around the same time as Twilio was attacked, we saw an attack with very similar characteristics also targeting Cloudflare’s employees...

By 
• Matthew Prince
• , Daniel Stinson-Diess
• , Sourov Zaman

June 21, 2022 12:38 PM

Today, June 21, 2022, Cloudflare suffered an outage that affected traffic in 19 of our data centers...

By 
• Tom Strickx
• , Jeremy Hartman

April 05, 2022 12:57 PM

This post tells the story of how a missing shell option called “pipefail” slowed Cloudflare down....

By 
• Alex Forster

March 22, 2022 4:57 PM

Today at 03:30 UTC we learnt of a compromise of Okta. We use Okta internally for employee identity as part of our authentication stack. We have investigated this compromise carefully and do not believe we have been compromised as a result...

By 
• John Graham-Cumming
• , Lucas Ferreira
• , Daniel Stinson-Diess

January 26, 2022 6:34 PM

On January 24, 2022, as a result of an internal product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin...

By 
• Jeremy Hartman

November 27, 2020 12:00 PM

At Cloudflare, we are always on the lookout for Single Points of Failure. In this post, we explore the role a failure mode known as a Byzantine fault played in a a real-world incident....

By 
• Tom Lianza
• , Chris Snook

August 30, 2020 8:55 PM

Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet. ...

By 
• Matthew Prince