PIPEFAIL: How a missing shell option slowed Cloudflare down
April 05, 2022 12:57 PM
This post tells the story of how a missing shell option called “pipefail” slowed Cloudflare down....
Cloudflare’s investigation of the January 2022 Okta compromise
March 22, 2022 4:57 PM
Today at 03:30 UTC we learnt of a compromise of Okta. We use Okta internally for employee identity as part of our authentication stack. We have investigated this compromise carefully and do not believe we have been compromised as a result...
Incorrect proxying of 24 hostnames on January 24, 2022
January 26, 2022 6:34 PM
On January 24, 2022, as a result of an internal product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin...
A Byzantine failure in the real world
November 27, 2020 12:00 PM
At Cloudflare, we are always on the lookout for Single Points of Failure. In this post, we explore the role a failure mode known as a Byzantine fault played in a a real-world incident....
August 30th 2020: Analysis of CenturyLink/Level(3) outage
August 30, 2020 8:55 PM
Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet. ...
MORE POSTS
July 12, 2019 3:45 PM
Details of the Cloudflare outage on July 2, 2019
Almost nine years ago, Cloudflare was a tiny company and I was a customer not an employee. Cloudflare had launched a month earlier and one day alerting told me that my little site, jgc.org, didn’t seem to have working DNS any more. ...
- By
July 02, 2019 3:50 PM
Cloudflare outage caused by bad software deploy (updated)
Starting at 1342 UTC today we experienced a global outage across our network that resulted in visitors to Cloudflare-proxied domains being shown 502 errors (“Bad Gateway”). The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Applicatio...
- By
June 26, 2019 10:22 PM
The deep-dive into how Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Monday
On Monday we wrote about a painful Internet wide route leak. We wrote that this should never have happened because Verizon should never have forwarded those routes to the rest of the Internet. Today we will dive into the archived routing data and analyze it....
- By
June 01, 2018 1:13 AM
Today we mitigated 1.1.1.1
Cloudflare is protected from attacks by the Gatebot DDoS mitigation pipeline. Gatebot performs hundreds of mitigations a day, shielding our infrastructure and our customers from L3 and L7 attacks. ...
- By
March 01, 2017 3:27 PM
Quantifying the Impact of "Cloudbleed"
Last Thursday we released details on a bug in Cloudflare's parser impacting our customers. It was an extremely serious bug that caused data flowing through Cloudflare's network to be leaked onto the Internet....
- By
February 23, 2017 11:01 PM
Incident report on memory leak caused by Cloudflare parser bug
Last Friday, Tavis Ormandy from Google’s Project Zero contacted Cloudflare to report a security problem with our edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare....
- By
January 01, 2017 10:40 PM
How and why the leap second affected Cloudflare DNS
At midnight UTC on New Year’s Day, deep inside Cloudflare’s custom RRDNS software, a number went negative when it should always have been, at worst, zero. A little later this negative value caused RRDNS to panic. ...
- By
June 21, 2016 6:03 AM
A Post Mortem on this Morning's Incident
We would like to share more details with our customers and readers on the internet outages that occurred this morning and earlier in the week, and what we are doing to prevent these from happening again....
- By
July 06, 2015 9:30 PM
The Internet is a cooperative system: CNAME to Dyn DNS outage of 6 July 2015
Today, shortly after 21:00 UTC, on our internal operations chat there was a scary message from one of our senior support staff: "getting DNS resolution errors on support.cloudflare.com", at the same time as automated monitoring indicated a problem....
- By
August 27, 2013 6:15 PM
Details Behind Today's Internet Hacks
At 1:19pm, a researcher noticed that the New York Times' website wasn't loading. We know the New York Times tech team, so we emailed to check in. Minutes later, the CTO of the NYT called us back. ...
- By
May 31, 2013 11:10 AM
Today's Network Issue
Today at 16:13 UTC a large amount of traffic began hitting our Los Angeles data center. We have an in-house team that monitors our network 24x7x365 and immediately all their alarms went off....
- By
March 03, 2013 1:47 PM
Today's Outage Post Mortem
This morning at 09:47 UTC CloudFlare effectively dropped off the Internet. The outage affected all of CloudFlare's services including DNS and any services that rely on our web proxy....
- By
September 16, 2012 5:40 PM
Post Mortem: What Yesterday's Network Outage Looked Like
Yesterday, around 16:36 GMT, we had an interruption to our network services. The interruption was caused by a combination of factors. First, we had an upstream bandwidth provider with some network issues that primarily affected our European data centers....
- By
June 04, 2012 10:02 PM
The Four Critical Security Flaws that Resulted in Last Friday's Hack
A core value CloudFlare is that security information should be shared between organizations to make the entire Internet safer. That is how CloudFlare's systems work: if one site is attacked, data about that attack is immediately shared with the rest of the network so other sites ...
- By