In the course of debugging, we were able to identify common characteristics of affected sites—CNAME-based users of CloudFlare, rather than complete domain hosted entirely on CloudFlare, which, ironically, included our own support site, support.cloudflare.com. When users point (via CNAME) to a domain instead of providing us with an IP address, our network resolves that name —- and is obviously unable to connect if the DNS provider has issues. (Our status page https://www.cloudflarestatus.com/ is off-network and was unaffected). Then, we were investigating why only certain domains were having issues—was the issue with the upstream DNS? Testing whether their domains were resolvable on the Internet (which they were) added a confounding data point.

Ultimately, the outage was identified as Dyn, another major DNS operator, having issues with their own DNS configuration. (https://www.dynstatus.com/incidents/4sbm48rdsdbq)

The Internet is made up of many networks, operated by companies, organizations, governments, and individuals around the world, all cooperating using a common set of protocols and agreed policies and behaviors. These systems interoperate in a number of ways, sometimes entirely non-obviously. The mutual goal is to provide service to end users, letting them communicate, enjoy new services, and explore together. When one provider has a technical issue, it can cascade throughout the Internet and it isn’t obvious to users exactly which piece is broken.

Fortunately, even when companies are competitors, the spirit of the Internet is to work together for the good of the users. Once we identified this issue, we immediately contacted Dyn and relayed what we knew, and worked with them to resolve the issue for everyone’s benefit. We have temporarily put in workarounds to address this issue on our side, and hope the underlying difficulties will be resolved shortly.

Update: The good folks at Dyn have posted a short explanation of what happened on their nameservers.

Portal uses open-source software and services to take inexpensive, commodity travel routers and turn them into powerful security devices. Since this is pretty far from CloudFlare's core business, it warrants a brief digression into why we support projects like this.

Computer security was for a very long time only of interest to hobbyists, academics, and obscure government agencies. Cryptography was an interesting offshoot of number theory, a foundational but very abstract part of mathematics, and many of the early infrastructure components of the Internet didn't include security at all -- there was an assumption that anyone who could gain access would be responsible and well-intentioned, a consequence of the academic origins; after all, why would they want to break or steal things which were freely available.

Before the "cambrian explosion" of commercial computer security, there was still a lot of great security research -- it was just done by academics and by individuals in the "security community", who were motivated by a desire to understand how things worked, and to make tools because they loved the technology and wanted to solve their own problems. Some of the most interesting and powerful security tools available today trace their origins to rather humble open-source, hobbyist, or academic beginnings -- PGP, Tor, OTR, various forms of electronic cash, and many others. Many of today's most respected people in computer security entered the field during this period, out of personal curiosity or academic interest.

While CloudFlare is an eager participant in the commercial security world (we're the easiest and fastest way to set up TLS for any website, and we provide edge security and performance to millions of sites, including some of the largest sites on the Internet -- both with free service and paid services in various tiers), we are also very aware of the broad and deep foundation of security tools and research on which we're built.

CloudFlare makes extensive use of open source software, such as the Nginx web server, community collections of Web Application Firewall (WAF) rules originally generated by OWASP, and powerful cryptographic algorithms developed in academia and implemented by open source efforts such as the OpenSSL Project.

Where possible, CloudFlare also contributes back to the community in those areas. We contribute bugfixes and new functionality back to open source packages, and we employ developers who in their spare time make additional contributions to open source software. CloudFlare's GitHub Open Source page is a great collection of many of our contributions to open source.

One of our biggest contributions to date has been CFSSL, CloudFlare's PKI toolkit. We're constantly hearing from various projects and companies how CFSSL has been helpful to them -- one of the most exciting being the Let's Encrypt community Certificate Authority project. Nick Sullivan has written in the CloudFlare blog announcing CFSSL, and exciting things are continuing to happen with that software.

CloudFlare, like many other companies in computer security, makes other contributions to the security community. One of the most interesting is that we, like some other companies, values having employees participate in the security community in a variety of ways. Encouraging side projects independent of work -- research, finding new vulnerabilities and responsibly disclosing them, creating new tools, participating in conferences or working groups, running tutorials, and being active in standards bodies -- sometimes doesn't have a direct connection to the company's products, but contributes to a vibrant security ecosystem. There are often unforeseen benefits of these collaborations -- learning about new tools, finding great engineers -- we're actively hiring for a variety of roles -- and many others.

Marc and I are grateful to CloudFlare for the time to work on this open source tool and to present it to the world, and we're looking forward to presenting at RSA.

Based on our analysis of the vulnerabilities and how CloudFlare uses the OpenSSL library, this batch of vulnerabilties primarily affects CloudFlare as a "Denial of Service" possibility (it can cause CloudFlare's proxy servers to crash), rather than as an information disclosure vulnerability. Customer traffic and customer SSL keys continue to be protected.

As is good security practice, we have quickly tested the patched version and begun a push to our production environment, to be completed within the hour. We encourage all customers to upgrade to the latest patched versions of OpenSSL on their own servers, particularly if they are using the 1.0.2 branch of the OpenSSL library.

The individual vulnerabilities included in this announcement are:

• OpenSSL 1.0.2 ClientHello sigalgs DoS (CVE-2015-0291)

• Reclassified: RSA silently downgrades to EXPORT_RSA [Client] (CVE-2015-0204)

• Multiblock corrupted pointer (CVE-2015-0290)

• Segmentation fault in DTLSv1_listen (CVE-2015-0207)

• Segmentation fault in ASN1_TYPE_cmp (CVE-2015-0286)

• Segmentation fault for invalid PSS parameters (CVE-2015-0208)

• ASN.1 structure reuse memory corruption (CVE-2015-0287)

• PKCS7 NULL pointer dereferences (CVE-2015-0289)

• Base64 decode (CVE-2015-0292)

• DoS via reachable assert in SSLv2 servers (CVE-2015-0293)

• Empty CKE with client auth and DHE (CVE-2015-1787)

• Handshake with unseeded PRNG (CVE-2015-0285)

• Use After Free following d2i_ECPrivatekey error (CVE-2015-0209)

• X509_to_X509_REQ NULL pointer deref (CVE-2015-0288)

We thank the OpenSSL project and the individual vulnerability reporters for finding, disclosing, and remediating these problems. All software has bugs, sometimes security critical bugs, and having a good process for handling them once identified is a necessary part of the world of computer software.

Downgrade attacks (also known as SSL stripping attacks) are a serious threat to web applications. This type of attack is a form of on-path attacker attack in which an attacker can redirect web browsers from a correctly configured HTTPS web server to an attacker controlled server. Once the attacker has successfully redirected a user, user data, including cookies, can be compromised. Unfortunately, this attack is outside the realm of pure SSL to prevent. This is why HSTS was created.

These attacks are very real: many major websites have been attacked through SSL stripping. They are a particularly powerful attack against otherwise well secured sites, as they bypass the protections of SSL.

HSTS headers consists of an HTTP header with several parameters -- including a configurable duration for client web browsers to cache and continue to enforce policy even if the site itself changes. Through CloudFlare, it is easy to configure on a per-domain basis with standard settings.

HSTS causes compliant browsers to strictly enforce web security practices. Specifically, it automatically turns all HTTP links into HTTPS links within an application, and it upgrades all SSL errors from warnings or bypassable errors into non-bypassable errors.

The configurable parameters for HSTS are:

• Enable HSTS (Strict-Transport-Security): On/Off.

• Max Age (max-age): This is essentially a "time to live" field for the HSTS header. We recommend 6 months in order to earn an A+ rating from Qualys SSL Labs. Web browsers will cache and enforce HSTS policy for the duration of this value. A value of "0" will disable HSTS.

• Apply HSTS Policy to subdomains (includeSubDomains): Applies HSTS policy to every host in a domain.

There is one caveat to HSTS: it's a policy cached in each browser. If you configure HSTS settings, browsers will cache those settings for the duration of max-age. We recommend 6 months. If your site becomes inaccessible over strongly-configured HTTPS, web browsers will refuse to connect to the site on HTTP until the policy expires in the browser. Therefore, it's important that you set up HSTS only after establishing a stable SSL configuration. Fortunately, CloudFlare's default SSL settings are perfectly compatible with HSTS.

In order to enable HSTS for your CloudFlare protected website, you will need to use our new dashboard, currently in beta. To access this beta dashboard, first log in to your CloudFlare account. In the lower right corner of the page there is a button labeled "Try Our New Dashboard." Click and log in again. At this point, you're in our new dashboard with access to all your existing domains and settings through a completely new user interface.

There will be more information about this new dashboard in the near future, but feel free to check it out. You can continue to freely switch between old and new dashboard.

This vulnerability is a serious risk to Internet infrastructure, as it allows remote code execution in many common configurations, and the severity is heightened due to bash being in the default configuration of most Linux servers. While bash is not directly used by remote users, it is used internally by popular software packages such as web, mail, and administration servers. In the case of a web server, a specially formatted web request, when passed by the web server to the bash application, can cause the bash software to run commands on the server for the attacker. More technical information was posted on the oss-sec mailing list.

The security community has assigned this bash vulnerability the ID CVE-2014-6271.

As soon as we became aware of this vulnerability, CloudFlare’s engineering and operations teams tested a patch to protect our servers, and deployed it across our infrastructure. As of now, all CloudFlare servers are protected against CVE-2014-6271.

Everyone who is using the bash software package should upgrade as soon as possible; operating system vendors and linux distributions have released new versions today.

Additionally, CloudFlare has prepared Web Application Firewall (WAF) rules to protect customers who have not yet patched their own servers. This firewall rule is available to Pro, Business, and Enterprise customers. We have enabled this rule by default, so no WAF configuration is necessary.

UPDATE (Wed Sep 24 20:59:46 PDT 2014): At the current time, there are reports the initial bash patch deployed by most OS vendors does not fully mitigate the vulnerability. CloudFlare continues to watch the situation closely and will update both our own systems and customer-protecting WAF rules as more information becomes available. MITRE has assigned reports of additional vulnerablities CVE-2014-7169.

UPDATE (Fri Sep 26 05:08:00 EDT 2014): Over the past day, Linux distributions have released updated bash packages which address both CVE-2014-6271 and CVE-2014-7169. CloudFlare has installed these packages on all of our servers, and strongly encourages all customers to do the same. Our WAF rule remains in place, protecting Pro, Business, and Enterprise customers for web traffic going through CloudFlare.

Lackey being hoisted onto Sealand in the North Sea circa 2000

I started using the Internet when I was young—in the early 1990s, before I was a teenager. I was drawn to security for two main reasons: First, I was interested in how individuals could stand up to large groups, even nation states, using mathematics. Also, learning about computer security meant I was able to subvert security systems, and this gave me access to things I wasn’t supposed to see. I never used my skills to harm anyone, I just thought it was fun to get an account on a supercomputer and things like that.

The best way to learn about computer security, as with most technology, is to get hands-on experience. Once you have some practical experience and have decided that you’re interested, then go back and learn the theory through formal education or certifications. If you use this approach, you have an intuitive understanding of how the different parts of the field are related and it is very powerful.

Probably one of the hardest problems for young people in this field is assessing how much you really know—once you understand the basics, you think you know everything, but there can be huge problems lurking just under the surface. I think it’s important to keep in mind that no matter how much you learn, there’s always something new—it’s a rapidly changing field. This fluidity means it is possible to become a relative expert on very specific things quite early in your career. The most important thing is to learn how to find information, perform experiments, and figure out how things work. Sometimes just being able to identify a problem by name is enough to solve it, since there’s a huge amount of academic, industrial, and hacker literature about most topics.

I’m very happy! My main interest in starting CryptoSeal was to get Trusted Computing technology into commercial use. At CryptoSeal, we were working on using that technology for a general cloud computing solution, key management, and overlay networks. These are all fairly sophisticated, difficult to use applications, and not really directly usable by end users. I wanted to change that.

I think CloudFlare has done an amazing job of bringing high-end anti-DDoS, caching, firewalling, and filtering technology to a huge number of users, and by working with CloudFlare to incorporate Trusted Computing technology, I get to accomplish everything I wanted to do with CryptoSeal.

Also, CloudFlare has a really amazing team—people with cryptographic and protocol expertise, great network engineers, peering specialists, and one of the best support teams in the tech industry—so I’m really excited to be working with them.

I was first attracted to CloudFlare because I was a customer for three years, and I was always impressed with their service. They are a great service for startups, and through my interactions as a customer, I got to know some of the team.

As I looked further into CloudFlare I realized that they are solving some really difficult problems, especially now that they are operating at Internet scale—5% of web requests. Sometimes projects at CloudFlare require actually fixing the underlying infrastructure of the Internet, and the company is willing—and able—to invest the resources to make that happen.

The three founders, Matthew, Lee, and Michelle, are actively involved, and they’ve created CloudFlare to be a flat organization without unneeded bureaucracy and process. As a company, it’s a great place to be -- the hiring bar is really high, so all of your coworkers are brilliant and hard-working. Everyone is focused on doing the right thing for CloudFlare’s users, and for the Internet as a whole. (If you’re interested, we’re hiring. Check out our openings here).

I’ve done a lot of different things. I worked on an early anonymous electronic cash system while living on a Caribbean island in the late 1990s. The electronic cash system didn’t work out, but we ended up building some useful cryptographic tools that we later used in other products.

My neighbors on the island, the folks who ran the “.ai” domain name, introduced me to Sean Hastings. After I left the Caribbean, Sean and I got together to figure out the best place to host content free of government interference, but we couldn’t find a country which would be good enough. At that point, we bought a book: “How to Start Your Own Country”, and soon after that we found this abandoned WW2 anti-aircraft fortress occupied by pirate radio people called Sealand in the North Sea. It was exciting, but the costs to provide service were really high: diesel fuel, helicopters, boats, etc. So I eventually left and moved back to the US.

For a couple years, I worked on cryptographic software for payment systems companies and RFID/NFC payments for credit cards. When the Iraq War started, and I got in touch with some Iraqi expats who needed help setting up Internet in Iraq after the US intervention. I flew into Iraq on a civilian flight, and spent six months working with them on satellite and wireless networking for a variety of military, government, and commercial customers. Then, as the country got more dangerous, I moved onto a US military base in Iraq and spent the next six years doing defense contracting, primarily building satellite, cellular, and wireless networks for a diverse set of customers in Iraq, Afghanistan, Kuwait, and elsewhere.

After enough close calls with explosions and helicopters, and missing the Bay Area, I moved back to work on a more “conventional” tech startup—CryptoSeal—which was also a great adventure.

Getting to focus on the parts of the company I really care about which are product and technology, and not having to constantly worry about administration, finance, etc. It’s more efficient, less stressful, and produces better results. Since CloudFlare has such a great team, I’m also really enjoying getting to learn from people across the company.