When you visit a secure website, it offers you a TLS certificate that asserts its identity. Every certificate has an expiration date, and when it’s passed due, it is no longer valid. The idea is almost as old as the web itself: limiting the lifetime of certificates is meant to reduce the risk in case a TLS server’s secret key is compromised.

Certificates aren’t the only cryptographic artifacts that expire. When you visit a site protected by Cloudflare, we also tell you whether its certificate has been revoked (see our blog post on OCSP stapling) — for example, due to the secret key being compromised — and this value (a so-called OCSP staple) has an expiration date, too.

Thus, to determine if a certificate is valid and hasn’t been revoked, your system needs to know the current time. Indeed, time is crucial for the security of TLS and myriad other protocols. To help keep clocks in sync, we are announcing a free, high-availability, and low-latency authenticated time service called Roughtime, available at roughtime.cloudflare.com on port 2002.

It may surprise you to learn that, in practice, clients’ clocks are heavily skewed. A recent study of Chrome users showed that a significant fraction of reported TLS-certificate errors are caused by client-clock skew. During the period in which error reports were collected, 6.7% of client-reported times were behind by more than 24 hours. (0.05% were ahead by more than 24 hours.) This skew was a causal factor for at least 33.5% of the sampled reports from Windows users, 8.71% from Mac OS, 8.46% from Android, and 1.72% from Chrome OS. These errors are usually presented to users as warnings that the user can click through to get to where they’re going. However, showing too many warnings makes users grow accustomed to clicking through them; this is risky, since these warnings are meant to keep users away from malicious websites.

Clock skew also holds us back from improving the security of certificates themselves. We’d like to issue certificates with shorter lifetimes because the less time the certificate is valid, the lower the risk of the secret key being exposed. (This is why Let’s Encrypt issues certificates valid for just 90 days by default.) But the long tail of skewed clocks limits the effective lifetime of certificates; shortening the lifetime too much would only lead to more warnings.

Endpoints on the Internet often synchronize their clocks using a protocol like the Network Time Protocol (NTP). NTP aims for precise synchronization, and even accounts for network latency. However, it is usually deployed without security features, as the added overhead on high-load servers degrades precision significantly. As a result, a man-in-the-middle attacker between the client and server can easily influence the client’s clock. By moving the client back in time, the attacker can force it to accept expired (and possibly compromised) certificates; by moving forward in time, it can force the client to accept a certificate that is not yet valid.

Fortunately, for settings in which both security and precision are paramount, workable solutions are on the horizon. But for many applications, precise network time isn’t essential; it suffices to be accurate, say, within 10 seconds of real time. This observation is the primary motivation of Google’s Roughtime protocol, a simple protocol by which clients can synchronize their clocks with one or more authenticated servers. Roughtime lacks the precision of NTP, but aims to be accurate enough for cryptographic applications, and since the responses are authenticated, man-in-the-middle attacks aren’t possible.

The protocol is designed to be simple and flexible. A client can get Roughtime from just one server it trusts, or it may contact many servers to make its calculation more robust. But its most distinctive feature is that it adds accountability to time servers. If a server misbehaves by providing the wrong time, then the protocol allows clients to produce publicly verifiable, cryptographic proof of this misbehavior. Making servers auditable in this manner makes them accountable to provide accurate time.

We are deploying a Roughtime service for two reasons.

First, the clock we use for this service is the same as the clock we use to determine whether our customers’ certificates are valid and haven’t been revoked; as a result, exposing this service makes us accountable for the validity of TLS artifacts we serve to clients on behalf of our customers.

Second, Roughtime is a great idea whose time has come. But it is only useful if several independent organizations participate; the more Roughtime servers there are, the more robust the ecosystem becomes. Our hope is that putting our weight behind it will help the Roughtime ecosystem grow.

At its most basic level, Roughtime is a one-round protocol in which the client requests the current time and the server sends a signed response. The response is comprised of a timestamp (the number of microseconds since the Unix epoch) and a radius (in microseconds) used to indicate the server’s certainty about the reported time. For example, a radius of 1,000,000μs means the server is reasonably sure that the true time is within one second of the reported time.

The server proves freshness of its response as follows. The request consists of a short, random string commonly called a nonce (pronounced /nän(t)s/, or sometimes /ˈen wən(t)s/). The server incorporates the nonce into its signed response so that it’s needed to verify the signature. If the nonce is sufficiently long (say, 16 bytes), then the number of possible nonces is so large that it’s extremely unlikely the server has encountered (or will ever encounter) a request with the same nonce. Thus, a valid signature serves as cryptographic proof that the response is fresh.

The client uses the server’s root public key to verify the signature. (The key is obtained out-of-band; you can get our key here.) When the server starts, it generates an online public/secret key pair; the root secret key is used to create a delegation for the online public key, and the online secret key is used to sign the response. The delegation serves the same function as a traditional X.509 certificate on the web: as illustrated in the figure below, the client first uses the root public key to verify the delegation, then uses the online public key to verify the response. This allows for operational separation of the delegator and the server and limits exposure of the root secret key.

Simplified Roughtime (without delegation)

Roughtime with delegation

Roughtime offers two features designed to make it scalable.  First, when the volume of requests is high, the server may batch-sign a number of clients’ requests by constructing a Merkle tree from the nonces. The server signs the root of the tree and sends in its response the information needed to prove to the client that its request is in the tree. (The data structure is a binary tree, so the amount of information is proportional to the base-2 logarithm of the number of requests in the batch; see the figure below) Second, the protocol is executed over UDP. In order to prevent the Roughtime server from being an amplifier for DDoS attacks, the request is padded to 1KB; if the UDP packet is too short, then it’s dropped without further processing. Check out this blog post for a more in-depth discussion.

Roughtime with batching

The protocol is flexible enough to support a variety of use cases. A web browser could use a Roughtime server to proactively synchronize its clock when validating TLS certificates. It could also be used retroactively to avoid showing the user too many warnings: when a certificate validation error occurs — in particular, when the browser believes it’s expired or not yet valid — Roughtime could be used to determine if the clock skew was the root cause. Instead of telling the user the certificate is invalid, it could tell the user that their clock is incorrect.

Using just one server is sufficient if that server is trustworthy, but a security-conscious user could make requests to many servers; the delta might be computed by eliminating outliers and averaging the responses, or by some more sophisticated method. This makes the calculation robust to one or more of the servers misbehaving.

The real power of Roughtime is that it’s auditable. Consider the following mode of operation. The client has a list of servers it will query in a particular order. The client generates a random string — called a blind in the parlance of Roughtime — hashes it, and uses the output as the nonce for its request to the server. For subsequent requests, it computes the nonce as follows: generate a blind, compute the hash of this string and the response from the previous server (including the timestamp and signature), and use this hash as the nonce for the next request.

Chaining multiple Roughtime servers

Creating a chain of timestamps in this way binds each response to the response that precedes it. Thus, the sequence of blinds and signatures constitute a publicly verifiable, cryptographic proof that the timestamps were requested in order (a “clockchain” if you will ?). If the servers are roughly synchronized, then we expect that the sequence to monotonically increase, at least roughly. If one of the servers were consistently behind or ahead of the others, then this would be evident in the sequence. Suppose you get the following sequence of timestamps, each from different servers:

Servers B and C corroborate the time given by server A, but — oh no! Cloudflare is two hours behind! Unless servers A, B, and C are in cahoots, it’s likely that the time offered by Cloudflare is incorrect. Moreover, you have verifiable, cryptographic proof. In this way, the Roughtime protocol makes our server (and all Roughtime servers) accountable to provide accurate time, or, at least, to be in sync with the others.

The infrastructure for monitoring and auditing the Roughtime ecosystem hasn’t been built yet. Right now there’s only a handful of servers: in addition to Cloudflare’s and Google’s, there’s also a really nice Rust implementation. The more diversity there is, the healthier the ecosystem becomes. We hope to see more organizations adopt this protocol.

For the initial deployment of this service, our primary goals are to ensure high availability and minimal maintenance overhead. Each machine at each Cloudflare location executes an instance of the service and responds to queries using its system clock. The server signs each request individually rather than batch-signing them as described above; we rely on our load balancer to ensure no machine is overwhelmed. There are three ways in which we envision this service could be used:

1. TLS authentication. When a TLS application (a web browser for example) starts, it could make a request to roughtime.cloudflare.com and compute the difference between the reported time and its system time. Whenever it authenticates a TLS server, it would add this difference to the system time to get the current time.

2. Roughtime daemon. One could implement an OS daemon that periodically requests the current time. If the reported time differs from the system time by more than a second, it might issue an alert.

3. Server auditing. As the Roughtime ecosystem grows, it will be important to ensure that all of the servers are in sync. Individuals or organizations may take it upon themselves to monitor the ecosystem and ensure that the servers are in sync with one another.

The service is reachable wherever you are via our anycast network. This is important for a service like Roughtime, because minimizing network latency helps improve accuracy. For information about how to configure a client to use Cloudflare-Roughtime, check out the developer documentation. Note that our initial release is somewhat experimental. As such, our root public key may change in the future. See the developer docs for information on obtaining the current public key.

If you want to see what time our Roughtime server returns, click the button below!

function getTime() { document.getElementById("time-txt-box").innerHTML = "Loading..." fetch("/cdn-cgi/trace").then((res) => res.text()).then((txt) => { let ts = txt.match(/ts=([0-9\.]+)/)[1] let str = new Date(parseFloat(ts) * 1000) document.getElementById("time-txt-box").innerHTML = str }) .catch((err) => { console.log(err) document.getElementById("time-txt-box").innerHTML = "Request failed." }) }

Get Time! 

Subscribe to the blog for daily updates on our announcements.

Digital certificates are the cornerstone of trust on the web. A digital certificate is like an identification card for a website. It contains identity information including the website’s hostname along with a cryptographic public key. In public key cryptography, each public key has an associated private key. This private key is kept secret by the site owner. For a browser to trust an HTTPS site, the site’s server must provide a certificate that is valid for the site’s hostname and a proof of control of the certificate’s private key. If someone gets access to a certificate’s private key, they can impersonate the site. Private key compromise is a serious risk to trust on the web.

Certificate revocation is a way to mitigate the risk of key compromise. A website owner can revoke a compromised certificate by informing the certificate issuer that it should no longer be trusted. For example, back in 2014, Cloudflare revoked all managed certificates after it was shown that the Heartbleed vulnerability could be used to steal private keys. There are other reasons to revoke, but key compromise is the most common.

Certificate revocation has a spotty history. Most of the revocation checking mechanisms implemented today don’t protect site owners from key compromise. If you know about why revocation checking is broken, feel free to skip ahead to the OCSP stapling section below.

There are several ways a web browser can check whether a site’s certificate is revoked or not. The most well-known mechanisms are Certificate Revocation Lists (CRL) and Online Certificate Status Protocol (OCSP). A CRL is a signed list of serial numbers of certificates revoked by a CA. OCSP is a protocol that can be used to query a CA about the revocation status of a given certificate. An OCSP response contains signed assertions that a certificate is not revoked.

Certificates that support OCSP contain the responder's URL and those that support CRLs contain a URLs where the CRL can be obtained. When a browser is served a certificate as part of an HTTPS connection, it can use the embedded URL to download a CRL or an OCSP response and check that the certificate hasn't been revoked before rendering the web page. The question then becomes: what should the browser do if the request for a CRL or OCSP response fails? As it turns out, both answers to that question are problematic.

When browsers encounter a web page and there’s a problem fetching revocation information, the safe option is to block the page and show a security warning. This is called a hard-fail strategy. This strategy is conservative from a security standpoint, but prone to false positives. For example, if the proof of non-revocation could not be obtained for a valid certificate, a hard-fail strategy will show a security warning. Showing a security warning when no security issue exists is dangerous because it can lead to warning fatigue and teach users to click through security warnings, which is a bad idea.

In the real world, false positives are unavoidable. OCSP and CRL endpoints subject to service outages and network errors. There are also common situations where these endpoints are completely inaccessible to the browser, such as when the browser is behind a captive portal. For some access points used in hotels and airplanes, unencrypted traffic (like OCSP endpoints) are blocked. A hard-fail strategy force users behind captive portals and other networks that block OCSP requests to click through unnecessary security warnings. This reality is unpalatable to browser vendors.

Another drawback to a hard-fail strategy is that it puts an increased burden on certificate authorities to keep OCSP and CRL endpoints available and online. A broken OCSP or CRL server becomes a central point of failure for all certificates issued by a certificate authority. If browsers followed a hard-fail strategy, an OCSP outage would be an Internet outage. Certificate authorities are organizations optimized to provide trust and accountability, not necessarily resilient infrastructure. In a hard-fail world, the availability of the web as a whole would be limited by the ability for CAs to keep their OCSP services online at all times; a dangerous systemic risk to the internet as a whole.

To avoid the downsides of a hard-fail strategy, most browsers take another approach to certificate revocation checking. Upon seeing a new certificate, the browser will attempt to fetch the revocation information from the CRL or OCSP endpoint embedded in the certificate. If the revocation information is available, they rely on it, and otherwise they assume the certificate is not revoked and display the page without any errors. This is called a “soft-fail” strategy.

The soft-fail strategy has a critical security flaw. An attacker with network position can block the OCSP request. If this attacker also has the private key of a revoked certificate, they can intercept the outgoing connection for the site and present the revoked certificate to the browser. Since the browser doesn’t know the certificate is revoked and is following a soft-fail strategy, the page will load without alerting the user. As Adam Langley described: “soft-fail revocation checks are like a seat-belt that snaps when you crash. Even though it works 99% of the time, it's worthless because it only works when you don't need it.”

A soft-fail strategy also makes connections slower. If revocation information for a certificate is not already cached, the browser will block the rendering of the page until the revocation information is retrieved, or a timeout occurs. This additional step causes a noticeable and unwelcome delay, with marginal security benefits. This tradeoff is a hard sell for the performance-obsessed web. Because of the limited benefit, some browsers have eliminated live revocation checking for at least some subset of certificates.

Live OCSP checking has an additional downside: it leaks private browsing information. OCSP requests are sent over unencrypted HTTP and are tied to a specific certificate. Sending an OCSP request tells the certificate authority which websites you are visiting. Furthermore, everyone on the network path between your browser and the OCSP server will also know which sites you are browsing.

Some client still perform soft-fail OCSP checking, but it’s becoming less common due to the performance and privacy downsides described above. To protect high-value certificates, some browsers have explored alternative mechanisms for revocation checking.

One technique is to pre-package a list of revoked certificates and distribute them through browser updates. Because the list of all revoked certificates is so large, only a few high-impact certificates are included in this list. This technique is called OneCRL by Firefox and CRLSets by Chrome. This has been effective for some high-profile revocations, but it is by no means a complete solution. Not only are not all certificates covered, this technique leaves a window of vulnerability between the time the certificate is revoked and the certificate list gets to browsers.

OCSP stapling is a technique to get revocation information to browsers that fixes some of the performance and privacy issues associated with live OCSP fetching. In OCSP stapling, the server includes a current OCSP response for the certificate included (or "stapled") into the initial HTTPS connection. That removes the need for the browser to request the OCSP response itself. OCSP stapling is widely supported by modern browsers.

Not all servers support OCSP stapling, so browsers still take a soft-fail approach to warning the user when the OCSP response is not stapled. Some browsers (such as Safari, Edge and Firefox for now) check certificate revocation for certificates, so OCSP stapling can provide a performance boost of up to 30%. For browsers like Chrome that don’t check for revocation for all certificates, OCSP stapling provides a proof of non-revocation that they would not have otherwise.

Cloudflare started offering OCSP stapling in 2012. Cloudflare’s original implementation relied on code from nginx that was able to provide OCSP stapling for a some, but not all connections. As Cloudflare’s network grew, the implementation wasn’t able to scale with it, resulting in a drop in the percentage of connections with OCSP responses stapled. The architecture we had chosen had served us well, but we could definitely do better.

In the last year we redesigned our OCSP stapling infrastructure to make it much more robust and reliable. We’re happy to announce that we now provide reliable OCSP stapling connections to Cloudflare. As long as the certificate authority has set up OCSP for a certificate, Cloudflare will serve a valid OCSP stapled response. All Cloudflare customers now benefit from much more reliable OCSP stapling.

In Cloudflare’s original implementation of OCSP stapling, OCSP responses were fetched opportunistically. Given a connection that required a certificate, Cloudflare would check to see if there was a fresh OCSP response to staple. If there was, it would be included in the connection. If not, then the client would not be sent an OCSP response, and Cloudflare would send a request to refresh the OCSP response in the cache in preparation for the next request.

If a fresh OCSP response wasn’t cached, the connection wouldn’t get an OCSP staple. The next connection for that same certificate would get a OCSP staple, because the cache will have been populated.

This architecture was elegant, but not robust. First, there are several situations in which the client is guaranteed to not get an OCSP response. For example, the first request in every cache region and the first request after an OCSP response expires are guaranteed to not have an OCSP response stapled. With Cloudflare’s expansion to more locations, these failures were more common. Less popular sites would have their OCSP responses fetched less often resulting in an even lower ratio of stapled connections. Another reason that connections could be missing OCSP responses is if the OCSP request from Cloudflare to fill the cache failed. There was a lot of room for improvement.

In order to be able to reliably include OCSP staples in all connection, we decided to change the model. Instead of fetching the OCSP response when a request came in, we would fetch it in a centralized location and distribute valid responses to all our servers. When a response started getting close to expiration, we’d fetch a new one. If the OCSP request fails, we put it into a queue to re-fetch at a later time. Since most OCSP staples are valid for around 7 days, there is a lot of flexibility in term of refreshing expiring responses.

To keep our cache of OCSP responses fresh, we created an OCSP fetching service. This service ensures that there is a valid OCSP response for every certificate managed by Cloudflare. We constantly crawl our cache of OCSP responses and refresh those that are close to expiring. We also make sure to never cache invalid OCSP responses, as this can have bad consequences. This system has been running for several months now, and we are now reliably including OCSP staples for almost every HTTPS request.

Reliable stapling improves performance for browsers that would have otherwise fetched OCSP, but it also changes the optimal failure strategy for browsers. If a browser can reliably get an OCSP staple for a certificate, why not switch back from a soft-fail to a hard-fail strategy?

As described above, the soft-fail strategy for validating OCSP responses opens up a security hole. An attacker with a revoked certificate can simply neglect to provide an OCSP response when a browser connects to it and the browser will accept their revoked certificate.

In the OCSP fetching case, a soft-fail approach makes sense. There many reasons the browser would not be able to obtain an OCSP: captive portals, broken OCSP servers, network unreliability and more. However, as we have shown with our high-reliability OCSP fetching service, it is possible for a server to fetch OCSP responses without any of these problems. OCSP responses are re-usable and are valid for several days. When one is close to expiring, the server can fetch a new one out-of-band and be able to reliably serve OCSP staples for all connections.

Public Domain

If the client knows that a server will always serve OCSP staples for every connection, it can apply a hard-fail approach, failing a connection if the OCSP response is missing. This closes the security hole introduced by the soft-fail strategy. This is where OCSP must-staple fits in.

OCSP must-staple is an extension that can be added to a certificate that tells the browser to expect an OCSP staple whenever it sees the certificate. This acts as an explicit signal to the browser that it’s safe to use the more secure hard-fail strategy.

Firefox enforces OCSP must-staple, returning the following error if such a certificate is presented without a stapled OCSP response.

Chrome provides the ability to mark a domain as “Expect-Staple”. If Chrome sees a certificate for the domain without a staple, it will send a report to a pre-configured report endpoint.

As a part of our push to provide reliable OCSP stapling, we put our money where our mouths are and put an OCSP must-staple certificate on blog.cloudflare.com. Now if we ever don’t serve an OCSP staple, this page will fail to load on browsers like Firefox that enforce must-staple. You can identify a certificate by looking at the certificate details for the “1.3.6.1.5.5.7.1.24” OID.

Cloudflare customers can choose to upload must-staple custom certificates, but we encourage them not to do so yet because there may be a multi-second delay between the certificate being installed and our ability to populate the OCSP response cache. This will be fixed in the coming months. Other than the first few seconds after uploading the certificate, Cloudflare’s new OCSP fetching is robust enough to offer OCSP staples for every connection thereafter.

As of today, an attacker with access to the private key for a revoked certificate can still hijack the connection. All they need to do is to place themselves on the network path of the connection and block the OCSP request. OCSP must-staple prevents that, since an attacker will not be able to obtain an OCSP response that says the certificate has not been revoked.

For browsers, an OCSP failure is not the end of the world. Most browsers are configured to soft-fail when an OCSP responder returns an error, so users are unaffected by OCSP server failures. Some Certificate Authorities have had massive multi-day outages in their OCSP servers without affecting the availability of sites that use their certificates.

There’s no strong feedback mechanism for broken or slow OCSP servers. This lack of feedback has led to an ecosystem of faulty or unreliable OCSP servers. We experienced this first-hand while developing high-reliability OCSP stapling. In this section, we’ll outline half a dozen unexpected behaviors we found when deploying high-reliability OCSP stapling. A big thanks goes out to all the CAs who fixed the issues we pointed out. CA names redacted to preserve their identities.

We noticed CA #1 certificates frequently missing their refresh-deadline, and during debugging we were lucky enough to see this:

$ date -u
Sat Mar  4 02:45:35 UTC 2017
$ ocspfetch <redacted, customer ID>
This Update: 2017-03-04 01:45:49 +0000 UTC
Next Update: 2017-03-04 02:45:49 +0000 UTC
$ date -u
Sat Mar  4 02:45:48 UTC 2017
$ ocspfetch <redacted, customer ID>
This Update: 2017-03-04 02:45:49 +0000 UTC
Next Update: 2017-03-04 03:45:49 +0000 UTC

It shows that CA #1 had configured their OCSP responders to use an incredibly short validity period with almost no overlap between validity periods, which makes it functionally impossible to always have a fresh OCSP response for their certificates. We contacted them, and they reconfigured the responder to produce new responses every half-interval.

Several certificates from CA #2 started failing with this error:

 bad OCSP signature: crypto/rsa: verification error

The issue is that the OCSP claims to be signed with SHA256-RSA, when it is actually signed with SHA1-RSA (and the reverse: indicates SHA1-RSA, actually signed with SHA256-RSA).

When we first started the project, our tool was unable to parse dozens of certificates in our database because of this error

asn1: structure error: integer not minimally-encoded

and many OCSP responses that we fetched from the same CA:

parsing ocsp response: bad OCSP signature: asn1: structure error: integer not minimally-encoded

What happened was that this CA had begun issuing <1% of certificates with a minor formatting error that rendered them unparseable by Golang’s x509 package. After contacting them directly, they quickly fixed the issue but then we had to patch Golang's parser to be more lenient about encoding bugs.

A small number of CA #4’s OCSP responders fell into a "bad state" without them knowing, and would return 404 on every request. Since the CA used load balancing to round-robin requests to a number of responders, it looked like 1 in 6 requests would fail inexplicably.

Two times between Jan 2017 and May 2017, this CA also experienced some kind of large data-loss event that caused them to return persistent "Try Later" responses for a large number of requests.

When a certificate is issued by CA #5, there is a large delay between the time a certificate is issued and the OCSP responder is able to start returning signed responses. This results in a delay between certificate issance and the availability of OCSP. It has mostly been resolved, but this general pattern is dangerous for OCSP must-staple. There have been some changes discussed by the CA/B Forum, an organization that regulates the issuance and management of certificates, to require CAs to offer OCSP soon after issuance.

It is typical to embed only one certificate in an OCSP response, if any. The one embedded certificate is supposed to be a leaf specially issued for signing an intermediate's OCSP responses. However, several CAs embed multiple certificates: the leaf they use for signing OCSP responses, the intermediate itself, and sometimes all the intermediates up to the root certificate.

We made OCSP stapling better and more reliable for Cloudflare customers. Despite the various strange behaviors we found in OCSP servers, we’ve been able to consistently serve OCSP responses for over 99.9% of connections since we’ve moved over to the new system. This great work was done by Brendan McMillion and Alessandro Ghedini. This is an important step in protecting the web community from attackers who have compromised certificate private keys.

We’re also happy to announce CFSSL’s new home: cfssl.org. From there you can try out CFSSL’s user interface, download binaries, and test some of its features.

Licensing: Public Domain

This 2013 National Security Agency (NSA) slide describing how data from Google’s internal network was collected by intelligence agencies was eye-opening—and shocking—to many technology companies. The idea that an attacker could read messages passed between services wasn’t technically groundbreaking, but it did reveal a security flaw in the way many distributed systems were designed. Many companies only encrypted the data to the border of their datacenter, not inside. The slide showed that private physical networks are being subverted to extract data passing through them. And just because a network has a security perimeter, it doesn’t mean that data can be safely sent between applications unencrypted inside that perimeter. In short: treat your own network as hostile.

This mentality helped shape CloudFlare’s philosophy for securing internal services and resulted in a simple rule:

Services should only communicate with each other using encrypted and mutually authenticated protocols.

With this in mind we started tackling the harder problem of how to manage the encryption keys for these services. To tackle the issue of service-to-service encryption, we built our own public key infrastructure using CFSSL. Much of the new features we’re introducing in this post came about from our effort to make this system robust.

We have also made an effort to use standards-compliant and interoperable technology. By incorporating support for certificate transparency, OSCP, and CRL, the standards used by the public Internet can now be used in your private infrastructure. Now, on to the new features.

CFSSL now has a full-featured TLS endpoint scanner.

Just because a server uses encryption, it doesn’t mean that it is secure. There have been a series of vulnerabilities in TLS that only affect some configurations. To keep your server and its visitors protected against the nearly monthly new attacks you need to pick the right configuration. Staying secure requires testing your configuration against the latest vulnerabilities, and keeping your configuration updated against new threats.

History of vulnerabilities in SSL/TLS

The gold standard for testing a website’s TLS configuration is Ivan Ristić’s SSL Labs. It provides a simple letter grade for your site’s configuration (sites using CloudFlare get an A, by the way, and A+ if you enable HSTS). The drawback of SSL Labs is that it only works on public websites: you can’t use it for internal services. At CloudFlare, we needed an easy way to check the configuration of our services as well as our customers’ origins (which are typically not publicly accessible).

To solve this, CloudFlare added functionality to CFSSL to scan a TLS endpoint to evaluate how securely it’s configured. With it, we are able to check the configuration of internal services and protected customer origins for the following configuration issues:

• IPv4/IPv6 connectivity

• Certificate validity (expiration, trust chain, hostnames, etc.)

• Supported cipher suites and algorithms

• Session resumption

• Revoked certificates

Each scan provides a grade of "Good" or "Bad". CFSSL Scan can also be used to scan entire IP ranges or lists of hosts. It can be used either as a CLI or as API-driven server.

Using the CLI is a simple command:

$ cfssl scan cloudflare.com
{
  "Connectivity": {
	"DNSLookup": {
	  "grade": "Good",
	  "output": [
		"198.41.215.162",
		"198.41.214.162",
		"2400:cb00:2048:1::c629:d6a2",
		"2400:cb00:2048:1::c629:d7a2"
	  ]
	},
	"TCPDial": {
	  "grade": "Good"
	},
	"TLSDial": {
	  "grade": "Good"
	}
  },
  "PKI": {
	"ChainExpiration": {
	  "grade": "Good",
	  "output": "2016-11-30T23:59:59Z"
	},
	"ChainValidation": {
	  "grade": "Warning",
	  "output": [
		"Certificate for COMODO ECC Extended Validation Secure Server CA is valid for too long"
	  ]
	},
	"MultipleCerts": {
	  "grade": "Good"
	}
  },
  "TLSHandshake": {
	"CertsByCiphers": {
	  "grade": "Good",
	  "output": {
		"TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_3DES_EDE_CBC_SHA": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_128_CBC_SHA": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_128_CBC_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_128_GCM_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_256_CBC_SHA": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_256_CBC_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_256_GCM_SHA384": "SHA256WithRSA"
	  }
	}
  }
}

CFSSL Scan also accessible as part of the new CFSSL UI.

An important design pattern in security engineering is secure defaults. Developers want to write secure software and aren’t always security experts, let alone crypto gurus. The two trickiest parts of deploying an application that speaks TLS are:

1. Configuration

2. Key management

Simplicity is the key to empowering developers to use encryption in their services. We created the CFSSL Transport package to make these two tasks easy for our Go developers.

Transport is a Go library that takes regular HTTP or TCP connections, and transparently turns them into encrypted connections. Transport handles all the sticky points so that the developer doesn’t have to. This includes creating a private key, getting a certificate for it using a CFSSL CA, renewing certificates before they expire, and choosing the correct cryptographic parameters. If you’re writing a service in Go, you no longer need to know how PKI works.

Certificate Issuance with CFSSL CA

Not only does the Transport handle setting up and rotating certificates, it automatically checks to make sure the services your service are connecting to are using a valid certificate, including checking for revocation (more on that later).

OCSP Check with CFSSL CA

Internal CAs can be used to set up coarse-grained authorization between services. For example, if you have both an API server and a database, you can set up a dedicated CA for each of them. In the example below, the API server CA is in orange and the DB CA is in red. You can then configure the DB to only trust connections from the API server and vice versa. This type of setup can provide a baseline level of authorization enforcement for your applications. The transport package lets you automate the setup of these mutually-authenticated connections. This type of setup is covered in a previous blog post.

Once you have a CFSSL CA (or a multi-root CA) up and running, it just takes a few lines of code to start using TLS in your Go application. Just swap your standard net.Dial or net.Listen/Accept with transport.Dial and transport.Listen/Accept.

Before:

conn, err := net.Dial("tcp", addr)
if err != nil {
	// handle error
}

After (configuration file location stored in the conf variable):

var id = new(core.Identity)
data, err := ioutil.ReadFile(conf)
if err != nil {
	// handle error
}
err = json.Unmarshal(data, id)
if err != nil {
	// handle error
}

    // Renew 5 minutes before expiry
tr, err := transport.New(5 * time.Minute, id)
if err != nil {
	// handle error
}
conn, err := transport.Dial(addr, tr)
if err != nil {
	// handle error
}

You can start playing around with the transport package with some examples from Github:

https://github.com/cloudflare/cfssl/tree/master/transport/example

CC Creative Commons Attribution-Share Alike 3.0 Unported

One of the nice things about CFSSL is that you can easily spin it up inside your infrastructure and have a certificate authority. One of the risks of running a PKI is infrastructure compromise. If the private key material for a certificate falls into the wrong hands, there need to be mechanisms so that the rest of the system knows to no longer trust that certificate.

The first step in knowing which certificates are trusted is knowing which certificates have been issued. To solve this, we added the ability to keep track, in a persistent database, of which certificates have been issued and the subset of those that have been revoked. We are big fans of PostgreSQL, so we built a database backend for CFSSL in PostgreSQL, but other backends like MySQL are in development. You can now set up CFSSL to use a certificate database with very little work, and we leveraged that integration to create an automated revocation system.

The two standard mechanisms for signaling that a certificate is no longer trusted are certificate revocation lists (CRLs) and the online certificate status protocol (OCSP). CFSSL now fully supports both of these mechanisms.

A CRL is simply a list of revoked certificate serial numbers. It covers all certificates issued by a CA that have not expired, and is digitally signed by the CA’s private key. When a client obtains a certificate, it can simply look at this list to check to see if the certificate has been revoked. CRL files can grow quite a bit if a lot of certificates are revoked, and can therefore cause some scalability issues. We saw this after Heartbleed, when we revoked a large number of customer certificates at once.

Partly to combat these scalability issues, OCSP was introduced. OCSP provides on-demand answers about the revocation status of a given certificate. An OCSP responder is a service that returns signed answers to the question "is this certificate revoked?". The response is either "Yes" or "No". Each response is signed by the CA and has a validity period so the client knows how long to cache the response.

CFSSL now has an OCSP responder service that can be configured to run in a distributed way, without access to the CA. There are also OCSP management tools in CFSSL to automatically populate the data for the OCSP responder and keep it fresh using the certificate database.

In CFSSL, you can now programmatically create CRLs and OCSP responses for certificates issued by your CA. Using standards-compatible revocation mechanisms allows these certificates to be shared outside of our infrastructure and to work with most software that implements TLS.

Another exciting new PKI standard is Certificate Transparency (CT). It helps provide (as the name implies) transparency into the workings of a certificate authority by providing an append-only log of issued certificates.

You can think of CT as a public ledger of all certificates issued. Any certificate on the list (even if issued for use on a private network) is made public and can be checked to see if it was issued according to the rules of the CA/Browser forum. If you encounter a certificate that is not on the ledger, then it may have been created fraudulently. Google Chrome currently requires all Extended Validation certificates used by websites to be in the CT log.

CFSSL now allows you to submit certificates to a CT log at issuance time and automatically embed the proof that it has been logged into the certificate. Running a CT log inside your internal infrastructure is a nice way to audit your CA and catch mis-issuances.

CFSSL is great for software deployments, as you can spin it up anywhere and run it on any platform that Go supports. You can even use our convenient Dockerfiles to deploy it in a containerized environment. However, in some situations (like running a publicly-trusted CA), keeping a private key in software is not secure enough. For these situations, hardware-based protection is needed.

The industry standard protocol for working with cryptographic hardware is called PKCS#11. With help from Richard Barnes of Mozilla and others we were able to add support for PKCS#11 into CFSSL. This feature is can be enabled in programs that use the signer/local package and the pkcs11key package. We also have plans to add command line support using the PKCS#11 URI specification. If you have a PKCS#11 interface to your HSM, certificate creation using that key is fully supported by the cfssl/signer package. Power users including Let’s Encrypt use CFSSL to run their publicly trusted CA while keeping the private key in a FIPS 140-2 certified HSM.

Open source is hard. Different people have different needs, and as a project maintainer you have to be respectful of these needs while honoring the spirit of the project. CloudFlare’s needs for CFSSL are not identical to the needs of its other users. We have attempted to strike a balance between building a tool for our own specific use cases and building a great general-purpose toolkit for PKI/TLS. We are grateful to the open source community for their valuable contributions to this project and are proud to be part of the tradition of free and open source software.

I’d like to thank one of the largest contributors to CFSSL over the last year: the Let’s Encrypt project. They have contributed code reviews and useful features while integrating CFSSL into Boulder, the software that manages their certificate authority. I’d also like to thank the Open Academy participants from Cornell and UCSD who worked on the project for a semester, and everyone else who helped contribute to this release. The core CFSSL team is Kyle Isom, Zi Lin, Jacob Haven, and Nick Sullivan.

A quick followup to our last blog post on our decision to reissue and revoke all of CloudFlare's customers' SSL certificates. One question we've received is why we didn't just reissue and revoke all SSL certificates as soon as we got word about the Heartbleed vulnerability? The answer is that the revocation process for SSL certificates is far from perfect and imposes a significant cost on the Internet's infrastructure.

Today, after having done a mass reissuance and revocation, we have a tangible sense of that cost. To understand it, you need to understand a bit about how your browser checks if an SSL certificate has been revoked.

When most browsers visit web pages over HTTPS they perform a check using one of two certificate revocation methods: Online Certificate Status Protocol (OCSP) or Certificate Revocation List (CRL). For OCSP, the browser pings the certificate authority and asks whether a particular site's certificate has been revoked. For CRL, the browser pings the certificate authority (CA) and downloads a complete list of all the certificates that have been revoked by that CA.

There are pluses and minuses to both systems. OCSP imposes a lighter bandwidth cost, but a higher number of requests and backend lookups. CRL doesn't generate as many requests, but, as the CRL becomes large, can impose a significant bandwidth burden. These costs are borne by visitors to websites, whose experience will be slower as a result, but even more so by the CAs who need significant resources in place to handle these requests.

Yesterday, CloudFlare completed the process of reissuing all the SSL certificates we manage for our customers. Once that was complete, we revoked all previously used certificates. You can see the spike in global CRL activity we generated:

What you can't see is the spike in bandwidth that imposed. Globalsign, who is CloudFlare's primary CA partner, saw their CRL grow to approximately 4.7MB in size from approximately 22KB on Monday. The activity of browsers downloading the Globalsign CRL generated around 40Gbps of net new traffic across the Internet. If you assume that the global average price for bandwidth is around \$10/Mbps, just supporting the traffic to deliver the CRL would have added \$400,000USD to Globalsign's monthly bandwidth bill.

Lest you think that’s an overestimate, to make the total costs more accurate, we ran the numbers using AWS’s CloudFront price calculator using a mix of traffic across regions that approximates what we see at CloudFlare. The total cost to Globalsign if they were using AWS’s infrastructure, would be at least \$952,992.40/month. Undoubtedly they’d give some additional discounts above the pricing they list publicly, but any way you slice it the costs are significant.

Beyond the cost, many CAs are not setup to be able to handle this increased load. Revoking SSL certificates threatens to create a sort of denial of service attack on their own infrastructures. Thankfully, CloudFlare helps power Globalsign's OCSP and CRL infrastructure. We were able to bear the brunt of the load, allowing us to move forward with revocation as quickly as we did. And, no, we didn’t charge them anything extra.

So, if you're wondering why some people are dragging their feet on mass certificate revocation, now you know why — it imposes a real cost. And if you're a CA who's wondering what you're going to do when you inevitably have to revoke all the certs you've issued over the last year, we're happy to help.

A little over a month ago, we published a couple of blog posts about how we were making SSL faster. Specifically, we enabled OCSP stapling across our network. In brief, when you visit a page over HTTPS, your browser checks to see if the SSL certificate is still valid via a protocol called OCSP. Those checks can be slow so we took multiple steps to make them faster. The net effect was that, for browsers that supported OCSP stapling, visitors to HTTPS sites on CloudFlare would see about a 30% performance increase on their SSL handshakes.

That was the good news. What happened next was a number of people checked our SSL setup to validate our claims. While we had increased performance around OCSP checks, these investigations turned up a number of ways in which we weren't optimally deploying SSL. In particular, William Chan wrote a blog post looking at our SSL deployment and suggesting a number of things we could do to make it better.

We took this criticism to heart and today released an improved SSL process. Our goal is to provide the fastest, strongest SSL with the most ubiquitous browser support. Since a number of other cloud service providers are likely to face the same challenges, and since we haven't found anyone else that was automatically optimizing certificate bundles intelligently, we wanted to document what we did.

In the simple case, SSL is easy. When your browser connects to a website over HTTPS, the site's web server returns a SSL certificate back to the browser. This certificate is used to verify the identity of the website and encrypt data exchanged between the browser and the web server.

In order to perform these functions, the certificate has to be trusted by the browser. A website's SSL certificate is issued by a Certificate Authority (CA). CAs vouch for a website's certificate as being valid. The CAs then have what are known as root certificates that are trusted by the browser. Since the CA's root is trusted by the browser, and since the CA trusts the web server's SSL certificate, by the transitive property the browser trusts the SSL certificate and a secure HTTPS connection can be established.

The challenge is that the world of SSL isn't always that clean. To begin, most CAs don't use their root certificate to directly sign the SSL certificates they issue to clients. Instead they use "intermediate" certificates. The chain of intermediate certificates can be of any arbitrary length, strung together to pass trust from the root to the eventual final certificate used by the web server. More complicated still, one intermediate certificate can have multiple parent certificates that vouch for its identity. GoDaddy, for example, has a case where their own root certificate as well as Valicert's root certificate both vouch for a GoDaddy intermediate certificate.

While trust can pass along any length of SSL chain, the longer the chain is the more of a performance impact there is on setting up a HTTPS connection. While some of this overhead comes from validating all the certs in the chain, much also has to do with just having to transmit all the data that makes up the intermediate certificates.

As William Chan pointed out, CloudFlare was including more than we needed to and thereby overflowing the maximum amount of data per packet. This can have an especially large impact on web performance since SSL data is the first thing to be exchanged, no more data can be exchanged until after the SSL handshake takes place, so getting rid of a round trip on the SSL handshake can speed up everything else down the line.

For SSL certificates we issued ourselves (such as those we create for Pro customers), this wasn't a problem. However, for custom SSL certificates, like those available for Business and Enterprise subscribers, we were not being smart about what we were including in the SSL bundle. The lowest hanging fruit in terms of reducing the size of these certificates was to remove the root certificates from the certificate bundle. There's no reason to include these since they should already be present in browsers and, even if they're not, the browser won't trust them.

We wanted to be even smarter about how we build bundles, so we spent some time developing a system that would find the shortest path between a certificate a user uploads to our system and one of the root certificates present in browsers. To do this, we needed to build a directory of the web's most common intermediate certificates. You'd think that's something someone would have assembled and published. We searched around for quite some time to find all these and didn't find it anywhere, so we created one ourselves. (PS - So no one else has to go through this same painful exercise, we're going to be publishing the directory on GitHub in the next few days and will keep it updated as we find more intermediate certificates.)

Today, when someone uploads a custom SSL certificate, we use our directory of intermediate certificates to build all the possible chains from the uploaded cert to a trusted browser root. We then rank these chains based on a number of factors including:

1. The length of the certificate chain

2. The ubiquity of the root certificate in browsers and other clients

3. The security of each step in the chain (e.g., does their Extended Key Usage include Server Authentication)

4. The length of the validity period of all the steps in the chain

The result is a server bundle that is small, fast and strong while having ubiquitous browser and client support. These are all optimizations that organizations concerned with performance and security should be doing by hand. What we're excited about is that we've automated this process and made it easy for anyone who wants the fastest possible SSL for their given certificate. If you were already using SSL through CloudFlare, your SSL bundle has been automatically optimized. If your site seemed a bit faster, that's why.

Going forward, in addition to releasing the directory of intermediate SSL certificates on Github, we plan on releasing our SSL bundler as a free service so you can package up your SSL certificates as efficiently as possible, even if you're not using CloudFlare. Just one more way we're working to make the web fast and safe.

Earlier this week we announced how CloudFlare enabled OCSP stapling in order to improve our customers' SSL performance. OCSP stapling is awesome and improves SSL performance by as much as 30%. However, it is limited to browsers that support OCSP stapling and only benefitsCloudFlare's customers. So, until every browser vendor updates to support OCSP stapling and until every website uses CloudFlare, we wantedto see if we could do something else to improve SSL performance across the web.

CloudFlare has worked with GlobalSign since we first launched in September 2010. Prior to that we surveyed nearly every certificate authority in an effort to find one that was forward thinking enough to support what we needed. GlobalSign has been a terrific partner and is shaking up what has been a commodity industry.

Several months ago, GlobalSign approached us to talk about SSL performance. Their goal was simple: become the fastest SSL provider on the Internet. As I've written about before, whenever you visit a website over a HTTPS connection your browser has to perform a check to see if the certificate has been revoked. Depending on your browser, these checks are either over the CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol) protocol. In either case, they require a request be sent back to the certificate authority and to get a response before content is downloaded. In other words, CRL and OCSP requests inherently slow down HTTPS performance.

The amount that these checks slow down performance varies depending on the certificate authority. On average, across the industry, a typical OCSP or CRL response time can be 500ms. That's half a second. In other words, every time you visit a site over HTTPS, you waste half a second waiting for the SSL check to complete. Talking with GlobalSign we realized we could do something about that.

This morning we officially announced our work with GlobalSign to make their CRL and OCSP requests the fastest on the Internet. GlobalSign's SSL checks (OCSP and CRL GET and POST requests) are now served from our cache across CloudFlare's global infrastructure. The results have been awesome. The requests that previously averaging around 500ms are now under 100ms. At GlobalSign's scale, that means we're now saving the web about a year and a half of time every day that people would have otherwise spent waiting for web pages to load. That's crazy.

This improvement accrues to sites using GlobalSign SSL certificates, regardless of whether the sites themselves are running on CloudFlare's network. Getting more sites using SSL is critical for increasing web security and promoting new performance protocols like SPDY. If you are choosing a CA, typically a commodity decision, now there's a good reason to pick GlobalSign over the other choices: they will ensure your site is as fast as possible over HTTPS. Put simply, GlobalSign is now the fastest certificate authority in the world, and nearly 3x as fast as Symantec/Verisign.

CloudFlare's mission is to power a faster, safer Internet so working with GlobalSign to make SSL as fast as possible has been a perfect fit. Our hope is that other certificate authorities will follow GlobalSign's lead and spend the time to optimize their SSL checks for optimal performance. As an added bonus, we've also helped GlobalSign be the first certificate authority to have their SSL checks be available over IPv6. This is all part of our efforts to help build a better Internet. As we like to tweet: #savetheweb.

CC BY-SA 3.0 image by Yathin sk

This week CloudFlare is announcing several things we're doing to significantly improve the performance of SSL. Too few sites are secured with SSL. One of the reasons sites don't implement SSL is that it can slow down web performance. One of the less frequently discussed, but most significant, performance hits to SSL is the OCSP/CRL check. These checks make up 30% or more of the HTTPS overhead. That's painful.

The best solution to speed up OCSP/CRL performance is something called OCSP Stapling. CloudFlare is committed to making the Internet faster and safer so we just enabled OCSP Stapling network wide in order to speed up all HTTPS connectionsand making the decision to secure a site with SSL a no-brainer. So what is the OCSP/CRL check? Why does it slow down page loads so significantly? And how have we eliminated this performance tax on HTTP connections with OCSP Stapling? Read on to find out.

To support secure web connections over HTTPS a website must have a SSL certificate. SSL certificates are issued by what is known as a Certificate Authority (CA). SSL certificates are issued for a period of time during which they will be trusted by browsers. If, however, a SSL certificate is stolen or compromised in some way before it expires, sites need a way to revoke the certificate so it will no longer be trusted.

The OCSP and CRL are the two protocols used to revoke certificates. CRL, which stands for Certificate Revocation List, is the older and cruder of the two protocols. When a CA receives a CRL request from a browser, it returns a complete list of all the certificates that CA manages that have been revoked. The browser then needs to parse the list and determine if the certificate of the visited site has been revoked.

With OCSP, the browser sends the certificate for the site in question to the CA. The CA then returns good, revoked, or unknown for the particular certificate. OCSP is generally preferable because less data needs to be sent and there's less overhead from the browser having to parse the CRL response. While every browser handles the revocation check process differently, generally modern browsers prefer OCSP to CRL checks.

Regardless of whether the browser performs an OCSP or CRL check, the check adds significant overhead. To give you a sense, the following connection flow is taken from this post on SSL overhead on mobile devices:

1. DNS (1334ms)

2. TCP handshake (240ms)

3. SSL handshake (376ms)

4. Follow certificate chain (1011ms)

5. DNS to CA (300ms)

6. TCP to CA (407ms)

7. OCSP to CA #1 (598ms)

8. TCP to CA #2 (317ms)

9. OCSP to CA #2 (444ms)

10. Finish SSL handshake (1270ms)

The red portions in the list above (steps 5 - 9) represent the overhead required for the revocation check requests. Add up the time for each step and you'll see that over 30% of the SSL overhead comes from checking whether the certificate has been revoked. And, unfortunately, this check is not done in parallel. In most browsers, until the revocation check is complete, the browser won't begin downloading any additional content. In other words, the OCSP check is blocking on content delivery and inherently adds a significant amount of time to the request. Painful.

The key to speeding up OCSP is to get rid of the requests that go back to the CA. Rather than needing to request the OCSP response from the CA directly, the OCSP response can be included in the initial SSL handshake (step 3 in the example above). In this sense, the OCSP response is "stapled" to the initial SSL handshake. While it seems like this approach would be less secure, the response is signed by the CA's root certificate so the browser can verify its authenticity even if it is not delivered directly from the CA's OCSP server.

While OCSP Stapling makes a ton of sense, it unfortunately hasn't previously been widely supported by web servers. Part of the problem is that it often requires a significant technical investment by web administrators. While that investment may not make sense for many individual sites, CloudFlare sits in a unique position to enable OCSP Stapling for a large number of sites in one fell swoop. So that's what we just did.

At CloudFlare, our mission is to make the web faster and more secure. Inherent to this mission is eliminating the performance penalty of SSL connections so as many sites as possible will support secure HTTPS connections. Our SSL performance was already best of class, now it's even faster. If you're already a CloudFlare customer with SSL enabled, your HTTPS performance is now about 30% faster than it was last week. If you're not yet a CloudFlare customer but you want to make sure your SSL performance it as fast as possible, it only takes about 5 minutes to sign up.

Stay tuned this week for more announcements on how we're helping improve SSL performance for the whole web.

The solution to problems like Ashton's is to use an encrypted HTTPS connection which requires a website to support SSL. The problem is that the SSL protocol imposes a heavy burden. I'm fairly technically savvy and still find the process of installing and maintaining SSL keys on a web server Byzantine. SSL sessions also add considerable CPU load to web servers and slow down web performance.

CloudFlare is the web performance and security company. For security, we believe it is incumbent on us to make SSL easier and more widely deployed. Given that, in order to maximize performance, we are spending significant resources in order to improve the performance of SSL.

SSL imposes an additional tax on web performance, but it's important to understand exactly how. The primary source of the SSL performance tax comes from the initial setup of a new connection. Before any web page data can be exchanged, a SSL session must be "negotiated" between the client and the server. A simplified version of this process is as follows.

If you have multiple resources on your web page under different domains, your visitors' browsers will need to negotiate an SSL session for each domain. Because each SSL session requires at least 4 additional exchanges, the problems of flakey connections with high loss rates (e.g., connections to mobile devices) are amplified over HTTPS.

SSL certificates are issued by a certificate authority (CA). The CA attests that the website behind the certificate is who they say they are. Sometimes, after a certificate is issued, it is lost or compromised and needs to be revoked. CAs maintain a list of certificates that have been revoked, known as a Certificate Revocation List (CRL). Browsers, when they access a SSL-protected site, can query to download a CA's CRL and parse the list to see if a particular site is on it. Alternatively, they can issue a request via the Online Certificate Status Protocol (OCSP) to check if a particular site has had its certificate revoked.

The problem is, for most CAs CRL and OCSP performance appears to be an afterthought. The CRL and OCSP servers are also not typically geographically distributed or tuned for performance. Verisign, one of the largest CAs, averages nearly 300ms to respond to a OCSP request. That inherently means an additional third of a second gets added to your page load time for any visitor connecting to your site if you're using Verisign's SSL. To add insult to injury, most CAs don't yet accept IPv6 requests to their CRL or OCSP servers.

The problem of CRL and OCSP slowness is a significant enough impact on SSL performance that browser vendors have begun to discuss removing the verifications entirely. While that may help performance, it would hurt overall web security and make it harder to reliably trust a site's SSL.

A number of other factors further contribute to SSL's overall slowness. Often certificates are chained using a number of intermediate certificates, which can increase the amount of data that needs to be exchanged during the initial session negotiation. Most sites haven't taken the time to optimize their crypto cypher. Choosing a crypto cypher that is too weak can subject your site to a number of potential vulnerabilities. On the other hand, choosing a stronger cypher can add load if your site is already CPU bound.

Overall, we regularly hear from customers who want to have SSL protection on their sites but find it either too difficult or to resource intensive. The end result is that many sites that should use SSL don't, sacrificing security for performance. And, for those sites that do support SSL, it is often implemented in a way that causes a significant performance penalty. Neither case is good for the web, so we wanted to see what we could do to help fix the problem.

CloudFlare is working on a number of initiatives to speed up SSL. First, we have a number of techniques to limit the number of connections that a browser needs to make and therefore reduce the number of session initiation handshakes that need be made. Rocket Loader, for example, combines requests for scripts, even across third party services, into a single connection under a site's own domain. We're also rolling out support for SPDY for all SSL-enabled sites, which allows requests to be multiplexed across a single connection.

Beyond reducing the number of connections to improve SSL performance, we're working with Globalsign, our primary CA, to speed up CRL and OCSP. Today, Globalsign's CRL and OCSP requests are powered through CloudFlare's global network and response times are under 100ms. We are also planning on rolling out OCSP stapling, which allows the query for whether a certificate is invalid to be sent from the server without making an additional request to the CA's infrastructure.

CloudFlare runs all its own infrastructure, which allowed us to spec the hardware for maximum SSL performance. We worked with Intel to integrate a custom tuned version of OpenSSL which takes advantage of special instruction sets on our servers' CPUs to make encryption and decryption as fast as possible. In our tests, it is about 30% faster and uses fewer CPU resources. This allows us to choose stronger cyphers that maximize performance without suffering a performance penalty.

Equally important, we've worked to make SSL as easy as possible, regardless of the capabilities of your own infrastructure. SSL is included automatically for every paid CloudFlare plan and is provisioned automatically with a single click. We allow customers to choose Full SSL, which ensures end-to-end encryption, or Flexible SSL, which encrypts the connection from the browser to CloudFlare's network, the riskiest part of the transaction, but doesn't require you to run SSL on your origin server. This allows you to add SSL to services like Google AppEngine and Tumblr, which don't allow SSL support themselves or only include it as an expensive add on.

While CloudFlare's Pro accounts include SSL certificates issued by our own systems, with the launch of our Business and Enterprise tiers, customers can now upload their own certificates to use on our network. This allows support of extended validation (EV) certificates on our optimized infrastructure.

As new performance technologies like SPDY require SSL support, and as new threats continue to emerge online, ensuring SSL runs as fast and secure as possible becomes increasingly important. CloudFlare will continue to work to make SSL performance faster in order to fulfill our mission of building a web that is both safer and faster.