A tour through Merkle Town, Cloudflare's Certificate Transparency dashboard

For a quick primer on Certificate Transparency, please read my colleague Nick Sullivan’s post from earlier today. The discussion below expands on that post and details how Cloudflare monitors the health and performance of Certificate Transparency (CT) logs.

The success of Certificate Transparency rests on the existence of a robust ecosystem of logs and log operators. Without logs that CAs can depend on, it’s not practical for browsers to require that SSL certificates have been logged to be trusted—as Chrome plans to do on April 30. With this deadline fast approaching and others browsers likely to follow suit, it’s critical that the CT ecosystem continues to strengthen and expand with new log operators.

As we wrote about earlier today, Cloudflare recently joined this group of trusted log operators, helping strengthen this critical ecosystem. Now, we’d like to take you on a quick guide through our new publicly accessible tool that tracks the health of all trusted logs. In addition to basic uptime and response times, Merkle Town provides statistics on the type and frequency of certificates issued, the top issuers, and the inter-dependencies CAs have on existing logs and each other. Click here to jump right into our CT ecosystem dashboard, or continue reading for a quick overview of what we’ve built.

The dashboard is divided into two pages: the list of monitored logs and analyses of certificates found while monitoring. The latter can be grouped into the following sections:

• Certificate Breakdown by Type

• Root Certificate Authorities

• Global Details

• Legacy Algorithms

• Issuance Per Day By Certificate Authority

• Log Utilization by Large Certificate Authorities

On this page we list all of the logs that we’re actively monitoring. The set is currently comprised of all logs trusted by Chrome, with one exception: WoSign has recently been distrusted and will be removed soon. We also plan to expand our list once Mozilla, Apple, and others (finally) codify their CT policies.

For each log listed, we fetch the current Signed Tree Head (STH) and set of trusted roots once per minute. New entries that we haven’t seen yet are crawled, and we periodically submit strictly-valid unexpired certificates or pre-certificates to each log (unless the log is sharded by year, in which case we submit a certificate expiring in that year regardless of expiration status).

All requests to a log count towards our estimate of its uptime and response time. A specific endpoint's uptime, e.g., get-sth or get-entries, is computed as the percentage of requests to that endpoint that succeed—return a 200 status code with a valid response—in a 90 day rolling window. An endpoint's response time is computed as the average amount of time it takes to fully receive and parse a response after sending the request (again over a 90 day rolling window). Putting these together, we can compute the log's uptime and response times by taking the fair average of the uptime and response times for all monitored endpoints.

Note that some parts of the monitor may enforce different timeouts or will retry in the event that a request fails, while others will not. For example, get-roots and get-sth will start retrying more often if there's a failure, but if add-chain fails that doesn't change when the next add-chain will get sent. This behavior means that downtime may be penalized unfairly relative to a more intuitive interpretation of “downtime”. Lastly, the response time of the get-entries endpoint is strongly correlated with the number of entries it returns, but this information is currently ignored by our monitor.

On this section of the dashboard we group all certificates found across CT logs we monitor into the following categories: Validation Level; Public Key Algorithm; Signature Algorithm; Entry Type; and Expired v. Current. Selecting one slice of a chart will filter the other charts by this selection, as well as the Root Certificate Authorities section below the pie charts.

In the example shown above, I start by clicking “Other” in the Validation Level chart to discard the 315 million DV certs found. Then, I click on “Extended” to show only EV certificates and “Current” in the Expired v. Current chart to filter out expired certificates. From here I hover over the Root Certificate Authorities horizontal bar chart to see the distribution of unexpired EVs across roots. Note that we don’t (yet) group roots by owner, e.g., GeoTrust is listed separately rather than part of DigiCert, who purchased the brand along with the rest of Symantec’s CA business.

EV was the first type of certificate that was required to have been logged in CT. Chrome enforced this starting in 2015 and any EV certificates that are not logged do not receive the semi-standardized treatment of showing the organization’s legal name and country.

Another interesting observation from Merkle Town, as observed the day this post was published, is that RSA represents 93% of all certificates in our database but only 91% of unexpired certificates. This data indicates increased adoption of elliptic curve cryptography, a technology that uses smaller keys than RSA (at equivalent cryptographic strength), resulting in less load on the terminating server and fewer bytes sent over the wire during the TLS handshake.

Here we break down certificate count based on the root to which it chains. The animation below has been filtered to show only current certificates, across all validation types. Note also that the roots displayed have not been grouped by owner.

As can be seen as I continuously drill into the “Other” category, there is a long tail of browser-trusted CAs with very few unexpired certificates. Certificate Transparency and policy enforcement does not discriminate based on issuer size: soon any certificate issued worldwide must be logged or risk not being trusted.

This section is quite simple, but helps put into perspective the scale at which web PKI operates. As captured on the day this post was written, we’re observing 33,906 certificates issued and 28,955 expiring per hour.

Remarkably, roughly three-quarters of this issuance can be attributed to Let’s Encrypt. Now that they offer wildcard certificates, it will be interesting to observe how this rate changes: wildcards expand the universe of users they can serve, but decreases the number of certificates required per domain.

In the future we plan to offer the ability to drill down into categories to view specific certificates, especially those using deprecated algorithms. For now, you can see counts of still-valid certificates signed using deprecated algorithms such as RSA-MD2 (4), RSA-MD5 (22), and RSA-SHA1 (744,732).

Globally there are upwards of 1,000,000 certificates issued per day. As previously mentioned, Let’s Encrypt represents approximately 75-80% of this issuance—except for days when Cloudflare accelerates issuance through DigiCert. On these days, as shown below in the two rightmost-bars, Let’s Encrypt represents 70%.

Chrome’s upcoming "not secure" labeling of HTTP sites has been driving intense adoption of HTTPS using Cloudflare's SSL for SaaS provider offering. Our customers are moving tens of thousands of sites per day from HTTP to HTTPS using our custom hostnames API.

As we explained earlier today, the most popular way to let browsers that know that a SSL certificate has been logged to CT is by embedding SCTs in the certificate. This process takes place before the certificate has been signed and provided to the website operator, so it is critical that it be performed reliably and quickly.

The chart below shows that CAs typically obtain their SCTs from the same set of logs, operated by other CAs. In some cases, CAs will allow other CAs to log for free, while in other cases they may charge a modest sum to offset the cost of operating the log. Cloudflare has been working with CAs to help them streamline this process, by providing APIs that (in parallel) acquire SCTs in accordance with browsers’ CT policies.

Merkle Town was built by Cloudflare Cryptography Team Engineer, Brendan McMillion. Want to work with Brendan, Nick Sullivan, and the rest of the Cloudflare Crypto team? You're in luck as they're hiring full time Cryptography Engineers and interns in San Francisco and London. Apply today and help us further strengthen encryption on the web.

Additionally, as I said at the outset of this post, Certificate Transparency relies on a robust ecosystem of logs and log operators. CT improves the security of everyone using HTTPS, so if your organization has the resources to stand up a log, free to reach out to us at ct-logs@cloudflare.com. Drop us a line and we’d be happy to point you in the right direction.