As of December 22, 2018, parts of the US Government have “shut down” because of a lapse in appropriation. The shutdown has caused the furlough of employees across the government and has affected federal contracts. An unexpected side-effect of this shutdown has been the expiration of TLS certificates on some .gov websites. This side-effect has emphasized a common issue on the Internet: the usage of expired certificates and their erosion of trust.
For an entity to provide a secure website, it needs a valid TLS certificate attached to the website server. These TLS certificates have both start dates and expiry dates. Normally certificates are renewed prior to their expiration. However, if there’s no one to execute this process, then websites serve expired certificates--a poor security practice.
This means that people looking for government information or resources may encounter alarming error messages when visiting important .gov websites:
The content of the website hasn’t changed; it’s just the cryptographic exchange that’s invalid (an expired certificate can’t be validated). These expired certificates present a trust problem. Certificate errors often dissuade people from accessing a website, and imply that the site is not to be trusted. Browsers purposefully make it difficult to continue to an insecure website by hiding the “proceed” option under an “Advanced Settings/Actions” button. In the example above, people seeking aid in the wake of a natural disaster may not be able to access government websites with crucial information.
Converse to the situation above, some Internet users may get desensitized to certificate error messages. Seeing expired certificates on otherwise trusted websites will teach users to dismiss certificate errors and bypass them even when a certificate (and website) is genuinely unsafe. Moreover, keys should be rotated on a regular basis to minimize the amount of traffic made vulnerable by a key breach. To use expired certificates is to extend the use of a public-private key pair beyond its expected lifetime, and opens up more traffic to potential snooping.
Tracking Expired .gov Certificates Using Certificate Transparency Monitors
TechCrunch recently published a list of soon-to-expire certificates for .gov domains. To create this list, they iterated over a dataset of all federal .gov domains furnished by 18F, the federal government’s digital services unit. For each .gov domain on the list, they pulled its certificate and checked its expiration date. They then filtered out the state and local government .gov domains.
Relying on 18F for this list, however, introduces a single point of failure. What if 18F’s list was not up-to-date? What if 18F was shut down? What if 18F’s list is not conclusive? (Their list actually does not include .gov subdomains). One organization alone cannot be the provider of all truth about federal .gov sites. Third-party collections of .gov certificates would bolster the thoroughness and availability of expired certificate information.
Cloudflare's Certificate Transparency (CT) monitor, Merkle Town, is one such third-party. Around the same time as TechCrunch did its research, Cloudflare used Merkle Town to find .gov certificates under imminent expiration. CT monitors are one part of the Certificate Transparency ecosystem. Certificate Transparency Logs are used to store all issued certificates on the Internet and hold Certificate Authorities accountable for the certificates they issue. This means that CT logs hold all issued .gov certificates, so one can consult them for an exhaustive list. Certificate Transparency Monitors, on the other hand, help keep the CT logs accountable as well as make their raw bulk data useful to the general public. In addition to Merkle Town, crt.sh and Cert Spotter are other examples of monitors.
The Nitty-Gritty
All the certificates that our monitor extracts from crawling CT logs are stored in an HBase table. HBase is a database similar to Google’s Bigtable, and is designed for storing large amounts of data and running MapReduce jobs. Using the MapReduce model, we wrote a small amount of code to look at each row of the database, parse the stored certificate and check if (1) it’s valid for a domain ending in “.gov” and (2) will expire in the next two months.
If (1) and (2) are true, the hostname, the name of the issuing certificate authority, and the expiration date were output.
Once the code was deployed, it took 90 minutes to scan over 1 billion unique certificates stored in all CT logs. This means that it was processing roughly 200,000 certificates per second!
The MapReduce job gave us an initial and comprehensive list. But just because a certificate was issued by a CA doesn’t mean that it’s being served. We did a second pass over the first list, this time actually contacting each domain and trying to complete a TLS handshake and observing if the old certificate was still being served. If so, the hostname was kept in the final list. If the handshake succeeded but a new certificate was being served, we discarded the hostname. If the handshake failed, the hostname was excluded and the error message was noted.
In our final dataset, we filter out .gov domains that correspond to state and local governments, as well as those federal government domains that appear to have been funded by earlier appropriations.
Our results can be found here.
Unexpected Mis-Configurations
As expected, a significant number of hostnames were excluded by the second pass because they had updated their certificates already. Another smaller number of hostnames were also excluded because those websites were unreachable or no longer operational. However, we also found many more hostnames than we expected with mis-configured TLS, even though they’re websites that seem to be for public consumption.
An example of this is https://cableplant.boulder.noaa.gov which currently fails to load with this error:
An error occurred during a connection to cableplant.boulder.noaa.gov. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG
A subtler issue we found was with https://www.indianaffairs.gov/ and https://www.volunteer.gov/. Our script was unable to validate the certificate chain for these websites, even though these websites seem to load fine in a browser. The reason is that these websites omit parts of their certificate chain which are necessary to verify that the certificate comes from a publicly trusted authority.
To improve page load times, browsers will often cache partial certificate chains on-disk. So even if a website does not send all of the necessary certificates, the browser may find what it needed in its cache, which has been well-populated by previous browsing activity. This is still just cache, though. It cannot be relied upon. In my case, after clearing my browser history, both of the websites above become inaccessible, same as for the script.
How Can Domains Stop Presenting Expired Certificates?
The presence of .gov expired certificates means that either (1) .gov certificates are manually renewed, or (2) .gov certificates cost money to renew, and the shutdown prevented spending on this important web security measure.
Automatic certificate issuance has become a standard for many domains, and services like Cloudflare offer automatic certificate renewal when you use Universal SSL or get a Cloudflare-issued certificate. CAs like Let’s Encrypt also offer automatic certificate renewal, which works as long as you run the certbot daemon on your webserver. Furthermore, automatic certificate renewal is free with both of these approaches.
Automating certificate renewals makes expired certificates and mis-configured TLS a problem of the past. We hope that this interesting blip with a few .gov certificates has encouraged domain owners to automate their certificate handling. If you haven’t automated your domain’s certificate renewal, try Universal SSL or Cloudflare certificates today!
Many thanks to Alissa Starzak for her help in filtering .gov domains for this blog post.