CAA of the Wild: Supporting a New Standard

One thing we take pride in at Cloudflare is embracing new protocols and standards that help make the Internet faster and safer. Sometimes this means that we’ll launch support for experimental features or standards still under active development, as we did with TLS 1.3. Due to the not-quite-final nature of some of these features, we limit the availability at the onset to only the most ardent users so we can observe how these cutting-edge features behave in the wild. Some of our observations have helped the community propose revisions to the corresponding RFCs.

We began supporting the DNS Certification Authority Authorization (CAA) Resource Record in June behind a beta flag. Our goal in doing so was to see how the presence of these records would affect SSL certificate issuance by publicly-trusted certification authorities. We also wanted to do so in advance of the 8 September 2017 enforcement date for mandatory CAA checking at certificate issuance time, without introducing a new and externally unproven behavior to millions of Cloudflare customers at once. This beta period has provided invaluable insight as to how CAA records have changed and will continue to change the commercial public-key infrastructure (PKI) ecosystem.

As of today, we’ve removed this beta flag and all users are welcome to add CAA records as they see fit—without having to first contact support. Note that if you’ve got Universal SSL enabled, we’ll automatically augment your CAA records to allow issuance from our CA partners; if you’d like to disable Universal SSL and provide your own certificates, you’re welcome to do that too.Below are some additional details on CAA, the purpose of this record type, and how its use has evolved since it was first introduced. If you’d rather just jump to the details of our implementation, click here and we’ll take you to the relevant section of the post.

The Publicly-Trusted PKI Ecosystem — Abridged

Before diving into CAA it’s helpful to understand the purpose of a public key infrastructure (PKI). Quite simply, PKI is a framework that’s used to secure communications between parties over an insecure public network. In “web PKI”, the PKI system that’s used to secure communications between your web browser and this blog (for example), the TLS protocol is used with SSL certificates and private keys to protect against eavesdropping and tampering.

While TLS handles the sanctity of the connection, ensuring that nobody can snoop on or mess with HTTPS requests, how does your browser know it’s talking to the actual owner of blog.cloudflare.com and not some imposter? Anyone with access to OpenSSL or a similar tool can generate a certificate purporting to be valid for this hostname but fortunately your browser only trusts certificates issued (or “signed by”) by certain well-known parties.

These well-known parties are known as certification authorities (CAs). The private and public key that form the certificate for blog.cloudflare.com were generated on Cloudflare hardware, but the stamp of approval—the signature—was placed on the certificate by a CA. When your browser receives this “leaf” certificate, it follows the issuer all the way to a “root” that it trusts, validating the signatures along the way and deciding whether to accept the certificate as valid for the requested hostname.

Before placing this stamp of approval, CAs are supposed to take steps to ensure that the certificate requester can demonstrate control over the hostname. (In some cases, as you’ll learn below, this is not always the case, and is one of the reasons that CAA was introduced.)

Anthropogenic Threats

Given that people are imperfect beings and prone to making mistakes or poor judgement calls, it should come to the surprise of no one that the PKI ecosystem has a fairly blemished track-record when it comes to maintaining trust. Clients, CAs, servers, and certificate requesters are all created or operated by people who have made mistakes.

Jurassic Park. 1993, Stephen Spielberg [Film] Universal Pictures.

Client providers have been known to add compromising certificates to the local trust store or install software to intercept secure connections; servers have been demonstrated to leak private keys or be unable to properly rotate session ticket keys; CAs have knowingly mis-issued certificates or failed to validate hostname ownership or control reliably; and individuals requesting certificates include phishers creating convincing imposter versions of popular domains and obtaining valid and trusted certificates. Every party in this ecosystem is not completely without blame in contributing to a diminished sense of trust.

Of these many problems (most of which have already been addressed or are in the process of being resolved), perhaps the most unsettling is the willful mis-issuance of certificates by trusted CAs. Knowingly issuing certificates to parties who haven't demonstrated ownership, or issuance outside the parameters defined by the CA/Browser Forum (a voluntary and democratic governing body for publicly-trusted certificates) by CAs who have certificates present in trust stores severely undermines the value of that trust store, all certificates issued by that CA, and the publicly-trusted PKI ecosystem as a whole.

Solving One Problem...

To help prevent future mis-issuance by publicly trusted CAs, a new DNS resource record was proposed by those CAs to help reduce the risk of certificate mis-issuance: The Certification Authority Authorization (CAA) Resource Record.

The general idea is that the owner of any given domain (e.g., example.com) would add CAA records at their authoritative DNS provider, specifying one or more CAs who are authorized to issue certificates for their domain.

RFC6844 currently specifies three property tags for CAA records: issue, issuewild, and iodef.

• The issue property tag specifies CAs who are authorized to issue certificates for a domain. For example, the record example.com. CAA 0 issue "certification-authority.net" allows the "Certification Authority" CA to issue certificates for example.com.

• The issuewild property tag specifies CAs that are only allowed to issue certificates that specify a wildcard domain. E.g., the record example.com. CAA 0 issuewild "certification-authority.net" only allows the "Certification Authority" CA to issue certificates containing wildcard domains, such as *.example.com.

• The iodef property tag specifies a means of reporting certificate issue requests or cases of certificate issuance for the corresponding domain that violate the security policy of the issuer or the domain name holder. E.g., the record example.com. CAA 0 iodef "mailto:example@example.com" instructs the issuing CA to send violation reports via email to the address provided at the attempted time of issuance.

CAA records with the issue and issuewild tags are additive; if more than one are returned as the response for a DNS query for a given hostname, the CAs specified in both records are both considered authorized.

If the authoritative DNS provider does not yet support CAA records or none are present in the zone file, the issuing CA is still authorized to issue when no records are present, largely preserving the issuance behavior before CAA records were an adopted standard.

As of 8 September 2017, all publicly-trusted CAs are now required to check CAA at issuance time for all certificates issued, thereby enabling certificate requestors (domain owners) to dictate which CAs can issue certificates for their domain.

... with More Problems.

RFC6844 specifies a very curious CAA record processing algorithm:

The search for a CAA record climbs the DNS name tree from the
   specified label up to but not including the DNS root '.'.

   Given a request for a specific domain X, or a request for a wildcard
   domain *.X, the relevant record set R(X) is determined as follows:

   Let CAA(X) be the record set returned in response to performing a CAA
   record query on the label X, P(X) be the DNS label immediately above
   X in the DNS hierarchy, and A(X) be the target of a CNAME or DNAME
   alias record specified at the label X.

   o  If CAA(X) is not empty, R(X) = CAA(X), otherwise

   o  If A(X) is not null, and R(A(X)) is not empty, then R(X) =
      R(A(X)), otherwise

   o  If X is not a top-level domain, then R(X) = R(P(X)), otherwise

   o  R(X) is empty.

While the above algorithm is not easily understood at first, the example immediately following it is much easier to comprehend:

For example, if a certificate is requested for X.Y.Z the issuer will
   search for the relevant CAA record set in the following order:

      X.Y.Z

      Alias (X.Y.Z)

      Y.Z

      Alias (Y.Z)

      Z

      Alias (Z)

      Return Empty

In plain English, this means that if the owner of example.com requests a certificate for test.blog.example.com, the issuing CA must

1. Query for a CAA record at test.blog.example.com.. If a CAA record exists for this hostname, the issuing CA stops checking for CAA records and issues accordingly. If no CAA record exists for this hostname and this hostname exists as an A or AAAA record, the CA then moves up the DNS tree to the next highest label.

2. Query for a CAA record at blog.example.com.. Just like the first check, if no CAA record exists for this hostname and this hostname exists as an A or AAAA record, the CA then continues traversing the DNS tree.

3. Query for a CAA record at example.com.

4. Query for a CAA record at com.

At the end of the last step, the issuing CA has climbed the entire DNS tree (excluding the root) checking for CAA records. This functionality allows a domain owner to create CAA records at the root of their domain and have those records apply to any and all subdomains.

However, the CAA record processing algorithm has an additional check if the hostname exists as a CNAME (or DNAME) record. In this case, the issuing CA must also check the target of the CNAME record. Revisiting the example above for test.blog.example.com. where this hostname exists as a CNAME record, the issuing CA must

1. Query for a CAA record at test.blog.example.com.. Since test.blog.example.com. exists as the CNAME test.blog.example.com. CNAME test.blog-provider.net., the issuing CA must next check the target for the presence of a CAA record before climbing the DNS tree.

2. Query for a CAA record at test.blog-provider.net.

The issuing CA in this example is only at step two in the CAA processing algorithm and it has already come across two separate issues.

First, the issuing CA has checked the hostname requested on the certificate (test.blog.example.com.) and since that hostname exists as a CNAME record, it has also checked the target of that record (test.blog-provider.net.). However, if test.blog-provider.net. itself is also a CNAME record, the the CAA record processing algorithm states that the issuing CA must check the target of that CNAME as well.

In this case it is fairly simple to create a CNAME loop (or very long CNAME chain) either via an accidental misconfiguration or with malicious intent to prevent the issuing CA from completing the CAA check.

Second, example.com and blog-provider.net might not be owned and operated by the same entity or even exist in the same network. The RFC authors appear to be operating under the assumption that CNAME records are still used as they were in November 1987:

A CNAME RR identifies its owner name as an alias, and specifies the corresponding canonical name in the RDATA section of the RR.

This behavior may have been true thirty years ago, but consider the number and prevalence of SaaS providers in 2017; who truly is authoritative for the content addressed by a DNS record, the content creator (and likely domain name owner who is using a CNAME) or the SaaS provider to whom the content creator subscribes?

Many if not all major service or application providers include clauses in their respective terms of service with regards to content added by the user to their account for the service. Intellectual property must be user-generated, properly licensed, or otherwise lawfully obtained. As content uploaded by users is difficult to control, liability limitations, indemnification, and service termination are all commonly used when the intellectual property added to the service is owned by another party, is unlawful, or is outside the definition of acceptable use. From a content perspective within the scope of a service provider's terms of service, it’s difficult to consider the service provider canonical for the content hosted there.

Using a real world example, there are currently 401,716 CNAME records in the zone files for domains on Cloudflare whose target is ghs.google.com. This hostname is given to subscribers of Google's Blogger service to use as the target of a CNAME so that a Blogger subscriber may use their own domain name in front of Google's service. With one CAA record, Google could dictate that nearly half a million blogs with a vanity domain name can only have one CA or no CAs issue certificates unless each of those hostnames created their own CAA records to allow issuance.

Even without following CNAMEs to their targets, the DNS climbing algorithm is not unproblematic. Operators of top-level domains may decide that only certain issuing CAs are trustworthy or certain CAs are advantageous for business and create CAA records only allowing those CAs to issue. We already see this in action today with the pseudo-top level domain nl.eu.org which has the CAA record nl.eu.org. CAA 0 issue "letsencrypt.org", which only allows issuance through Let's Encrypt without CAA records being present at the subdomains of nl.eu.org.

The authors of RFC6844 were also unwilling to secure their record and its use from any potential on-path attacks—

Use of DNSSEC to authenticate CAA RRs is strongly RECOMMENDED but not required.

Without DNSSEC, DNS responses are returned to the requestor in plain-text. Anyone in a privileged network position (a recursive DNS provider or ISP) could alter the response to a CAA query to allow or deny issuance as desired. As only about 820,000 .com domain names out of the more than 130 million registered .com domain names are secured with DNSSEC, perhaps the low adoption rates influenced the decision to not make DNSSEC with CAA mandatory.

Moving beyond the RFC, the CA/Browser Forum's Baseline Requirements (BR) attempt to clarify aspects regarding the behavior a CA should take when checking for CAA.

CAs are permitted to treat a record lookup failure as permission to issue if:

• the failure is outside the CA's infrastructure;

• the lookup has been retried at least once; and

• the domain's zone does not have a DNSSEC validation chain to the ICANN root

Effectively, this means that if the CAA response is either a SERVFAIL, REFUSED, or the query times out, regardless of whether or not a CAA record exists, the CA is permitted to issue if this query fails more than once while attempting to issue a certificate. However, multiple CAs have told us that DNS lookup failures will prevent issuance regardless of the above conditions in the BR. In this case, something as benign as a transient network error could result in a denial of issuance or worse, any recursor that doesn’t understand and a query for CAA record will prevent issuance. We actually saw this with Comcast’s resolvers and reported the bug to their DNS provider.

There’s an additional security gap in that neither the RFC nor the BR indicate where the issuing CA should query for CAA records. It is acceptable within the current standards to query any DNS recursor for these records as well as the authoritative DNS provider for a domain. For example, an issuing CA could query Google’s Public DNS or a DNS recursor provided by their ISP for these responses. This enables compromised DNS recursors or one run by a rogue operator to alter these responses, either denying issuance or allowing issuance by a CA not approved by the domain owner. The RFC and BR should be amended so that an issuing CA must always query these records at the authoritative provider to close this gap.

CAA and Cloudflare

As of today, CAA records are no longer in beta and all customers are able to add CAA records for their zones. This can be done in the DNS tab of the Cloudflare dashboard or via our API.

When creating CAA records in the dashboard, an additional modal appears to help clarify the different CAA tag options and format the record correctly as an incorrectly formatted record would result in every CA not being able to issue a certificate.

Cloudflare is in a unique position to be able to complete SSL validation for a domain and have certificates issued on the domain owner's behalf. This is how Cloudflare is able to automatically provision and have our Universal SSL certificates issued for every domain active on Cloudflare for free.

Cloudflare partners with multiple CAs to issue certificates for our managed SSL products: Universal SSL, Dedicated Certificates, and SSL for SaaS. Since CAA record checking is now mandatory for publicly-trusted CAs, Cloudflare automatically adds the requisite CAA records to the zone when a user adds one or more CAA records in the Cloudflare dashboard for their domain in order to allow our partner CAs to continue to be able to issue certificates for each of our SSL products.

Some site owners may want to manage their own SSL certificates in order to be compliant with their own standard operating procedures or policies. Alternately, domain owners may only want to trust specific CAs that are not the CAs Cloudflare currently partners with to issue Universal SSL certificates. In the latter case, users now have the ability to disable Universal SSL.

When Universal SSL is disabled, the CAA records added to allow our partner CAs to issue certificates are deleted and all Universal SSL certificates available for the zone are removed from our edge. Dedicated certificates, custom certificates, as well as SSL for SaaS certificates are all individually managed by each customer and can be added or removed as needed.

A Bright and Hopefully Not-Too-Distant Future

CAA as it exists today does very little to reduce the attack surface around certificate issuance while making it more difficult for well-intended parties to participate. That said, CAA is a young standard recently adopted by the web PKI community with many involved individuals as well as active parties working toward addressing some of the gaps present in the current RFC and committed to making the overall CAA experience better for both the certificate requesters and CAs.

To start, an errata report exists to clarify the CAA record processing algorithm and to reduce the degree in which the targets of CNAME records should be checked. Similarly, the DNS tree-climbing behavior of the CAA record processing algorithm is still up for debate. There are also active discussions around implementation issues, such as the recognition that some authoritative DNS resolvers incorrectly sign empty responses to CAA queries when DNSSEC is enabled for a zone and how to handle these cases in a way that would still allow CAs to issue. Proposals exist suggesting new tags or property fields in CAA records, such as a CA requiring an account number in the property value for issue tags, or only allowing specific validation types (e.g., DV, OV, or EV). The Electronic Frontier Foundation (EFF) has expressed interest in hardening CAs against non-cryptographic attacks, particularly with a focus on the domain validation process for obtaining certificates. While not directly pertaining to CAA, any ideas or proposed hardening solutions might increase reliance on CAA or completely obviate the need for it altogether.

CC BY-NC 2.5 by xkcd.com

As with all internet standards, none are perfect and all are far from permanent—CAA included. Being in the position to implement new standards at scale, seeing what effect adoption of those standards has, and working with the Internet community to address any issues or gaps is a privilege and allows us to live up to our mission of building a better Internet.

We're thrilled to be involved in efforts to make CAA (and any other standard) better for anyone and everyone.