The core functionality of Spectrum is its ability to block large DDoS attacks. Spectrum benefits from Cloudflare’s existing DDoS mitigation (which this week blocked a 900 Gbps flood). Spectrum’s DDoS protection has already been battle tested. Just soon as we opened up Spectrum for beta, Spectrum received its first SYN flood.

One of Spectrum's earliest deployments was in front of Hypixel’s infrastructure. Hypixel runs the largest minecraft server, and because gamers can be - uh, passionate - they were one of the earliest targets of the terabit-per-second Mirai botnet. “Hypixel was one of the first subjects of the Mirai botnet DDoS attacks and frequently receives large attacks. Before Spectrum, we had to rely on unstable services and techniques that increased latency, worsening user's experience. Now, we're able to be continually protected without added latency, which makes it the best option for any latency and uptime sensitive service such as online gaming,” Bruce Blair, the CTO at Hypixel, told us.

Another early team we talked to about Spectrum was the security team at Montecito Bank & Trust. As a financial institution, they have a highly technical and active security team; they were also one of the first customers to use Cloudflare’s DNSSEC when it was brand new. Paul Abramson, Montecito Bank & Trust’s Director of Technology told us, “We were looking for a security solution to protect additional services like email and SSH so that if we are subject to attack, our operations can continue to run reliably and securely.”

Security and encryption go hand in hand. With Spectrum, you can terminate TLS at Cloudflare’s edge. The main benefit of TLS termination at the edge is that is speeds up performance (there’s less distance to travel for the three round trips of the TLS handshake).

We think the most interesting outcome is that just by adding support for TLS in the client, Cloudflare can now add encryption to legacy protocols and services that don’t traditionally support encrypted transit.

Spectrum integrates with Cloudflare’s IP Firewall so that you can choose which connections should be forwarded to your servers and which should be blocked at Cloudflare’s edge.

This can be managed via API too, so you can write scripts that allow and deny access on the fly.

curl -X POST "https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules" \
     -H "X-Auth-Email: hello@example.com" \
     -H "X-Auth-Key: 0000000000000000000" \
     -H "Content-Type: application/json" \
     --data '{"mode":"block","configuration":{"target":"ip","value":"192.0.2.1"}}'

Many TCP load balancers and proxies can be cumbersome to set up, but Spectrum takes a few clicks. Tito Esterline on our team recorded a demo you can watch below. My suggestion is to play it with audio so you can hear the play by play.

If you want to get started, get in touch with our team. Today Spectrum is available for applications on the Enterprise plan.

Why just Enterprise? While HTTP can use the Host header to identify services, TCP relies on each service having a unique IP address in order to identify it. Since IPv4 addresses are endangered, it’s quite expensive for us to delegate an IP per application and we needed to limit use. We’re actively thinking about ways to bring Spectrum to everyone. One idea is to offer IPv6-only Spectrum to non-Enterprise customers. Another idea is let anyone use Spectrum but pay for the IPv4 address. We’re not sure yet, but if you prefer one to the other, feel free to comment and let us know.

Oh and P.S. If you want to read about how Spectrum works, Marek wrote a great blog post about the Linux behavior that let us build it.

Photo from Wikimedia Commons

Today we’re introducing Argo Tunnel, a private connection between your web server and Cloudflare. Tunnel makes it so that only traffic that routes through Cloudflare can reach your server.

You can think of Argo Tunnel as a virtual P.O. box. It lets someone send you packets without knowing your real address. In other words, it’s a private link. Only Cloudflare can see the server and communicate with it, and for the rest of the internet, it’s unroutable, as if the server is not even there.

This type of private deployment used to be accomplished with GRE tunnels. But GRE tunnels are expensive and slow, they don’t really make sense in a 2018 internet.

GRE is a tunneling protocol for sending data between two servers by simulating a physical link. Configuring a GRE tunnel requires coordination between network administrators from both sides of the connection. It is an expensive service that is usually only available for large corporations with dedicated budgets. The GRE protocol encapsulates packets inside other packets, which means that you will have to either lower the MTU of your origin servers, or have your router do packet fragmentation, leading to slower responses.

We wanted to find a way to emulate the same security of a GRE tunnel but without the expense or hassle. And at the same time maybe it could speed up connections instead of slowing them down. And with that direction, the team started to build Tunnel.

Argo Tunnel is fast to install and run - it’s just three commands to expose a locally running web application:

$ install cloudflared // binaries available for Linux, Mac and Windows https://developers.cloudflare.com/argo-tunnel/downloads/
$ cloudflared login
$ cloudflared --hostname example.com http://localhost:8080

This can be run on any device from a Raspberry Pi, to a DigitalOcean droplet, to a hardware load balancer in your data center.

Netwrk is one of the companies using Argo Tunnel. Their Co-Founder and CTO Johan Bergström told us:

"I've been able to reduce the administrative overhead of firewalls, reduce the attack surface and get the added benefit of higher performance through the tunnel."

One reason why traffic through Argo Tunnel gets a performance boost is that Tunnel is built on top of Argo, Cloudflare’s optimized smart routing (think Waze for the internet).

Tunnel is included for free for anyone that has Argo enabled.

In order for Tunnel to work we needed to get visitor traffic to reach one of the data centers closest to the origin. The right way to do this is by taking advantage of Argo. We decided it made sense to bundle Tunnel with Argo and include it at no additional cost. That way you get the best of both worlds: a secure, protected origin and the fastest path across the Internet to get to it.

Of course, we want you to one day be able to test out Tunnel without having to buy Argo, so we’re considering offering a free version of Tunnel on a Cloudflare domain. If you’re interested in testing out an early version in the future, sign up here.

During the beta period, Argo Tunnel went under a different name: Warp. While we liked Warp as a name, as soon as we realized that it made sense to bundle Warp with Argo, we wanted it to be under the Argo product name. Plus, a tunnel is what the product is so it's more descriptive.

To get started, download Argo Tunnel and follow our quickstart guide. If you’re curious how it works, you can also check out the source.

ARP you going to be there? We R going to have three epoch sessions by Cloudflare speakers. Ifdown, seems apt you could SELECT to JOIN. Cat make it? Not a bg deal, wget it (though it mega hertz we won’t C you). All the audio from the three sessions will be recorded, you can listen to the cd.

WPS! I almost forgot to tel(net) you whois going to be there, and WAN and where to go.

On Friday, March 9, I’m moderating a panel with Emily Schechter from Google, Aaron DeVera from Deloitte and Gabe Kassel from eero about how Wi-Fi networks work and WEP happens when attackers coax people into joining insecure networks. It’s at Salon K in the Hilton at 3:30PM.

On Sunday the 11th, Nitin Rao is on a panel with Heather West from Mozilla, Stefan Lederer from Bitmovin and Fred Benenson from Unlimited Liability Corporation LLC about the impact of the recent revocation of Net Neutrality rules on online video streaming. It’s at 11AM at Salon J in the Hilton. Nitin is really good at putting concepts in a GNU frame, so I expect this to be a bit (g)rate. Whoami kidding? It will be emacs-ulate.

And then on Thursday March 15th, Marc Rogers is giving a talk about how he hacked the Tesla Model S and the current state of automotive security. (The TLD(r) is that it’s a long road ahead, but automotive companies are driven to make it better). It’s at 11am in Room 203-204 at the JW Marriot. He gave a similar talk at Defcon so this is kind of a SQL.

There’s a lot but looking forward to packet all in. IEEE hope you can join!

--

MAR 9, 2018 | 3:30PM – 4:30PM | HILTON AUSTIN DOWNTOWN SALON KShould I Use This Wi-Fi?

You’re sitting at the gate, about to connect to the free airport Wi-Fi when you stop and think. You’ve heard something about insecure public Wi-Fi before, but could using a bad Wi-Fi network really leak your data or let someone hack your Facebook? Just as there will never be perfect code, there will always be inherent flaws in the tools we use to communicate, work and pay our bills. Find out how vulnerabilities in the web are exploited and what that means in the future of our online privacy.

MAR 11, 2018 | 11:00AM – 12:00PM | HILTON AUSTIN DOWNTOWN SALON JFighting the Demise of Net Neutrality w/Innovation

As consumer demand for streaming video explodes, digital publishers are scrambling to upgrade their video production capabilities in the fight for viewer eyeballs. But in light of eroding net neutrality protections, publishers must ensure their streaming infrastructure delivers a stellar consumer experience in order to stay relevant and compete with giants like Netflix and YouTube. Hear from industry leaders on how the war on net neutrality will alter how audiences consume video.

MAR 15, 2018 | 11:00AM – 12:00PM | JW MARRIOTT ROOM 203-204Who’s Really in Control of Self-Driving Cars?

A few years back, I hacked the Tesla Model S. Scary, right? Actually, it’s not as scary as it seems. In this session, I’ll discuss the current state of automotive security as I see it and give a deep dive discussion on where security needs to be moving forward. I’ll discuss the steps needed to hack a car, the cars that are most vulnerable, the security of self-driving cars, and how concerned consumers can protect themselves.

Around the fall of 2016, we started seeing DDoS attacks that looked a little different than usual. One attack we saw around that time had traffic coming from 52,467 unique IP addresses. The clients weren’t servers or desktop computers; when we tried to connect to the clients over port 80, we got the login pages to CCTV cameras.

Obviously it’s important to lock down IoT devices so that they can’t be co-opted into evil botnet armies, but when we talk to some IoT developers, we hear a few concerning security patterns. We’ll dive into two problematic areas and their solutions: software updates and TLS.

With PCs, the end user is ultimately responsible for securing their devices. People understand that they need to update their computers and phones. Just 4 months after Apple released iOS 10, it was installed on 76% of active devices.

People just don’t know that they are supposed to update IoT things like they are supposed to update their computers because they’ve never had to update things in the past. My parents are never going to install a software update for their thermometer.

And the problem gets worse over time. The longer a device stays on an older software version, the less likely it will be compatible with the newer version. At some point, an update may not be possible anymore. This is a very real concern as the shelf life of a connected thing can be 10 years in the case of a kitchen appliance - have you ever bought a refrigerator?

This is if the device can be patched at all. First, devices that are low battery are programmed not to receive updates because it’s too draining on the battery. Second, IoT devices are too lightweight to run a full operating system, they run just a compiled binary on firmware which means there’s a limit to the code that can later be pushed to it. Some devices cannot receive specific patches.

The other thing we hear about updates from IoT developers is that often they are afraid to push a new update because it could mean breaking hundreds of thousands of devices at once.

All this may not seem like a big deal - ok, so a toaster can get hacked, so what - but two very real things are at stake. First, every device that’s an easy target makes it easier to make other applications a target. Second, once someone is sitting on a device, they are in your network, which can put at stake any traffic sent over the wire.

The security model that worked for PC doesn’t work for IoT — the end user can’t be responsible, and patching isn’t reliable. We need something else. What’s the solution?

Traffic to an IoT device passes through many different networks: the transit provider from the application server, the content delivery network used to deliver device traffic, the ISP to the building where the device sits.

It is at those network layers that protection can be added. As IoT device traffic moves through these networks, packets can be filtered to only let in good traffic. Even if a device is running vulnerable code, filters added in the network level can keep hackers out.

TLS is used in two ways in IoT devices: First, TLS is used to encrypt data in transit. This is used for data privacy and to make it harder to reverse engineer the communications used by the device. Second, devices store client TLS certificates that are used to authenticate the devices to the application - makes it one step harder to fake a device.

There are three problems developers run into when they want to implement TLS in IoT. The first is that while IoT traffic needs to be quick and lightweight, TLS adds an additional two round trips to the start of every session. The second is that certificates can be large files, and device memory is limited in IoT. And the third is that some of the protocols that are being developed for IoT are plaintext by default.

IoT devices run on low power chips. An IoT device may only have 256 or 512 KB of RAM and often need to conserve battery. They send and receive lots of small information constantly. Imagine an internet connected wind sensor - it measures wind speed and every 30 seconds, sends the new wind speed to the application server. It’s just a few bytes of data it needs to get over the wire and it wants to be able to do so without as much overhead as possible to conserve RAM and battery life.

Here’s an HTTP POST to do that:

But let’s say the same device is going to use TLS. Here’s what the same POST looks like with the TLS handshake — this is with TLS 1.2:

Depending on distance between the device and the application server and the latency of the server, this can be hundreds of milliseconds added. The solution is likely the newest version of TLS, TLS 1.3.

TLS 1.3 eliminates a complete round trip in the TLS handshake, which makes TLS much lighter and faster. It cuts the number of round trips in the handshake by half by predicting what key agreement protocol and algorithm the server will decide to use and sends those guessed parameters and the key share directly in the client hello. And if the server likes that, it sends back its own key share for the same algorithm, and the whole handshake is done.

If the same IoT device talks to the same server again, there’s actually no round trip at all. The parameters chosen in the initial handshake are sent alongside application data in the first packet.

Why isn’t every IoT device using 1.3 today? TLS 1.3 is still actively being developed in the IETF standards track and while Chrome as of version 56 in January and Firefox as of version 52 in March support 1.3, not everything does. The biggest problem today are middleboxes that are used by ISP’s and enterprises that panic when they see a 1.3 handshake and close the connection. This also happened when the world was upgrading to TLS 1.2 and middleboxes only understood TLS 1.1, so it’s just a matter of time.

In a TLS handshake, the server can use a server-side TLS certificate to authenticate itself to the client, and the client can use a client-side certificate to authenticate itself to the server. Devices often store certificates to authenticate themselves to the application server. However, device memory is often limited in IoT, and certificates can be large. What can we do?

Most certificates today use the RSA algorithm, which has been around since the 70’s. The certificates are large because the keys in RSA to be secure need to be large - either 1,024 to 2,048 bytes, however, a newer algorithm using elliptic curve cryptography has been in wide use since the early 2000’s that can solve this problem. With elliptic curve cryptography we can use smaller keys with the same level of security as a larger RSA key and save space on the device.

IoT devices need to be lightweight so two emerging protocols are replacing HTTP as the dominant transfer protocol for some IoT devices: MQTT and CoAP.

MQTT is a pub/sub protocol that has been around almost 20 years. In MQTT, a proxy server acts as a broker. An IoT device or web app publishes a message to the broker, and the broker distributes those messages to all the other IoT devices that need to receive that message.

When MQTT was written almost 20 years ago, it was written without security by intention. It was written for oil and gas companies and they were just sending sensor data and no one thought it needed to be encrypted.

CoAP was standardized just three years ago. It has all the same methods as HTTP, but it’s over UDP so it’s really light.

The problem is, if you want to add TLS (DTLS really because CoAP is over UDP), it no longer is light anymore.

It will be quite interesting to see how update mechanisms and TLS implementations change as the number of deployed IoT devices continues to grow. If this type of thing interests you, come join us.

Phishing is the absolute worst.

Unfortunately, sometimes phishing campaigns use Cloudflare for the very convenient, free DNS. To be clear –– there’s a difference between a compromised server being leveraged to send phishing emails and an intentionally malicious website dedicated to this type of activity. The latter clearly violates our terms of service.

In the past, our Trust and Safety team would kick these intentional phishers off the platform, but now we have a new trick up our sleeve and a way for their malicious emails to mysteriously disappear into the ether.

SMTP - the protocol used for sending email - was finalized in 1982, when it was just a small community online. Many of them knew and trusted each other, and so the protocol was built entirely on trust. In an SMTP message, the MAIL FROM field can be arbitrarily defined. That means you could send an email from any email address, even one you don’t own.

This is great for phishers, and bad for everyone else.

The solution to prevent email spoofing was to create the Sender Policy Framework (SPF). SPF allows the domain owner to specify which servers are allowed to send email from that domain. That policy is stored in a DNS TXT record like this one from cloudflare.com:

$ dig cloudflare.com txt
"v=spf1 ip4:199.15.212.0/22 ip4:173.245.48.0/20 include:_spf.google.com include:spf1.mcsv.net include:spf.mandrillapp.com include:mail.zendesk.com include:customeriomail.com include:stspg-customer.com -all"

This says that email clients should only accept cloudflare.com emails if they come from an IP in the ranges 199.15.212.0/22, 173.245.48.0/20, or one of the IP ranges found in the SPF records for the other domains listed. Then if a receiving email server receives an email from someone@cloudflare.com from the server at 185.12.80.67, that email server would check the SPF records of all the allowed domains until it finds that 185.12.80.67 is allowed because 185.12.80.0/22 is listed in mail.zendesk.com’s SPF record:

$ dig txt mail.zendesk.com
"v=spf1 ip4:192.161.144.0/20 ip4:185.12.80.0/22 ip4:96.46.150.192/27 ip4:174.137.46.0/24 ip4:188.172.128.0/20 ip4:216.198.0.0/18 ~all"

Additional methods for securing email were created after SPF. SPF only validates the email sender but doesn’t do anything about verifying the content of the email. (While SMTP can be sent over an encrypted connection, SMTP is notoriously easy to downgrade to plaintext with an on-path attacker.)

To verify the content, domain owners can sign email messages using DKIM. The email sender includes the message signature in an email header called DKIM-Signature and stores the key in a DNS TXT record.

$ dig txt smtpapi._domainkey.cloudflare.com
"k=rsa\; t=s\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDPtW5iwpXVPiH5FzJ7Nrl8USzuY9zqqzjE0D1r04xDN6qwziDnmgcFNNfMewVKN2D1O+2J9N14hRprzByFwfQW76yojh54Xu3uSbQ3JP0A7k8o8GutRF8zbFUA8n0ZH2y0cIEjMliXY4W4LwPA7m4q0ObmvSjhd63O9d8z1XkUBwIDAQAB"

There’s one more mechanism for controlling email spoofing called DMARC. DMARC sets the overarching email policy, indicates what to do if the policies are not met and sets a reporting email address for logging invalid mail attempts. Cloudflare’s DMARC record says that noncomplying emails should be sent to junk mail, 100% of messages should be subject to filtering and if policies are not met, send the report to the two email addresses below.

$ dig txt _dmarc.cloudflare.com
"v=DMARC1\; p=quarantine\; pct=100\; rua=mailto:rua@cloudflare.com, mailto:gjqhulld@ag.dmarcian.com"

When an email server receives an email from someone@cloudflare.com, it first checks SPF, DKIM and DMARC records to know whether the email is valid, and if not, how to route it.

For known phishing campaigns using the Cloudflare platform for evil, we have a DNS trick for getting their phishing campaigns to stop. If you remember, there are three DNS records required for sending email: SPF, DKIM and DMARC. The last one is the one that defines the overarching email policy for the domain.

What we do is rewrite the DMARC record so that the overarching email policy instructs email clients to reject all emails from that sender. We also remove the other DNS record types used for sending email.

"v=DMARC1; p=reject"

When an email client receives a phishing email, the corresponding DNS records instruct the client not to accept the email and the phishing email is not delivered.

You can see it in action on our fake phish domain, astronautrentals.com.

astronautrentals.com is configured with an SPF record, a DKIM record, and a DMARC record with a policy to accept all email.

However, because it is a known (fake) phishing domain, when you query DNS for these records, SPF will be missing:

$ dig astronautrentals.com txt
astronautrentals.com.	3600	IN	SOA	art.ns.cloudflare.com. dns.cloudflare.com. 2026351035 10000 2400 604800 3600

DKIM will be missing:

$ dig _domainkey.astronautrentals.com txt
astronautrentals.com.	3600	IN	SOA	art.ns.cloudflare.com. dns.cloudflare.com. 2026351035 10000 2400 604800 3600

And DMARC policy will be rewritten to reject all emails:

$ dig _dmarc.astronautrentals.com txt
"v=DMARC1\; p=reject"

If we try to send an email from @astronautrentals.com, the email never reaches the recipient because the receiving client sees the DMARC policy and rejects the email.

This DMARC alteration happens on the fly –– it's a computation we do at the moment when we answer the DNS query –– so the original DNS records are still shown to the domain owner in the Cloudflare DNS editor. This adds some mystery to why the phish attempts are failing to send.

Phishing is the absolute worst, and the problem is that it sometimes succeeds. Last year Verizon reported that 30% of phishing emails are opened, and 13% of those opened end with the receiver clicking on the phishing link.

Keeping people safe on the internet means decreasing the number of successful phishing attempts. We're glad to be able to fight phish using the DNS.

Tomorrow is Thanksgiving in the United States. It’s a holiday for getting together with family characterized by turkey dinner and whatever it is that happens in American football. While celebrating with family is great, if you use a computer for your main line of work, sometimes the conversation turns to how to setup the home wifi or can Russia really use Facebook to hack the US election. Just in case you’re a geek who finds yourself in that position this week, we wanted to give you something to play with. To that end, we’re opening the Warp beta to all Cloudflare users. Feel free to tell your family there’s been an important technical development you need to attend to immediately and enjoy!

Warp allows you to expose a locally running web server to the internet without having to open up ports in the firewall or even needing a public IP address. Warp connects a web server directly to the Cloudflare network where Cloudflare acts as your web server’s network gateway. Every request reaching your origin must travel to the Cloudflare network where you can apply rate limits, access policies and authentication before the request hits your origin. Plus, because your origin is never exposed directly to the internet, attackers can’t bypass protections to reach your origin.

Warp is really easy to get started with. If you use homebrew (we also have packages for Linux and Windows) you can do:

$ brew install cloudflare/cloudflare/warp
$ cloudflare-warp login
$ cloudflare-warp --hostname warp.example.com --hello-world

In this example, replace example.com with the domain you chose at the login command. The warp.example.com subdomain doesn’t need to exist yet in DNS, Warp will automatically add it for you.

That last command spins up a web server on your machine serving the hello warp world webpage. Then Warp starts up an encrypted virtual tunnel from that web server to the Cloudflare edge. When you visit warp.example.com (or whatever domain you chose), your request first hits a Cloudflare data center, then is routed back to your locally running hello world web server on your machine.

If someone far away visits warp.example.com, they connect to the Cloudflare data center closest to them, and then are routed to the Cloudflare data center your Warp instance is connected to, and then over the Warp tunnel back to your web server. If you want to make that connection between Cloudflare data centers really fast, enable Argo, which bypasses internet latencies and network congestions on optimized routes linking the Cloudflare data centers.

To point Warp at a real web server you are running instead of the hello world web server, replace the hello-world flag with the location of your locally running server:

$ cloudflare-warp --hostname warp.example.com http://localhost:8080

Let’s say you have multiple instances of your application running and you want to balance load between them or always route to the closest one for any given visitor. As you spin up Warp, you can register the origins behind Warp to a load balancer. For example, I can run this on 2 different servers (e.g. one on a container in ECS and one on a container in GKE):

$ cloudflare-warp --hostname warp.example.com --lb-pool origin-pool-1 http://localhost:8080

And connections to warp.example.com will be routed seamlessly between the two servers. You can do this with an existing origin pool or a brand new one. If you visit the load balancing dashboard you will see the new pool created with your origins in it, or the origins added to an existing pool.

You can also set up a health check so that if one goes offline, it automatically gets deregistered from the load balancer pool and requests are only routed to the online pools.

You can add Warp to your Dockerfile so that as containers spin up or as you autoscale, containers automatically register themselves with Warp to connect to Cloudflare. This acts as a kind of service discovery.

A reference Dockerfile is available here.

If you use Warp to expose dashboards, staging sites and other internal tools to the internet that you don’t want to be available for everyone, we have a new product in beta that allows you to quickly put up a login page in front of your Warp tunnel.

To get started, go to the Access tab in the Cloudflare dashboard.

There you can define which users should be able to login to use your applications. For example, if I wanted to limit access to warp.example.com to just people who work at Cloudflare, I can do:

Enjoy the Warp beta! (But don't wander too deep into the Warp tunnel and forget to enjoy time with your family.) The whole Warp team is following this thread for comments, ideas, feedback and show and tell. We’re excited to see what you build.

I’m also speaking at 1:30PM on Wednesday in OCCC W414 hashing out encryption and updates for IoT –– DES should be a fun session.

ACK! I did NAT tell you how to find us. Check for sum women in capes a few hops away from the booths with the lava LAMP stack. I'm the one with cURLs.

In D air! Excited to LANd. C you soon.

I work at a company whose job it is to be attacked. As I’m writing this, an automatic mitigation is fighting two ongoing DDoS attacks. Any machine that’s publicly routable on the internet today can be a vector for attack, and that’s a problem.

Today we want to turn the tables and give you a new way of exposing services to the internet without having them be directly, publicly routable. Meet Cloudflare Warp.

CC BY-SA 2.0 image by Christian Ortiz

Cloudflare internally runs about 4,000 containers that make up about 1.5K services and applications. Some of these containers need to network with other local containers, and others need to accept connections over the wire.

Every devops engineer knows that bad things happen to good machines, and so our platform operations team tries to hide servers altogether from the internet. There are several ways to do this:

• Rotate IP addresses

• Deploy proxies

• Create firewall rules

• Configure IP tables

• Limit connections by client certificate

• Cross connect with an upstream provider

• Configure a GRE tunnel

• Authentication mechanisms like OAuth or OIDC

These can be complicated or time consuming, yet none of them are guarantees.

We knew we could make it easier. We started building an internal tool for ourselves - a safer way to expose services running on our own infrastructure (with some service discovery and automation benefits as well...more on that later) and after talking to developers and security engineers that use Cloudflare, we realized there was benefit in opening it up to everyone.

Cloudflare Warp is a security-conscious tool for exposing web applications without needing to expose the server they run on. With Cloudflare Warp, traffic to your application is run over a private, encrypted, virtual tunnel from the Cloudflare edge and traffic is only able to find and access your server if it routes through Cloudflare.

Only Cloudflare knows how to dial back to the application through the virtual tunnel created between the application and Cloudflare. Traffic can never hit your origin directly because it can never find it, your origin isn’t on the internet, it’s only there if you go through Cloudflare, via Warp. Instead, the client connects to the nearest Cloudflare data center, never directly to the application itself.

To start up Cloudflare Warp, it’s just one command. For example, if I want to run Cloudflare Warp to expose an application running locally on port 4000, I run:

cloudflare-warp --hostname example.com https://localhost:4000

Behind the scenes, Cloudflare Warp issues an SSL certificate, installs it on the application server and uses it to generate an encrypted, tunnelled connection back to Cloudflare. (The internal project name for Cloudflare Warp was E.T. because of this ‘phoning home’ behavior). Cloudflare Warp then sets up the corresponding DNS records for the application so that when a visitor next goes to your application, they will be connected through the virtual tunnel back to the application running locally at port 4000.

With this setup, Cloudflare’s edge acts as a network shield in front of your infrastructure. At Cloudflare’s edge you can describe policies (allow 50 connections per second, only to these routes, only from these IP’s and only if they are authenticated) and because traffic through Warp can only reach your servers after it’s traveled through Cloudflare, you can drop unexpected traffic at the edge, only receive clean traffic on your server, and know that it’s been validated by Cloudflare. As you continue to set up applications connected to Cloudflare using Warp, you only have to configure this once with Cloudflare and it can apply holistically across all of your applications, protecting your entire infrastructure.

One of the side benefits of Cloudflare Warp is that immediately when you spin up the Cloudflare Warp agent, it registers DNS records for your application, making it an effective tool for service discovery.

We also allow you to tag tunnels the way you would label your kubernetes pods with key-value pairs like release:stable and release:canary. Soon you’ll also be able to configure routing based on these labels (send 90% of my traffic to the stable release and 10% to the canary release).

The Cloudflare Warp beta is available today and we’re gradually adding people every day. Ready to get started? You can jump in and read the docs or sign up for access to the beta.

F-Root is a single IPv4 address plus a single IPv6 address which both ISC and Cloudflare announce to the global Internet as a shared Anycast. This document reviews how F-Root has performed since that date in March 2017.

The DNS root servers are an important utility provided to all clients on the Internet for free - all F root instances including those hosted on the Cloudflare network are a free service provided by both ISC and Cloudflare for public benefit. Because every online request begins with a DNS lookup, and every DNS lookup requires the retrieval of information stored on the DNS root servers, the DNS root servers plays an invaluable role to the functioning of the internet.

At Cloudflare, we were excited to work with ISC to bring greater security, speed and new software diversity to the root server system. First, the root servers, because of their crucial role, are often the subject of large scale volumetric DDoS attacks, which Cloudflare specializes in mitigating (Cloudflare is currently mitigating two concurrently ongoing DDoS attacks as we write this). Second, with a distributed network of data centers in well over 100 global cities, Cloudflare DNS is close to the end client which reduces round trip times. And lastly, the F-root nodes hosted by Cloudflare also run Cloudflare’s in-house DNS software, written in Go, which brings new code diversity to the root server system.

Throughout the deployment, ISC and Cloudflare paid close attention to telemetry measurements to ensure positive impact on the global DNS and root server system. Here is what both organizations observed when transit was enabled to Cloudflare DNS servers for F-Root.

Using RIPE atlas probe measurements, we can see an immediate performance benefit to the F-Root server, from 8.24 median RTT to 4.24 median RTT.

F-Root actually became one of the fastest performing root servers:

The biggest performance improvement was in the 90th percentile, or what are the top 10% of queries that received the slowest replies. This graph below shows the 90th percentile response time for any given RIPE atlas probe. Each probe is represented by two markers, a red X for before Cloudflare enabled transit and a blue X for after Cloudflare began announcing. You can see a drop in 90th percentile response times, the blue X’s are much lower than the red X’s.

One of the optimizations that DNS resolvers do is preferring the faster root servers. As F-Root picked up speed, DNS resolvers also started sending it more traffic. Here you can see the aggregate number of queries received by each root letter per day, with an increase to F starting on March 30th.

One large public DNS resolver shared with us their internal metrics, where you can also see a large shift of traffic to F-Root as F-Root increased in speed.

When one external DNS monitor published their Root Server measurement report in June 2017, they mentioned F-Root’s increased performance. “9 more countries than in 2015 observed F-Root as the fastest — this is the biggest change across all of the root servers, so in this sense F-Root is “most improved.” F-Root has increasingly become the fastest root server in significant portions of Asia Pacific, Latin America and Eastern Europe.” They noted that “F-Root is now the fastest for roughly one quarter of the countries we tested from.”

We are happy to be working with ISC on delivering answers for F-Root and aim in the process to improve the speed and security of the F-Root server.

You can use Cloudflare for service discovery. By deploying microservices behind Cloudflare, microservices’ origins are masked, secured from DDoS and L7 exploits and authenticated, and service discovery is natively built in. Cloudflare is also cloud platform agnostic, which means that if you have distributed infrastructure deployed across cloud platforms, you still get a holistic view of your services and the ability to manage your security and authentication policies in one place, independent of where services are actually deployed.

Service locations and metadata are stored in a distributed KV store deployed in all 100+ Cloudflare edge locations (the service registry).

Services register themselves to the service registry when they start up and deregister themselves when they spin down via a POST to Cloudflare’s API. Services provide data in the form of a DNS record, either by giving Cloudflare the address of the service in an A (IPv4) or AAAA (IPv6) record, or by providing more metadata like transport protocol and port in an SRV record.

Services are also automatically registered and deregistered by health check monitors so only healthy nodes are sent traffic. Health checks are over HTTP and can be setup with custom configuration so that responses to the health check must return a specific response body and or response code otherwise the nodes are marked as unhealthy.

Traffic is distributed evenly between redundant nodes using a load balancer. Clients of the service discovery query the load balancer directly over DNS. The load balancer receives data from the service registry and returns the corresponding service address. If services are behind Cloudflare, the load balancer returns a Cloudflare IP address to route traffic to the service through Cloudflare’s L7 proxy.

Traffic can also be sent to specific service nodes based on client geography, so the data replication service in North America, for example, can talk to a specific North American version of the billing service, or European data can stay in Europe.

Clients query the service registry over DNS, and service location and metadata is packaged in A, AAAA, CNAME or SRV records. The benefit of this is that no additional client software needs to be installed on service nodes beyond a DNS client. Cloudflare works natively over DNS, meaning that if your services have a DNS client, there’s no extra software to install, manage, upgrade or patch.

While usually, TTL’s in DNS mean that if a service location changes or deregisters, clients may still get stale information, Cloudflare DNS keeps low TTL’s (it’s able to do this and maintain fast performance because of its distributed network) and if you are using Cloudflare as a proxy, the DNS answers always point back to Cloudflare even when the IP’s of services behind Cloudflare change, removing the effect of cache staleness.

If your services communicate over HTTP/S and websockets, you can additionally use Cloudflare as a L7 proxy for added security, authentication and optimization. Cloudflare prevents DDoS attacks from hitting your infrastructure, masks your IP’s behind its network, and routes traffic through an optimized edge PoP to edge PoP route to shave latency off the internet.

Once service <--> service traffic is going through Cloudflare, you can use TLS client certificates to authenticate traffic between your services. Cloudflare can authenticate traffic at the edge by ensuring that the client certificate presented during the TLS handshake is signed by your root CA.

Sign up for Cloudflare account

During the signup process, add all your initial services as DNS records in the DNS editor.

To finish sign up, move DNS to Cloudflare by logging into your registrar and changing your nameservers to the Cloudflare nameservers assigned to you when you signed up for Cloudflare. If you want traffic to those services to be proxied through Cloudflare, click on the cloud next to each DNS record to make it orange.

Run a script on each node so that:

On startup, the node sends a POST to the DNS record API to register itself and PUT to load balancing API to add itself to the origin pool.

On shutdown, the node sends a DELETE to the DNS record API to deregister itself and PUT to load balancing API to remove itself to the origin pool.

These can be accomplished via startup and shutdown scripts on Google Compute Engine or user data scripts or auto scaling lifecycle hooks on AWS.

Registration:

curl -X POST "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"type":"SRV","data":{"service":"_http","proto":"_tcp","name":"name","priority":1,"weight":1,"port":80,"target":"staging.badtortilla.com"},"ttl":1,"zone_name":"badtortilla.com","name":"_http._tcp.name.","content":"SRV 1 1 80 staging.badtortilla.com.","proxied":false,"proxiable":false,"priority":1}'

De-Registration:

curl -X DELETE "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json"

Add or remove an origin from an origin pool (this should be a unique IP per node added to the pool):

curl -X PUT "https://api.cloudflare.com/client/v4/user/load_balancers/pools/17b5962d775c646f3f9725cbc7a53df4" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"description":"Primary data center - Provider XYZ","name":"primary-dc-1","enabled":true,"monitor":"f1aba936b94213e5b8dca0c0dbf1f9cc","origins":[{"name":"app-server-1","address":"1.2.3.4","enabled":true}],"notification_email":"someone@example.com"}'

Create a health check. You can do this in the API or in the Cloudflare dashboard (in the Load Balancer card).

curl -X POST "https://api.cloudflare.com/client/v4/organizations/01a7362d577a6c3019a474fd6f485823/load_balancers/monitors" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"type":"https","description":"Login page monitor","method":"GET","path":"/health","header":{"Host":["example.com"],"X-App-ID":["abc123"]},"timeout":3,"retries":0,"interval":90,"expected_body":"alive","expected_codes":"2xx"}'

Create an initial load balancer, either through the API or in the Cloudflare dashboard.

curl -X POST "https://api.cloudflare.com/client/v4/zones/699d98642c564d2e855e9661899b7252/load_balancers" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"description":"Load Balancer for www.example.com","name":"www.example.com","ttl":30,"fallback_pool":"17b5962d775c646f3f9725cbc7a53df4","default_pools":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196","00920f38ce07c2e2f4df50b1f61d4194"],"region_pools":{"WNAM":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"ENAM":["00920f38ce07c2e2f4df50b1f61d4194"]},"pop_pools":{"LAX":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"LHR":["abd90f38ced07c2e2f4df50b1f61d4194","f9138c5d07c2e2f4df57b1f61d4196"],"SJC":["00920f38ce07c2e2f4df50b1f61d4194"]},"proxied":true}'

(optional) Setup geographic routing rules. You can do this via API or in the Cloudflare dashboard.

curl -X POST "https://api.cloudflare.com/client/v4/zones/699d98642c564d2e855e9661899b7252/load_balancers" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"description":"Load Balancer for www.example.com","name":"www.example.com","ttl":30,"fallback_pool":"17b5962d775c646f3f9725cbc7a53df4","default_pools":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196","00920f38ce07c2e2f4df50b1f61d4194"],"region_pools":{"WNAM":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"ENAM":["00920f38ce07c2e2f4df50b1f61d4194"]},"pop_pools":{"LAX":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"LHR":["abd90f38ced07c2e2f4df50b1f61d4194","f9138c5d07c2e2f4df57b1f61d4196"],"SJC":["00920f38ce07c2e2f4df50b1f61d4194"]},"proxied":true}'

(optional) Setup Argo for faster PoP to PoP transit in the traffic app of the Cloudflare dashboard.

(optional) Setup rate limiting via API or in the dashboard

curl -X POST "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/rate_limits" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","disabled":false,"description":"Prevent multiple login failures to mitigate brute force attacks","match":{"request":{"methods":["GET","POST"],"schemes":["HTTP","HTTPS"],"url":"*.example.org/path*"},"response":{"status":[401,403],"origin_traffic":true}},"bypass":[{"name":"url","value":"api.example.com/*"}],"threshold":60,"period":900,"action":{"mode":"simulate","timeout":86400,"response":{"content_type":"text/xml","body":"<error>This request has been rate-limited.</error>"}}}'

(optional) Setup TLS client authentication. (Enterprise only) Send your account manager your root CA certificate and which options you would like enabled.

"No matter what happens in life, be good to people. Being good to people is a wonderful legacy to leave behind." - Taylor Swift (whose website has been IPv6 enabled for many many years)

Starting today with free domains, IPv6 is no longer something you can toggle on and off, it’s always just on.

Cloudflare has always been a gateway for visitors on IPv6 connections to access sites and applications hosted on legacy IPv4-only infrastructure. Connections to Cloudflare are terminated on either IP version and then proxied to the backend over whichever IP version the backend infrastructure can accept.

That means that a v6-only mobile phone (looking at you, T-Mobile users) can establish a clean path to any site or mobile app behind Cloudflare instead of doing an expensive 464XLAT protocol translation as part of the connection (shaving milliseconds and conserving very precious battery life).

That IPv6 gateway is set by a simple toggle that for a while now has been default-on. And to make up for the time lost before the toggle was default on, in August 2016 we went back retroactively and enabled IPv6 for those millions of domains that joined before IPv6 was the default. Over those next few months, we enabled IPv6 for nearly four million domains –– you can see Cloudflare’s dent in the IPv6 universe below –– and by the time we were done, 98.1% of all of our domains had IPv6 connectivity.

As an interim step, we added an extra feature –– when you turn off IPv6 in our dashboard, we remind you just how archaic we think that is.

With close to 100% IPv6 enablement, it no longer makes sense to offer an IPv6 toggle. Instead, Cloudflare is offering IPv6 always on, with no off-switch. We’re starting with free domains, and over time we’ll change the toggle on the rest of Cloudflare paid-plan domains.

In November we published stats about the IPv6 usage we see on the Cloudflare network in an attempt to answer who and what is pushing IPv6. The top operating systems by percent IPv6 traffic are iOS, ChromeOS, and MacOS respectively. These operating systems push significantly more IPv6 traffic than their peers because they use a routing choice algorithm called Happy Eyeballs. Happy Eyeballs opportunistically chooses IPv6 when available by doing two DNS lookups –– one for an IPv6 address (this IPv6 address is stored in the DNS AAAA record - pronounced quad-A) and then one for the IPv4 address (stored in the DNS A record). Both DNS queries are flying over the Internet at the same time and the client chooses the address that comes back first. The client even gives IPv6 a few milliseconds head start (iOS and MacOS give IPv6 lookups a 25ms head start for example) so that IPv6 may be chosen more often. This works and has fueled some of IPv6’s growth. But it has fallen short of the goal of a 100% IPv6 world.

While there are perfectly good historical reasons why IPv6 and IPv4 addresses are stored in separate DNS types, today clients are IP version agnostic and it no longer makes sense for it to require two separate round trips to learn what addresses are available to fetch a resource from.

Alongside OpenDNS, we are testing a new idea - what if you could ask for all the addresses in just one DNS query?

With OpenDNS, we are prototyping and testing just that –– a new DNS metatype that returns all available addresses in one DNS answer –– A records and AAAA records in one response. (A metatype is a query type in DNS that end users can’t add into their DNS zone file, it’s assembled dynamically by the authoritative nameserver.)

What this means is that in the future if a client like an iPhone wants to access a mobile app that uses Cloudflare DNS or using another DNS provider that supports the spec, the iPhone DNS client would only need to do one DNS lookup to find where the app’s API server is located, cutting the number of necessary round trips in half.

This reduces the amount of bandwidth on the DNS system, and pre-populates global DNS caches with IPv6 addresses, making IPv6 lookups faster in the future, with the side benefit that Happy Eyeballs clients prefer IPv6 when they can get the address quickly, which increases the amount of IPv6 traffic that flows through the Internet.

We have the metaquery working in code with the reserved TYPE65535 querytype. You can ask a Cloudflare nameserver for TYPE65535 of any domain on Cloudflare and get back all available addresses for that name.

$ dig cloudflare.com @ns1.cloudflare.com -t TYPE65535 +short
198.41.215.162
198.41.214.162
2400:cb00:2048:1::c629:d6a2
2400:cb00:2048:1::c629:d7a2
$

Did we mention Taylor Swift earlier?

$ dig taylorswift.com @ns1.cloudflare.com -t TYPE65535 +short
104.16.193.61
104.16.194.61
104.16.191.61
104.16.192.61
104.16.195.61
2400:cb00:2048:1::6810:c33d
2400:cb00:2048:1::6810:c13d
2400:cb00:2048:1::6810:bf3d
2400:cb00:2048:1::6810:c23d
2400:cb00:2048:1::6810:c03d
$

We believe in proving concepts in code and through the IETF standards process. We’re currently working on an experiment with OpenDNS and will translate our learnings to an Internet Draft we will submit to the IETF to become an RFC. We’re sure this is just the beginning to faster, better deployed IPv6.

Have you noticed something new in your Cloudflare analytics dashboard this morning? You can now see detailed DNS analytics for your domains on Cloudflare.

If you want to skip to the punch and start exploring, go check it out here. Otherwise, hop on the DNS magic school bus - and let us show you all the neat stats in your now-available DNS analytics.

At the top of the DNS analytics dashboard you can see your DNS traffic health. This “Queries by Response Codes” graph breaks down queries by what response code Cloudflare DNS answered to the visitor. Like HTTP response codes, DNS response codes give an indication of what is happening behind the scenes. Mostly you will just see NOERROR, the HTTP 200 of DNS response codes, and NXDOMAIN, the HTTP 404 of DNS response codes. NXDOMAIN is particularly interesting - what are people querying for that doesn’t exist?

If you are an enterprise customer and you want to know what all the NXDOMAIN queries are, just scroll down a little bit where we show you the top queries for your domain and top queries for your domain for DNS records that don’t exist (aka top NXDOMAIN queries).

If you are curious then where all of these NXDOMAIN queries are coming from, all Pro, Business and Enterprise plan customers can scroll down a little bit further to the geography section where we show you the breakdown of where your queries come from and also where your queries returning NXDOMAIN come from.

If you are an Enterprise customer and want to dive in deeper just for one or a few names, you can filter down the entire dashboard by hostname. You can even filter the dashboard by names that don’t exist in your DNS records so you can explore traffic for misconfigured records resolvers are looking for that don’t exist.

Lastly, back up at the top, business and enterprise customers can see a breakdown of traffic by record type. If your domain has DNSSEC enabled, you’ll even see queries for DNSKEY records, meaning DNS resolvers that are DNSSEC-aware are asking Cloudflare for your DNSSEC keys to verify DNS records are untampered with.

If you like what you’re seeing and wish you could use the data to piece together your own dashboards? You can. We’ve developed a Grafana plugin you can use to build your own dashboards and an API you can use to create your own tools. Plus the API gives you some added neat information like the distribution of queries over IPv6 vs IPv4 and UDP vs TCP, as well as the distribution of answers by query and response size. (DNS people rejoice -- now you can finally see if there’s a difference in query size between IPv6 and IPv4, and what is the breaking point where Cloudflare DNS answers switch from UDP to TCP.)

DNS Firewall customers, you’re next! Stay tuned! We have a really cool dashboard coming up for you that shows latency and errors on a per origin basis. We can’t wait for you to see it. In the meantime, you can see all your data through the Grafana plugin and API.

If you use Cloudflare but don’t see traffic in the DNS analytics dashboard, it’s because you use Cloudflare through a CNAME setup where your DNS is hosted outside of Cloudflare. Your DNS isn’t on us, so we can’t give you cool DNS analytics. If you’re interested in moving your DNS over to Cloudflare to try out DNS analytics and the newly announced Cloudflare Load Balancing, give our team a holler, they will swap your setup for you.

We are excited for you to get started digging into your DNS traffic. Let us know what you think by sending your feedback to dns-analytics-feedback@cloudflare.com. Cloudflare DNS team eagerly awaits your feedback :). Now go on vacation, we’ll look after your DNS!

In a traditional TLS handshake, the client authenticates the server, and the server doesn’t know too much about the client. However, starting now, Cloudflare is offering enterprise customers TLS with client authentication, meaning that the server additionally authenticates that the client connecting to it is authorized to connect.

TLS Client Authentication is useful in cases where a server is keeping track of hundreds of thousands or millions of clients, as in IoT, or in a mobile app with millions of installs exchanging secure information. For example, an IoT company can issue a unique client certificate per device, and then limit connections to their IoT infrastructure to only their devices by blocking connections where the client doesn’t present a certificate signed by the company’s certificate authority.

Or in the case of a mobile banking app, where the bank wants to ensure customers’ secure financial data doesn’t get stolen by bots spoofing their mobile app, they can issue a unique certificate to every app install and in the TLS handshake validate requests are coming from their mobile app. Client authentication is also useful for VPNs, enterprise networks or staging sites, where corporations and developers need to lock down connections to only laptops and phones owned by their employees and teammates.

You may be thinking - don’t we have API keys for that? But client certificates offer a layer of security that API keys cannot provide. If an API key gets compromised mid-connection, it can be reused to fire its own valid, trusted requests to the backend infrastructure. However, the private key of the client certificate is used to create a digital signature in every TLS connection, and so even if the certificate is sniffed mid-connection, new requests can’t be instantiated with it.

In a handshake with TLS Client Authentication, the server expects the client to present a certificate, and sends the client a client certificate request with the server hello. Then in the key exchange in the next trip to the server, the client also sends its client certificate. The client certificate is then used to sign the TLS handshake and the digital signature is sent to the server for verification. You can see the whole handshake here:

TLS Client Authentication can be CPU intensive to implement - it’s an additional cryptographic operation on every request. And if there’s a flood of invalid traffic, each request in that traffic flood kicks off a verification step. Companies can move the TLS client authentication to Cloudflare’s edge to offload the expensive verification.

If we are performing TLS Client Authentication for a company, the company sends us the root certificate(s) we should validate the client certificates against. Then the company can set TLS Client Authentication to one of two modes: enforce mode returns a 403 and optional custom JSON or HTML when the client certificate is invalid, and report mode forwards all requests to the origin, even if the certificate is invalid. Cloudflare will send a header including the status of the certificate (none, valid, invalid) and the certificate Subject Key Identifier (SKI) to the origin. For companies that use the client certificate for identification, Cloudflare can also forward any field of the client certificate as a custom header.

To use TLS client authentication, you must first set up PKI (Public Key Infrastructure) infrastructure to issue client certificates. If you are interested in running TLS client authentication but don’t have PKI infrastructure set up to issue client certificates, we have open sourced our PKI for you to use. Here is great documentation by our friends at CoreOS on how to use cfssl to issue client certificates. If you prefer not to run your own CA and rely on an established certificate authority, we have partnered with a few certificate authorities who can provide the client certificates for you.

If you are an enterprise customer and would like to get started using TLS client authentication with Cloudflare, reach out to your account team and we’ll help you get setup. If you are not yet an enterprise customer but are interested in trying out TLS client authentication, get in touch.

Within the next year, we’ll be adding TLS client authentication support for all Cloudflare plans. After all, using encryption to make the web more trusted is what we’re about. Stay tuned.

UPDATE - 1/22/19: This functionality has changed and is being incorporated into Cloudflare Access. A beta is currently underway. Apologies for any confusion.

In October, we wrote about a 1.75M rps DDoS attack we mitigated on our network, launched by 52,467 unique IP’s, mostly hacked CCTV cameras.

We continued to see more IoT devices in DDoS attacks, and so we started to put together a security solution to protect the devices from becoming part of the botnet in the first place. Today we’re announcing it: Cloudflare Orbit.

As we talked to IoT companies, over and over again we heard the same thing. In the consumer electronics space, IoT manufacturers were telling us that they were shipping patches to their devices, but their end users didn’t always download and install them. (Reserve your judgment, how many times have you pressed ignore when your phone asked you to update its operating system?) In the industrial control, medical and automotive spaces, where devices are used in life-critical functions, we heard a different story. Even if someone wanted to apply a patch, it just wasn’t that easy. For example, even if the manager of a nuclear power plant wants to update software on their thermostats, shutting down operations long enough to do that means the update has to be scheduled.

This is if the device can be patched at all - most devices are patchable, but up to a point. When Jeep was shown to be vulnerable to remote code execution, Chrysler had to recall 1.4 million vehicles.

This model where the end user is responsible for the security of the overall device is a relic of the PC age, where users knew and understood that their computers could have vulnerabilities, and as their software vendors released patches ––on so called “Patch Tuesdays,” for example––users knew to go and download them.

PC security does not work for IoT. Consumers do not share that similar understanding that they need to update their toasters, lightbulbs and cars, because they’ve never needed to in the past. And since we will never write perfect code, we need a better way of securing devices without waiting for users to do it for us.

Thinking about this challenge, we were reminded of when the ShellShock vulnerability was discovered, Cloudflare was able to automatically keep the requests that would trigger the vulnerability from reaching our customers. With Orbit, Cloudflare can do the same thing, only for devices. For example, when Jeep was shown to be vulnerable, instead of recalling 1.4 million vehicles, Fiat Chrysler could have patched the bug in all the vehicles with just a simple rule in Cloudflare's firewall restricting access to the vulnerable DBUS service listening on port 6667 of every Jeep.

Orbit sits one layer before the device and provides a shield of security, so even if the device is running past its operating system’s expiration date, Cloudflare protects it from exploits. And while devices may be seldom patched, the Cloudflare security team is shipping code every day, adding new firewall rules to Cloudflare’s edge. Think of it like changing IoT to I*oT — devices can still access the Internet, but only after passing through Cloudflare where malicious requests can be filtered.

For the last year, Cloudflare has been working with a number of IoT vendors to develop Orbit. Already more than 120 million IoT devices are safer behind Cloudflare’s network. Lockitron is one of the IoT companies using Cloudflare. “Keeping our products and customers secure is our primary concern,” says Paul Gerhardt, co-founder of Lockitron. “Cloudflare provides an extra layer of security that allows us to keep our devices continually updated and ahead of any vulnerabilities.”

Instead of writing and shipping a patch, IoT companies can write logic on Cloudflare’s edge, and write their own firewall rules to run on Cloudflare, and it updates the Cloudflare Orbit layer immediately, for all of their devices, without their users ever being so much as nudged to install something.

Plus, with requests going through Cloudflare, Cloudflare can compress transmitted data and speed up traffic, meaning less time is spent waiting on open connections and more time left in battery.

A common challenge we heard from IoT device manufacturers was how to authenticate devices and know which connecting clients were authorized company devices, and which were bots only pretending to be. Starting today, Cloudflare now offers Enterprise domains TLS Client Authentication, a TLS handshake where the client authenticates the server’s certificate (as with any TLS handshake) and also the client has a certificate that the server authenticates.

Some IoT vendors already implement their own Client Authentication, but do so at the same origin servers that handle the rest of their IoT infrastructure. Not only is this computationally expensive, but any invalid traffic flood causes a burden on the whole server.

With Client Authentication on Cloudflare, Cloudflare’s edge handles the load of the TLS handshakes, validating the device client certificates and only sending the IoT infrastructure traffic from authorized devices.

“This approach of adding security to the network is extremely important for industrial manufacturers. Being able to patch vulnerabilities from the network rather than at the device level is a major shift in the way we secure IoT devices, and one that is completely necessary.” — Sam Cece, CEO of Swift Sensors, an industrial IoT company.

“Car controllers are IoT devices. Karamba Security hardens these devices and prevents cyberattacks with zero false positives to maintain driver and passenger safety. We view Cloudflare’s Orbit as a complementary solution that enables secure connectivity between the cars’ hardened controllers and the car company’s data center for trusted, over-the-air updates.” — Ami Dotan, CEO of Karamba Security, which is building secure platforms for smart automobiles.

“We are at the beginning of a new era in which a vast number of devices will be connecting to the Internet and security will play a critical role in the successful roll-out and adoption of IoT devices. Cloudflare’s Orbit adds another layer of defense that compliments other security measures such as strong hardware-based device security and helps ensure a safer Internet of Things." — Quinn Li, VP and global head of Qualcomm Ventures, the investment arm of Qualcomm Incorporated, the leading supplier of components for IoT devices.

"IoT devices create a distinct security challenge both because of the inability of most end users to update their software, as well as the cost that manufacturers bear if they release an update that bricks devices. This is even worse for legacy devices, many of which are effectively unpatchable. Cloudflare's Orbit provides a unique approach to help with these challenges, by deploying a defensive layer in the network where security updates can be safely made without end-user intervention or on-device changes." — Michael Freedman, professor of computer science at Princeton University and CTO of Timescale.

Orbit is available now to all IoT device companies. To learn more or get started, visit the site, or get in touch. We’re excited to hear from you.

We’ve been working with registrars and registries in the IETF on making DNSSEC easier for domain owners, and over the next two weeks we’ll be starting out by enabling DNSSEC automatically for .dk domains.

Before we get into the details of how we've improved the DNSSEC experience, we should explain why DNSSEC is important and the function it plays in keeping the web safe.

DNSSEC’s role is to verify the integrity of DNS answers. When DNS was written in the early 1980’s, it was only a few researchers and academics on the internet. They all knew and trusted each other, and couldn’t imagine a world in which someone malicious would try to operate online. As a result, DNS relies on trust to operate. When a client asks for the address of a hostname like www.cloudflare.com, without DNSSEC it will trust basically any server that returns the response, even if it wasn’t the same server it originally asked. With DNSSEC, every DNS answer is signed so clients can verify answers haven’t been manipulated over transit.

If DNSSEC is so important, why do so few domains support it? First, for a domain to have the opportunity to enable DNSSEC, not only do its DNS provider, its registrar and its registry all have to support DNSSEC, all three of those parties have to also support the same encryption algorithms.

For domains that do have the ability to enable DNSSEC, DNSSEC is just not easy enough -- domain owners need to first enable DNSSEC with their DNS provider, and then copy and paste some values (called a DS record) from their DNS provider’s dashboard to their registrar’s dashboard, making sure not to miss any characters when copying and pasting, because that would cut off traffic to their whole domain. What we need here is automation.

It's been Cloudflare's long-standing statement that as the DNS operator, we would like to update the DS automatically for a user, but DNS operates on a legacy model where the registrar is able to talk directly to the registry, but the DNS operator (Cloudflare) is left completely out of that model.

Here at Cloudflare, we’re determined it’s time to change that outdated system. We have published an Internet Draft to propose a new model for how DNS operators, registries and registrars could operate and communicate to make specific user-authorized changes to domains. It’s important to point out that the IETF works on the principle of rough consensus and running code. Cloudflare, in conjunction with the .dk registry, has produced running code, and we’re very close to getting consensus. That Internet Draft is now making its way through the Standards Track within the IETF and on it’s way to becoming an fully-fledged RFC.

The ccTLD operator for Denmark (ie. the .dk domains) has also realized that the model is outdated. They provide their users (and the operators of nameservers associated with .dk domains) a programmatic way of installing and updating DS records. This is exactly what operators like Cloudflare need.

Cloudflare has been testing their API and is now ready to kick off an automated, clean, safe and reliable way of updating DS records for our .dk customers. Over the next two weeks we will enable DNSSEC for .dk domains that have started to in the past, but haven’t finished the process.

Of course, for Cloudflare, there’s no surprise that Denmark is the home to forward thinkers like this!

If you have a .dk domain on Cloudflare, you really don’t need to do anything except flip the switch enabling DNSSEC within the Cloudflare login console before we do the migration on Tuesday, April 18, 2017.

We are excited to work with the .dk registry on this first step to making DNSSEC automatic, and are looking for other TLD’s looking to make DNSSEC easy to use.

Cloudflare has been a long time supporter of AMP, an open-source markup language 1.5 billion web pages are using to accelerate their mobile web performance. Cloudflare runs Ampersand, the only alternative to Google’s AMP cache, and earlier this year we launched Accelerated Mobile Links, a way for sites on Cloudflare to open external links on their site in AMP format, as well as Firebolt, leveraging AMP to speed up ad performance.

One of the biggest challenges developers face in converting their web pages to AMP is testing their AMP pages for valid AMP syntax before deploying. It's not enough to make the templates work at dev time, you also need to validate individual pages before they’re published.

Imagine, for example, a publishing company where content creators who are unfamiliar with AMP are modifying pages. Because the AMP markup language is so strict, one person adding an interactive element to a page can all of a sudden break the AMP formatting and stop the page from validating.

We wanted to make it as easy as possible to move webpages and sites to AMP so we built an AMP linter API for developers to check that their AMP pages are formatted correctly, even before they are deployed.

To check if a webpage’s AMP markup is correct, just send the AMP page to the endpoint URL Removed like this:

curl URL REMOVED/amp.usatoday.com/story/82055560/
{
  "source": "http://amp.usatoday.com/story/82055560/", 
  "valid": true, 
  "version": "1488238516283"
}

The API has options to send just the markup content, or point the linter to the live site. To send a file, add the --data-binary flag:

curl -X POST --data-binary @amp_page.html -H 'Content-Type: text/html; charset=UTF-8' URL REMOVED

If you send an AMP page with invalid AMP syntax, the message returned will tell you exactly what breaks your AMP page, and will point you to the specific place in the AMP reference where you can see the implementation guide for the broken element.

curl -X POST --data-binary @invalid_amp.html -H 'Content-Type: text/html; charset=UTF-8' URL REMOVED
{
  "errors": [
    {
      "code": "MANDATORY_TAG_MISSING", 
      "col": 7, 
      "error": "The mandatory tag 'link rel=canonical' is missing or incorrect.", 
      "help": "https://www.ampproject.org/docs/reference/spec.html#required-markup", 
      "line": 13
    }
  ], 
  "source": "POST", 
  "valid": false, 
  "version": "1485227592804"
}

Here’s a reference in python, and if you want to send html directly instead of a live webpage, replace line two with r = requests.post([URL Removed], data=html)

import requests

u = 'www.bbc.co.uk/news/amp/39192025'
r = requests.get('URL REMOVED' + u)
validation = r.json()
if validation['valid']:
  print u, 'is valid'
else:
  print u, 'failed!'
  for e in validation['errors']: 
    print e

Let us know what you think - you can send us feedback at amp-publisher@cloudflare.com. Whether you embed this tool into your build and continuous integration processes, or into your CMS workflows, we’re excited to hear how you use it.

We protect entire corporate networks, help customers build Internet-scale applications efficiently, accelerate any website or Internet application, ward off DDoS attacks, keep hackers at bay, and can help you on your journey to Zero Trust.

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.

To learn more about our mission to help build a better Internet, start here. If you're looking for a new career direction, check out our open positions.

Over the last six years, we’ve built the tooling, infrastructure and expertise to run a DNS network that handles our scale - we’ve answered a few million DNS queries in the few seconds since you started reading this.

DNS is the backbone of the internet. Every email, website visit, and API call ultimately begins with a DNS lookup. Internet is built on DNS, so every hosting company, registrar, TLD operator, and cloud provider must be able to run reliable DNS.

Last year CloudFlare launched Virtual DNS, providing DDoS mitigation and a strong caching layer of 100 global data centers to those running DNS infrastructure.

Today we’re expanding that offering with two new features for an extra layer of reliability: Serve Stale and DNS Rate Limiting.

Virtual DNS sits in front of your DNS infrastructure. When DNS resolvers lookup answers on your authoritative DNS, the query first goes to CloudFlare Virtual DNS. We either serve the answer from cache if we have the answer in cache, or we reach out to your nameservers to get the answer to respond to the DNS resolver.

Even if your DNS servers are down, Virtual DNS can now answer on your behalf by serving a stale answer from cache. It’s a backup for you when your nameservers aren’t up, keeping your customers’ records online.

Virtual DNS can now help you control what hits your network. Because Virtual DNS sits in front of your DNS nameservers, it shields your infrastructure from the load, and only sends you the traffic that you want to get.

With rate limiting, you configure a threshold of how many queries per second Virtual DNS should send through to your DNS servers. The rate limit only counts queries that are sent to your server, not what gets answered from cache. Queries that exceed the rate limit may still be answered with answers in cache, even if they are stale, with our new Serve Stale functionality.

If you’re running your own DNS infrastructure and would like to get started with Virtual DNS, get in touch.

Since CloudFlare launched Page Rules in 2012, our Free, Pro and Business users have been asking for a way to get more Page Rules without committing to the next plan up. Starting today, anyone on CloudFlare can add 5 additional Page Rules for just $5/month.

Page Rules allows you to fine tune your site speed and to apply CloudFlare’s wide range of features to specific parts of your site. Page Rules are also accessible over our API, so you can integrate them into your build process or sync them across your domains.

To help you get the most out of Page Rules, we’re also launching a tutorial site that features videos to help you setup Page Rules for specific content management systems like WordPress, Magento and Drupal, and for specific goals like optimizing your website's speed, increasing security, and saving on your bandwidth costs.

CC BY-SA 2.0 image by Chris Short

I will briefly explain a few concepts you need to know about DNSSEC and negative answers, and then we will dive into how CloudFlare saves on compute when asked for names that don’t exist.

Here’s a quick summary of DNSSEC:

This is an unsigned DNS answer (unsigned == no DNSSEC):

cloudflare.com.		299	IN	A	198.41.214.162
cloudflare.com.		299	IN	A	198.41.215.162

This is an answer with DNSSEC:

cloudflare.com.		299	IN	A	198.41.214.162
cloudflare.com.		299	IN	A	198.41.215.162
cloudflare.com.		299	IN	RRSIG	A 13 2 300 20160311145051 20160309125051 35273     cloudflare.com. RqRna0qkih8cuki++YbFOkJi0DGeNpCMYDzlBuG88LWqx+Aaq8x3kQZX TzMTpFRs6K0na9NCUg412bOD4LH3EQ==

Answers with DNSSEC contain a signature for every record type that is returned. (In this example, only A records are returned so there is only one signature.) The signatures allow DNS resolvers to validate the records returned and prevent on-path attackers from intercepting and changing the answers.

There are two types of negative answers. The first is NXDOMAIN, which means that the name asked for does not exist. An example of this is a query asking for missing.cloudflare.com. missing.cloudflare.com doesn’t exist at all.

The second type is NODATA, which means that the name does exist, just not in the requested type. An example of this would be asking for the MX record of blog.cloudflare.com. There are A records for blog.cloudflare.com but no MX records so the appropriate response is NODATA.

To see what gets returned in a negative NXDOMAIN answer, let’s look at the response for a query for bogus.ietf.org.

The first record that has to be returned in a negative answer with DNSSEC is an SOA, just like in an unsigned negative answer. The SOA contains some metadata about the zone and lets the recursor know how long to cache the negative answer for.

ietf.org.	1179	IN	SOA	ns0.amsl.com. glen.amsl.com. 1200000325 1800 1800 604800 1800

Because the domain is signed with DNSSEC, the signature for the SOA is also returned:

ietf.org.	1179	IN	RRSIG	SOA 5 2 1800 20170308083354 20160308073501 40452 ietf.org. S0gIjTnQGA6TyIBjCeBXL4ip8aEQEgg2y+kCQ3sLtFa3oNy9vj9kj4aP 8EVu4oIexr8X/i9L8Oj5ec4HOrQoYsMGObRUG0FGT0MEbxepi+wWrfed vD/3mq8KZg/pj6TQAKebeSQGkmb8y9eP0PdWdUi6EatH9ZY/tsoiKyqg U4vtq9sWZ/4mH3xfhK9RBI4M7XIXsPX+biZoik6aOt4zSWR5WDq27pXI 0l+BLzZb72C7McT4PlBiF+U86OngBlGxVBnILyW2aUisi2LY6KeO5AmK WNT0xHWe5+JtPD5PgmSm46YZ8jMP5mH4hSYr76jqwvlCtXvq8XgYQU/P QyuCpQ==

The next part of the negative answer in DNSSEC is a record type called NSEC. The NSEC record returns the previous and next name in the zone, which proves to the recursor that the queried name cannot possibly exist, because nothing exists between the two names listed in the NSEC record.

www.apps.ietf.org.	1062	IN	NSEC	cloudflare-verify.ietf.org. A RRSIG NSEC

This NSEC record above tells you that bogus.ietf.org does not exist because no names exist canonically between www.apps.ietf.org and cloudflare-verify.ietf.org. Of course, this record also has a signature contained in the answer:

www.apps.ietf.org.	1062	IN	RRSIG	NSEC 5 4 1800 20170308083322 20160308073501 40452 ietf.org. NxmjhCkTtoiolJUow/OreeBRxTtf2AnIPM/r2p7oS/hNeOdFI9tpgGQY g0lTOYjcNNoIoDB/r56Kd+5wtuaKT+xsYiZ4K413I+cmrNQ+6oLT+Mz6 Kfzvo/TcrJD99PVAYIN1MwzO42od/vi/juGkuKJVcCzrBKNHCZqu7clu mU3DEqbQQT2O8dYIUjLlfom1iYtZZrfuhB6FCYFTRd3h8OLfMhXtt8f5 8Q/XvjakiLqov1blZAK229I2qgUYEhd77n2pXV6SJuOKcSjZiQsGJeaM wIotSKa8EttJELkpNAUkN9uXfhU+WjouS1qzgyWwbf2hdgsBntKP9his 9MfJNA==

A second NSEC record is also returned to prove that there is no wildcard that would have covered bogus.ietf.org:

ietf.org.	1062	IN	NSEC	ietf1._domainkey.ietf.org. A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY SPF

This record above tells you that a wildcard (*. ietf.org) would have existed between those two names. Because there is no wildcard record at *.ietf.org, as proven by this NSEC record, the DNS resolver knows that really nothing should have been returned for bogus.ietf.org. This NSEC record also has a signature:

ietf.org.	1062	IN	RRSIG	NSEC 5 2 1800 20170308083303 20160308073501 40452 ietf.org. homg5NrZIKo0tR+aEp0MVYYjT7J/KGTKP46bJ8eeetbq4KqNvLKJ5Yig ve4RSWFYrSARAmbi3GIFW00P/dFCzDNVlMWYRbcFUt5NfYRJxg25jy95 yHNmInwDUnttmzKuBezdVVvRLJY3qSM7S3VfI/b7n6++ODUFcsL88uNB V6bRO6FOksgE1/jUrtz6/lEKmodWWI2goFPGgmgihqLR8ldv0Dv7k9vy Ao1uunP6kDQEj+omkICFHaT/DBSSYq59DVeMAAcfDq2ssbr4p8hUoXiB tNlJWEubMnHi7YmLSgby+m8b97+8b6qPe8W478gAiggsNjc2gQSKOOXH EejOSA==

All in all, the negative answer for bogus.ietf.org contains an SOA + SOA RRSIG + (2) NSEC + (2) NSEC RRSIG. It is 6 records in total, returning an answer that is 1095 bytes (this is a large DNS answer).

What you may have noticed is that because the negative answer returns the previous and next name, you can keep asking for next names and essentially “walk” the zone until you learn every single name contained in it.

For example, if you ask for the NSEC on ietf.org, you will get back the first name in the zone, ietf1._domainkey.ietf.org:

ietf.org.		1799	IN	NSEC 	ietf1._domainkey.ietf.org.  A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY SPF

Then if you ask for the NSEC on ietf1._domainkey.ietf.org you will get the next name in the zone:

ietf1._domainkey.ietf.org. 1799	IN	NSEC 	apps.ietf.org. TXT RRSIG NSEC

And you can keep going until you get every name in the zone:

apps.ietf.org.		1799	IN	NSEC 	mail.apps.ietf.org. MX RRSIG NSEC

The root zone uses NSEC as well, so you can walk the root to see every TLD:

The root NSEC:

.			21599	IN	NSEC 	aaa. NS SOA RRSIG NSEC DNSKEY

.aaa NSEC:

aaa.			21599	IN	NSEC 	aarp. NS DS RRSIG NSEC

.aarp NSEC:

aarp.			21599	IN	NSEC	 abb. NS DS RRSIG NSEC

Zone walking was actually considered a feature of the original design:

The complete NXT chains specified in this document enable a resolver to obtain, by successive queries chaining through NXTs, all of the names in a zone. - RFC2535

(NXT is the original DNS record type that NSEC was based off of)

However, as you can imagine, this is a terrible idea for some zones. If you could walk the .gov zone, you could learn every US government agency and government agency portal. If you owned a real estate company where every realtor got their own subdomain, a competitor could walk through your zone and find out who all of your realtors are.

CC BY 2.0 image by KIUI

So the DNS community rallied together and found a solution. They would continue to return previous and next names, but they would hash the outputs. This was defined in an upgrade to NSEC called NSEC3.

6rmo7l6664ki2heho7jtih1lea9k6los.icann.org. 3599 IN NSEC3 1 0 5 2C21FAE313005174 6S2J9F2OI56GPVEIH3KBKJGGCL21SKKL A RRSIG

NSEC3 was a “close but no cigar” solution to the problem. While it’s true that it made zone walking harder, it did not make it impossible. Zone walking with NSEC3 is still possible with a dictionary attack. An attacker can use a list of the most common hostnames, hash them with the hashing algorithm used in the NSEC3 record (which is listed in the record itself) and see if there are any matches. Even if the domain owner uses a salt on the hash, the length of the salt is included in the NSEC3 record, so there are a finite number of salts to guess.

The Salt Length field defines the length of the salt in octets, ranging in value from 0 to 255.”

• RFC5155

If you recall from above, NODATA is the response from a server when it is asked for a name that exists, but not in the requested type (like an MX record for blog.cloudflare.com). NODATA is similar in output to NXDOMAIN. It still requires SOA, but it only takes one NSEC record to prove the next name, and to specify which types do exist on the queried name.

For example, if you look for a TXT record on apps.ietf.org, the NSEC record will tell you that while there is no TXT record on apps.ietf.org, there are MX, RRSIG and NSEC records.

apps.ietf.org.		1799	IN	NSEC 	mail.apps.ietf.org. MX RRSIG NSEC

There are two problems with negative answers:

The first is that the authoritative server needs to return the previous and next name. As you’ll see, this is computationally expensive for CloudFlare, and as you’ve already seen, it can leak information about a zone.

The second is that negative answers require two NSEC records and their two subsequent signatures (or three NSEC3 records and three NSEC3 signatures) to authenticate the nonexistence of one name. This means that answers are bigger than they need to be.

CloudFlare has a custom in house DNS server built in Go called RRDNS. What's unique about RRDNS is that unlike standard DNS servers, it does not have the concept of a zone file. Instead, it has a key value store that holds all of the DNS records of all of the domains. When it gets a query for a record, it can just pick out the record that it needs.

Another unique aspect of CloudFlare's DNS is that a lot of our business logic is handled in the DNS. We often dynamically generate DNS answers on the fly, so we don't always know what we will respond with before we are asked.

Traditional negative answers require the authoritative server to return the previous and next name of a missing name. Because CloudFlare does not have the full view of the zone file, we'd have to ask the database to do a sorted search just to figure out the previous and next names. Beyond that, because we generate answers on the fly, we don’t have a reliable way to know what might be the previous and next name, unless we were to precompute every possible option ahead of time.

One proposed solution to the previous and next name, and secrecy problems is RFC4470, dubbed 'White Lies'. This RFC proposes that DNS operators make up a previous and next name by randomly generating names that are canonically slightly before and after the requested name.

White lies is a great solution to block zone walking (and it helps us prevent unnecessary database lookups), but it still requires 2 NSEC records (one for previous and next name and another for the wildcard) to say one thing, so the answer is still bigger than it needs to be.

We decided to take lying in negative answers to its fullest extent. Instead of white lies, we do black lies.

For an NXDOMAIN, we always return \000.(the missing name) as the next name, and because we return an NSEC directly on the missing name, we do not have to return an additional NSEC for the wildcard. This way we only have to return SOA, SOA RRSIG, NSEC and NSEC RRSIG, and we do not need to search the database or precompute dynamic answers.

Our negative answers are usually around 300 bytes. For comparison, negative answers for ietf.org which uses NSEC and icann.org, which uses NSEC3 are both slightly over 1000 bytes, three times the size. The reason this matters so much is that the maximum size of an unsigned UDP packet is typically 512 octets. DNSSEC requires support for at least 1220 octets long messages over UDP, but above that limit, the client may need to upgrade to DNS over TCP. A good practice is to keep enough headroom in order to keep response sizes below fragmentation threshold during zone signing key rollover periods.

NSEC: 1096 bytes

ietf.org.		1799	IN	SOA	ns0.amsl.com. glen.amsl.com. 1200000317 1800 1800 604800 1800
ietf.org.		1799	IN	RRSIG	SOA 5 2 1800 20170213210533 20160214200831 40452 ietf.org. P8XoJx+SK5nUZAV/IqiJrsoKtP1c+GXmp3FvEOUZPFn1VwW33242LVrJ GMI5HHjMEX07EzOXZyLnQeEvlf2QLxRIQm1wAnE6W4SUp7TgKUZ7NJHP dgLr2gqKYim4CI7ikYj3vK7NgcaSE5jqIZUm7oFxxYO9/YPz4Mx7COw6 XBOMYS2v8VY3DICeJdZsHJnVKlgl8L7/yqrL8qhkSW1yDo3YtB9cZEjB OVk8uRDxK7aHkEnMRz0LODOJ10AngJpg9LrkZ1CO444RhZGgTbwzN9Vq rDyH47Cn3h8ofEOJtYCJvuX5CCzaZDInBsjq9wNAiNBgIQatPkNriR77 hCEHhQ==
ietf.org.		1799	IN	NSEC	ietf1._domainkey.ietf.org. A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY SPF
ietf.org.		1799	IN	RRSIG	NSEC 5 2 1800 20170213210816 20160214200831 40452 ietf.org. B9z/JJs30tkn0DyxVz0zaRlm4HkeNY1TqYmr9rx8rH7kC32PWZ1Fooy6 16qmB33/cvD2wtOCKMnNQPdTG2qUs/RuVxqRPZaQojIVZsy/GYONmlap BptzgOJLP7/HOxgYFgMt5q/91JHfp6Mn0sd218/H86Aa98RCXwUOzZnW bdttjsmbAqONuPQURaGz8ZgGztFmQt5dNeNRaq5Uqdzw738vQjYwppfU 9GSLkT7RCh3kgbNcSaXeuWfFnxG1R2SdlRoDICos+RqdDM+23BHGYkYc /NEBLtjYGxPqYCMe/7lOtWQjtQOkqylAr1r7pSI2NOA9mexa7yTuXH+x o/rzRA==
www.apps.ietf.org.	1799	IN	NSEC	cloudflare-verify.ietf.org. A RRSIG NSEC
www.apps.ietf.org.	1799	IN	RRSIG	NSEC 5 4 1800 20170213210614 20160214200831 40452 ietf.org. U+hEHcTps2IC8VKS61rU3MDZq+U0KG4/oJjIHVYbrWufQ7NdMdnY6hCL OmQtsvuZVRQjWHmowRhMj83JMUagxoZuWTg6GuLPin3c7PkRimfBx7jI wjqORwcuvpBh92A/s/2HXBma3PtDZl2UDLy4z7wdO62rbxGU/LX1jTqY FoJJLJfJ/C+ngVMIE/QVneXSJkAjHV96FSEnreF81V62x9azv3AHo4tl qnoYvRDtK+cR072A5smtWMKDfcIr2fI11TAGIyhR55yAiollPDEz5koj BfMstC/JXVURJMM+1vCPjxvwYzTZN8iICf1AupyyR8BNWxgic5yh1ljH 1AuAVQ==

Black Lies: 357 bytes

cloudflare.com.		1799	IN	SOA	ns3.cloudflare.com. dns.cloudflare.com. 2020742566 10000 2400 604800 3600
blog.cloudflare.com.	3599	IN	NSEC	\000.blog.cloudflare.com. RRSIG NSEC
cloudflare.com.		1799	IN	RRSIG	SOA 13 2 86400 20160220230013 20160218210013 35273 cloudflare.com. kgjtJDuuNC/yX8yWQpol4ZUUr8s8yAXZi26KWBI6S3HDtry2t6LnP1ou QK10Ut7DXO/XhyZddRBVj3pIpWYdBQ==
blog.cloudflare.com.	3599	IN	RRSIG	NSEC 13 3 3600 20160220230013 20160218210013 35273 cloudflare.com. 8BKAAS8EXNJbm8DxEI1OOBba8KaiimIuB47mPlteiZf3sVLGN1edsrXE +q+pHaSHEfYG5mHfCBJrbi6b3EoXOw==

Our take on NODATA responses is also unique. Traditionally, NODATA responses contain one NSEC record to tell the resolver which types exist on the requested name. This is highly inefficient. To do this, we’d have to search the database for all the types that do exist, just to answer that the requested type does not exist. Remember that’s not even always possible because we have dynamic answers that are generated on the fly.

What we realized was that NSEC is a denial of existence. What matters in NSEC are the missing types, not the present ones. So what we do is we set all the types. We say, this name does exist, just not on the one type you asked for.

For example, if you asked for an TXT record of blog.cloudflare.com we would say, all the types exist, just not TXT.

blog.cloudflare.com.	3599	IN	NSEC	\000.blog.cloudflare.com. A WKS HINFO MX AAAA LOC SRV CERT SSHFP IPSECKEY RRSIG NSEC TLSA HIP OPENPGPKEY SPF

And then if you queried for a MX on blog.cloudflare.com, we would return saying we have every record type, even TXT, but just not MX.

blog.cloudflare.com.	3599	IN	NSEC	\000.blog.cloudflare.com. A WKS HINFO TXT AAAA LOC SRV CERT SSHFP IPSECKEY RRSIG NSEC TLSA HIP OPENPGPKEY SPF

This saves us a database lookup and from leaking any zone information in negative answers. We call this the DNS Shotgun.

We put a lot of care to ensure CloudFlare’s negative answers are standards compliant. We’re even pushing for them to become an Internet Standard by publishing an Internet Draft earlier this year.

RFC4470, White Lies, allows us to randomly generate next names in NSEC. Not setting the second NSEC for the wildcard subdomain is also allowed, so long as there exists an NSEC record on the actual queried name. And lastly, our lie of setting every record type in NSEC records for NODATA, is okay too –– after all, domains are constantly changing, it’s feasible that the zone file changed in between the time the NSEC record indicated to you there was no MX record on blog.cloudflare.com and when you queried successfully for MX on blog.cloudflare.com.

We’re proud of our negative answers. They help us keep packet size small, and CPU consumption low enough for us to provide DNSSEC for free for any domain. Let us know what you think, we’re looking forward to hearing from you.

CloudFlare and Gandi have been hosting a speaker series on DNS, previously bringing in the founder of DNS Paul Mockapetris and Dan Kaminsky, who uncovered one of the most critical vulnerabilities in DNS.

Our third and final talk is coming up on June 21st at 6PM PST at the Gandi office in San Francisco (live stream link will be posted on the Meetup page) and you won’t want to miss it, because our speaker is none other than Paul Vixie.

Paul wrote most of BIND Version 8 and hired the team who wrote BIND Version 9, the most widely used DNS software on the Internet. He founded ISC, home of BIND and F-root and later operated C-root.

We'll talk about alternative DNS root servers, email security and spam (Paul founded the first ever anti-spam company), and what needs to be done about DNS and security (Paul added most of BIND’s current security systems).

So come grab a beer and let's geek out about DNS one more time.

RSVP here.