Programmable Flow Protection is currently in beta and available to all Magic Transit Enterprise customers for an additional cost. Contact your account team to join the beta or sign up at this page.

Our existing DDoS mitigation systems have been designed to understand and protect popular, well-known protocols from DDoS attacks. For example, our Advanced TCP Protection system uses specific known characteristics about the TCP protocol to issue challenges and establish a client’s legitimacy. Similarly, our Advanced DNS Protection builds a per-customer profile of DNS queries to mitigate DNS attacks. Our generic DDoS mitigation platform also understands common patterns across a variety of other well known protocols, including NTP, RDP, SIP, and many others.

However, custom or proprietary UDP protocols have always been a challenge for Cloudflare’s DDoS mitigation systems because our systems do not have the relevant protocol knowledge to make intelligent decisions to pass or drop traffic. 

Programmable Flow Protection addresses this gap. Now, customers can write their own eBPF program that defines what “good” and “bad” packets are and how to deal with them. Cloudflare then runs the program across our entire global network. The program can choose to either drop or challenge “bad” packets, preventing them from reaching the customer’s origin. 

UDP is a connectionless transport layer protocol. Unlike TCP, UDP has no handshake or stateful connections. It does not promise that packets will arrive in order or exactly once. UDP instead prioritizes speed and simplicity, and is therefore well-suited for online gaming, VoIP, video streaming, and any other use case where the application requires real-time communication between clients and servers.

Our DDoS mitigation systems have always been able to detect and mitigate attacks against well-known protocols built on top of UDP. For example, the standard DNS protocol is built on UDP, and each DNS packet has a well-known structure. If we see a DNS packet, we know how to interpret it. That makes it easier for us to detect and drop DNS-based attacks. 

Unfortunately, if we don’t understand the protocol inside a UDP packet’s payload, our DDoS mitigation systems have limited options available at mitigation time. If an attacker sends a large flood of UDP traffic that does not match any known patterns or protocols, Cloudflare can either entirely block or apply a rate limit to the destination IP and port combination. This is a crude “last line of defense” that is only intended to keep the rest of the customer’s network online, and it can be painful in a couple ways. 

First, a block or a generic rate limit does not distinguish good traffic from bad, which means these mitigations will likely cause legitimate clients to experience lag or connection loss — doing the attacker’s job for them! Second, a generic rate limit can be too strict or too lax depending on the customer. For example, a customer who expects to receive 1Gbps of legitimate traffic probably needs more aggressive rate limiting compared to a customer who expects to receive 25Gbps of legitimate traffic.

^{An illustration of UDP packet contents. A user can define a valid payload and reject traffic that doesn’t match the defined pattern.}

The Programmable Flow Protection platform was built to address this problem by allowing our customers to dictate what “good” versus “bad” traffic actually looks like. Many of our customers use custom or proprietary UDP protocols that we do not understand — and now we don’t have to.

In previous blog posts, we’ve described how “flowtrackd”, our stateful network layer DDoS mitigation system, protects Magic Transit users from complex TCP and DNS attacks. We’ve also described how we use Linux technologies like XDP and eBPF to efficiently mitigate common types of large scale DDoS attacks. 

Programmable Flow Protection combines these technologies in a novel way. With Programmable Flow Protection, a customer can write their own eBPF program that decides whether to pass, drop, or challenge individual packets based on arbitrary logic. A customer can upload the program to Cloudflare, and Cloudflare will execute it on every packet destined to their network. Programs are executed in userspace, not kernel space, which allows Cloudflare the flexibility to support a variety of customers and use cases on the platform without compromising security. Programmable Flow Protection programs run after all of Cloudflare’s existing DDoS mitigations, so users still benefit from our standard security protections. 

There are many similarities between an XDP eBPF program loaded into the Linux kernel and an eBPF program running on the Programmable Flow Protection platform. Both types of programs are compiled down to BPF bytecode. They are both run through a “verifier” to ensure memory safety and verify program termination. They are also executed in a fast, lightweight VM to provide isolation and stability.

However, eBPF programs loaded into the Linux kernel make use of many Linux-specific “helper functions” to integrate with the network stack, maintain state between program executions, and emit packets to network devices. Programmable Flow Protection offers the same functionality whenever a customer chooses, but with a different API tailored specifically to implement DDoS mitigations. For example, we’ve built helper functions to store state about clients between program executions, perform cryptographic validation, and emit challenge packets to clients. With these helper functions, a developer can use the power of the Cloudflare platform to protect their own network.

Let’s step through an example to illustrate how a customer’s protocol-specific knowledge can be combined with Cloudflare’s network to create powerful mitigations.

Say a customer hosts an online gaming server on UDP port 207. The game engine uses a proprietary application header that is specific to the game. Cloudflare has no knowledge of the structure or contents of the application header. The customer gets hit by DDoS attacks that overwhelm the game server and players report lag in gameplay. The attack traffic comes from highly randomized source IPs and ports, and the payload data appears to be random as well. 

To mitigate the attack, the customer can use their knowledge of the application header and deploy a Programmable Flow Protection program to check a packet’s validity. In this example, the application header contains a token that is unique to the gaming protocol. The customer can therefore write a program to extract the last byte of the token. The program passes all packets with the correct value present and drops all other traffic:

#include <linux/ip.h>
#include <linux/udp.h>
#include <arpa/inet.h>

#include "cf_ebpf_defs.h"
#include "cf_ebpf_helper.h"

// Custom application header
struct apphdr {
    uint8_t  version;
    uint16_t length;   // Length of the variable-length token
    uint8_t  token[0]; // Variable-length token
} __attribute__((packed));

uint64_t
cf_ebpf_main(void *state)
{
    struct cf_ebpf_generic_ctx *ctx = state;
    struct cf_ebpf_parsed_headers headers;
    struct cf_ebpf_packet_data *p;

    // Parse the packet headers with provided helper function
    if (parse_packet_data(ctx, &p, &headers) != 0) {
        return CF_EBPF_DROP;
    }

    // Drop packets not destined to port 207
    struct udphdr *udp_hdr = (struct udphdr *)headers.udp;
    if (ntohs(udp_hdr->dest) != 207) {
        return CF_EBPF_DROP;
    }

    // Get application header from UDP payload
    struct apphdr *app = (struct apphdr *)(udp_hdr + 1);
    if ((uint8_t *)(app + 1) > headers.data_end) {
        return CF_EBPF_DROP;
    }

    // Perform memory checks to satisfy the verifier
    // and access the token safely
    if ((uint8_t *)(app->token + token_len) > headers.data_end) {
        return CF_EBPF_DROP;
    }

    // Check the last byte of the token against expected value
    uint8_t *last_byte = app->token + token_len - 1;
    if (*last_byte != 0xCF) {
        return CF_EBPF_DROP;
    }

    return CF_EBPF_PASS;
}

^{An eBPF program to filter packets according to a value in the application header.}

This program leverages application-specific information to create a more targeted mitigation than Cloudflare is capable of crafting on its own. Customers can now combine their proprietary knowledge with the capacity of Cloudflare’s global network to absorb and mitigate massive attacks better than ever before.

Many pattern checks, like the one performed in the example above, can be accomplished with traditional firewalls. However, programs provide useful primitives that are not available in firewalls, including variables, conditional execution, loops, and procedure calls. But what really sets Programmable Flow Protection apart from other solutions is its ability to statefully track flows and challenge clients to prove they are real. A common type of attack that showcases these abilities is a replay attack.

In a replay attack, an attacker repeatedly sends packets that were valid at some point, and therefore conform to expected patterns of the traffic, but are no longer valid in the application’s current context. For example, the attacker could record some of their valid gameplay traffic and use a script to duplicate and transmit the same traffic at a very high rate.

With Programmable Flow Protection, a user can deploy a program that challenges suspicious clients and drops scripted traffic. We can extend our original example as follows:

#include <linux/ip.h>
#include <linux/udp.h>
#include <arpa/inet.h>

#include "cf_ebpf_defs.h"
#include "cf_ebpf_helper.h"

uint64_t
cf_ebpf_main(void *state)
{
    // ...
 
    // Get the status of this source IP (statefully tracked)
    uint8_t status;
    if (cf_ebpf_get_source_ip_status(&status) != 0) {
        return CF_EBPF_DROP;
    }

    switch (status) {
        case NONE:
		// Issue a custom challenge to this source IP
             issue_challenge();
             cf_ebpf_set_source_ip_status(CHALLENGED);
             return CF_EBPF_DROP;


        case CHALLENGED:
		// Check if this packet passes the challenge
		// with custom logic
             if (verify_challenge()) {
                 cf_ebpf_set_source_ip_status(VERIFIED);
                 return CF_EBPF_PASS;
             } else {
                 cf_ebpf_set_source_ip_status(BLOCKED);
                 return CF_EBPF_DROP;
             }


        case VERIFIED:
		// This source IP has passed the challenge
		return CF_EBPF_PASS;

	 case BLOCKED:
		// This source IP has been blocked
		return CF_EBPF_DROP;

        default:
            return CF_EBPF_PASS;
    }


    return CF_EBPF_PASS;
}

^{An eBPF program to challenge UDP connections and statefully manage connections. This example has been simplified for illustration purposes.}

The program statefully tracks the source IP addresses it has seen and emits a packet with a cryptographic challenge back to unknown clients. A legitimate client running a valid gaming client is able to correctly solve the challenge and respond with proof, but the attacker’s script is not. Traffic from the attacker is marked as “blocked” and subsequent packets are dropped.

With these new abilities, customers can statefully track flows and make sure only real, verified clients can send traffic to their origin servers. Although we have focused the example on gaming, the potential use cases for this technology extend to any UDP-based protocol.

We’re excited to offer the Programmable Flow Protection feature to Magic Transit Enterprise customers. Talk to your account manager to learn more about how you can enable Programmable Flow Protection to help keep your infrastructure safe.

We’re still in active development of the platform, and we’re excited to see what our users build next. If you are not yet a Cloudflare customer, let us know if you’d like to protect your network with Cloudflare and Programmable Flow Protection by signing up at this page: https://www.cloudflare.com/lp/programmable-flow-protection/.

^{A mitigated 3.8 Terabits per second DDoS attack that lasted 65 seconds}

^{A mitigated 2.14 billion packet per second DDoS attack that lasted 60 seconds}

Cloudflare customers using Cloudflare’s HTTP reverse proxy services (e.g. Cloudflare WAF and Cloudflare CDN) are automatically protected.

Cloudflare customers using Spectrum and Magic Transit are also automatically protected. Magic Transit customers can further optimize their protection by deploying Magic Firewall rules to enforce a strict positive and negative security model at the packet layer.

The scale and frequency of these attacks are unprecedented. Due to their sheer size and bits/packets per second rates, these attacks have the ability to take down unprotected Internet properties, as well as Internet properties that are protected by on-premise equipment or by cloud providers that just don’t have sufficient network capacity or global coverage to be able to handle these volumes alongside legitimate traffic without impacting performance. 

Cloudflare, however, does have the network capacity, global coverage, and intelligent systems needed to absorb and automatically mitigate these monstrous attacks. 

In this blog post, we will review the attack campaign and why its attacks are so severe. We will describe the anatomy of a Layer 3/4 DDoS attack, its objectives, and how attacks are generated. We will then proceed to detail how Cloudflare’s systems were able to autonomously detect and mitigate these monstrous attacks without impacting performance for our customers. We will describe the key aspects of our defenses, starting with how our systems generate real-time (dynamic) signatures to match the attack traffic all the way to how we leverage kernel features to drop packets at wire-speed.

We have observed this attack campaign targeting multiple customers in the financial services, Internet, and telecommunication industries, among others. This attack campaign targets bandwidth saturation as well as resource exhaustion of in-line applications and devices.

The attacks predominantly leverage UDP on a fixed port, and originated from across the globe with larger shares coming from Vietnam, Russia, Brazil, Spain, and the US. 

The high packet rate attacks appear to originate from multiple types of compromised devices, including MikroTik devices, DVRs, and Web servers, orchestrated to work in tandem and flood the target with exceptionally large volumes of traffic. The high bitrate attacks appear to originate from a large number of compromised ASUS home routers, likely exploited using a CVE 9.8 (Critical) vulnerability that was recently discovered by Censys.

^{Share of packets observed by source location}

Before we discuss how Cloudflare automatically detected and mitigated the largest DDoS attacks ever seen, it‘s important to understand the basics of DDoS attacks. 

^{Simplified diagram of a DDoS attack}

The goal of a Distributed Denial of Service (DDoS) attack is to deny legitimate users access to a service. This is usually done by exhausting resources needed to provide the service. In the context of these recent Layer 3/4 DDoS attacks, that resource is CPU cycles and network bandwidth.

Processing a packet consumes CPU cycles. In the case of regular (non-attack) traffic, a legitimate packet received by a service will cause that service to perform some action, and different actions require different amounts of CPU processing. However, before a packet is even delivered to a service there is per-packet work that needs to be done. Layer 3 packet headers need to be parsed and processed to deliver the packet to the correct machine and interface. Layer 4 packet headers need to be processed and routed to the correct socket (if any). There may be multiple additional processing steps that inspect every packet. Therefore, if an attacker sends at a high enough packet rate, then they can potentially saturate the available CPU resources, denying service to legitimate users.

To defend against high packet rate attacks, you need to be able to inspect and discard the bad packets using as few CPU cycles as possible, leaving enough CPU to process the good packets. You can additionally acquire more, or faster, CPUs to perform the processing — but that can be a very lengthy process that bears high costs. 

Network bandwidth is the total amount of data per time that can be delivered to a server. You can think of bandwidth like a pipe to transport water. The amount of water we can deliver through a drinking straw is less than what we could deliver through a garden hose. If an attacker is able to push more garbage data into the pipe than it can deliver, then both the bad data and the good data will be discarded upstream, at the entrance to the pipe, and the DDoS is therefore successful.

Defending against attacks that can saturate network bandwidth can be difficult because there is very little that can be done if you are on the downstream side of the saturated pipe. There are really only a few choices: you can get a bigger pipe, you can potentially find a way to move the good traffic to a new pipe that isn’t saturated, or you can hopefully ask the upstream side of the pipe to stop sending some or all of the data into the pipe.

If we think about what this means from the attackers point of view you realize there are similar constraints. Just as it takes CPU cycles to receive a packet, it also takes CPU cycles to create a packet. If, for example, the cost to send and receive a packet were equal, then the attacker would need an equal amount of CPU power to generate the attack as we would need to defend against it. In most cases this is not true — there is a cost asymmetry, as the attacker is able to generate packets using fewer CPU cycles than it takes to receive those packets. However, it is worth noting that generating attacks is not free and can require a large amount of CPU power.

Saturating network bandwidth can be even more difficult for an attacker. Here the attacker needs to be able to output more bandwidth than the target service has allocated. They actually need to be able to exceed the capacity of the receiving service. This is so difficult that the most common way to achieve a network bandwidth attack is to use a reflection/amplification attack method, for example a DNS Amplification attack. These attacks allow the attacker to send a small packet to an intermediate service, and the intermediate service will send a large packet to the victim.

In both scenarios, the attacker needs to acquire or gain access to many devices to generate the attack. These devices can be acquired in a number of different ways. They may be server class machines from cloud providers or hosting services, or they can be compromised devices like DVRs, routers, and webcams that have been infected with the attacker’s malware. These machines together form the botnet.

Now that we understand the fundamentals of how DDoS attacks work, we can explain how Cloudflare can defend against these attacks.

The first not-so-secret ingredient is that Cloudflare’s network is built on anycast. In brief, anycast allows a single IP address to be advertised by multiple machines around the world. A packet sent to that IP address will be served by the closest machine. This means when an attacker uses their distributed botnet to launch an attack, the attack will be received in a distributed manner across the Cloudflare network. An infected DVR in Dallas, Texas will send packets to a Cloudflare server in Dallas. An infected webcam in London will send packets to a Cloudflare server in London.

^{Anycast vs. Unicast networks}

Our anycast network additionally allows Cloudflare to allocate compute and bandwidth resources closest to the regions that need them the most. Densely populated regions will generate larger amounts of legitimate traffic, and the data centers placed in those regions will have more bandwidth and CPU resources to meet those needs. Sparsely populated regions will naturally generate less legitimate traffic, so Cloudflare data centers in those regions can be sized appropriately. Since attack traffic is mainly coming from compromised devices, those devices will tend to be distributed in a manner that matches normal traffic flows sending the attack traffic proportionally to datacenters that can handle it. And similarly, within the datacenter, traffic is distributed across multiple machines.

Additionally, for high bandwidth attacks, Cloudflare’s network has another advantage. A large proportion of traffic on the Cloudflare network does not consume bandwidth in a symmetrical manner. For example, an HTTP request to get a webpage from a site behind Cloudflare will be a relatively small incoming packet, but produce a larger amount of outgoing traffic back to the client. This means that the Cloudflare network tends to egress far more legitimate traffic than we receive. However, the network links and bandwidth allocated are symmetrical, meaning there is an abundance of ingress bandwidth available to receive volumetric attack traffic.

By the time you’ve reached an individual server inside a datacenter, the bandwidth of the attack has been distributed enough that none of the upstream links are saturated. That doesn’t mean the attack has been fully stopped yet, since we haven’t dropped the bad packets. To do that, we need to sample traffic, qualify an attack, and create rules to block the bad packets. 

Sampling traffic and dropping bad packets is the job of our l4drop component, which uses XDP (eXpress Data Path) and leverages an extended version of the Berkeley Packet Filter known as eBPF (extended BPF). This enables us to execute custom code in kernel space and process (drop, forward, or modify) each packet directly at the network interface card (NIC) level. This component helps the system drop packets efficiently without consuming excessive CPU resources on the machine. 

^{Cloudflare DDoS protection system overview}

We use XDP to sample packets to look for suspicious attributes that indicate an attack. The samples include fields such as the source IP, source port, destination IP, destination port, protocol, TCP flags, sequence number, options, packet rate and more. This analysis is conducted by the denial of service daemon (dosd). Dosd holds our secret sauce. It has many filters which instruct it, based on our curated heuristics, when to initiate mitigation. To our customers, these filters are logically grouped by attack vectors and exposed as the DDoS Managed Rules. Our customers can customize their behavior to some extent, as needed.

As it receives samples from XDP, dosd will generate multiple permutations of fingerprints for suspicious traffic patterns. Then, using a data streaming algorithm, dosd will identify the most optimal fingerprints to mitigate the attack. Once an attack is qualified, dosd will push a mitigation rule inline as an eBPF program to surgically drop the attack traffic. 

The detection and mitigation of attacks by dosd is done at the server level, at the data center level and at the global level — and it’s all software defined. This makes our network extremely resilient and leads to almost instant mitigation. There are no out-of-path “scrubbing centers” or “scrubbing devices”. Instead, each server runs the full stack of the Cloudflare product suite including the DDoS detection and mitigation component. And it is all done autonomously. Each server also gossips (multicasts) mitigation instructions within a data center between servers, and globally between data centers. This ensures that whether an attack is localized or globally distributed, dosd will have already installed mitigation rules inline to ensure a robust mitigation.

Our software-defined, autonomous DDoS detection and mitigation systems run across our entire network. In this post we focused mainly on our dynamic fingerprinting capabilities, but our arsenal of defense systems is much larger. The Advanced TCP Protection system and Advanced DNS Protection system work alongside our dynamic fingerprinting to identify sophisticated and highly randomized TCP-based DDoS attacks and also leverages statistical analysis to thwart complex DNS-based DDoS attacks. Our defenses also incorporate real-time threat intelligence, traffic profiling, and machine learning classification as part of our Adaptive DDoS Protection to mitigate traffic anomalies. 

Together, these systems, alongside the full breadth of the Cloudflare Security portfolio, are built atop of the Cloudflare network — one of the largest networks in the world — to ensure our customers are protected from the largest attacks in the world.

We're proud to introduce the Advanced DNS Protection system, a robust defense mechanism designed to protect against the most sophisticated DNS-based DDoS attacks. This system is engineered to provide top-tier security, ensuring your digital infrastructure remains resilient in the face of evolving threats.

Our existing systems have been successfully detecting and mitigating ‘simpler’ DDoS attacks against DNS, but they’ve struggled with the more complex ones. The Advanced DNS Protection system is able to bridge that gap by leveraging new techniques that we will showcase in this blog post.

Advanced DNS Protection is currently in beta and available for all Magic Transit customers at no additional cost. Read on to learn more about DNS DDoS attacks, how the new system works, and what new functionality is expected down the road.

Register your interest to learn more about how we can help keep your DNS servers protected, available, and performant.

Distributed Denial of Service (DDoS) attacks are a type of cyber attack that aim to disrupt and take down websites and other online services. When DDoS attacks succeed and websites are taken offline, it can lead to significant revenue loss and damage to reputation.

Distribution of DDoS attack types for 2023

One common way to disrupt and take down a website is to flood its servers with more traffic than it can handle. This is known as an HTTP flood attack. It is a type of DDoS attack that targets the website directly with a lot of HTTP requests. According to our last DDoS trends report, in 2023 our systems automatically mitigated 5.2 million HTTP DDoS attacks — accounting for 37% of all DDoS attacks.

Diagram of an HTTP flood attack

However, there is another way to take down websites: by targeting them indirectly. Instead of flooding the website servers, the threat actor floods the DNS servers. If the DNS servers are overwhelmed with more queries than their capacity, hostname to IP address translation fails and the website experiences an indirectly inflicted outage because the DNS server cannot respond to legitimate queries.

One notable example is the attack that targeted Dyn, a DNS provider, in October 2016. It was a devastating DDoS attack launched by the infamous Mirai botnet. It caused disruptions for major sites like Airbnb, Netflix, and Amazon, and it took Dyn an entire day to restore services. That’s a long time for service disruptions that can lead to significant reputation and revenue impact.

Over seven years later, Mirai attacks and DNS attacks are still incredibly common. In 2023, DNS attacks were the second most common attack type — with a 33% share of all DDoS attacks (4.6 million attacks). Attacks launched by Mirai-variant botnets were the fifth most common type of network-layer DDoS attack, accounting for 3% of all network-layer DDoS attacks.

Diagram of a DNS query flood attack

DNS-based DDoS attacks can be easier to mitigate when there is a recurring pattern in each query. This is what’s called the “attack fingerprint”. Fingerprint-based mitigation systems can identify those patterns and then deploy a mitigation rule that surgically filters the attack traffic without impacting legitimate traffic.

For example, let’s take a scenario where an attacker sends a flood of DNS queries to their target. In this example, the attacker only randomized the source IP address. All other query fields remained consistent. The mitigation system detected the pattern (source port is 1024 and the queried domain is example.com) and will generate an ephemeral mitigation rule to filter those queries.

A simplified diagram of the attack fingerprinting concept

However, there are DNS-based DDoS attacks that are much more sophisticated and randomized, lacking an apparent attack pattern. Without a consistent pattern to lock on to, it becomes virtually impossible to mitigate the attack using a fingerprint-based mitigation system. Moreover, even if an attack pattern is detected in a highly randomized attack, the pattern would probably be so generic that it would mistakenly mitigate legitimate user traffic and/or not catch the entire attack.

In this example, the attacker also randomized the queried domain in their DNS query flood attack. Simultaneously, a legitimate client (or server) is also querying example.com. They were assigned a random port number which happened to be 1024. The mitigation system detected a pattern (source port is 1024 and the queried domain is example.com) that caught only the part of the attack that matched the fingerprint. The mitigation system missed the part of the attack that queried other hostnames. Lastly, the mitigation system mistakenly caught legitimate traffic that happened to appear similar to the attack traffic.

A simplified diagram of a randomized DNS flood attack

This is just one very simple example of how fingerprinting can fail in stopping randomized DDoS attacks. This challenge is amplified when attackers “launder” their attack traffic through reputable public DNS resolvers (a DNS resolver, also known as a recursive DNS server, is a type of DNS server that is responsible for tracking down the IP address of a website from various other DNS servers). This is known as a DNS laundering attack.

Diagram of the DNS resolution process

During a DNS laundering attack, the attacker queries subdomains of a real domain that is managed by the victim’s authoritative DNS server. The prefix that defines the subdomain is randomized and is never used more than once. Due to the randomization element, recursive DNS servers will never have a cached response and will need to forward the query to the victim’s authoritative DNS server. The authoritative DNS server is then bombarded by so many queries until it cannot serve legitimate queries or even crashes altogether.

Diagram of a DNS Laundering attack

The complexity of sophisticated DNS DDoS attacks lies in their paradoxical nature: while they are relatively easy to detect, effectively mitigating them is significantly more difficult. This difficulty stems from the fact that authoritative DNS servers cannot simply block queries from recursive DNS servers, as these servers also make legitimate requests. Moreover, the authoritative DNS server is unable to filter queries aimed at the targeted domain because it is a genuine domain that needs to remain accessible.

The rise in these types of sophisticated DNS-based DDoS attacks motivated us to develop a new solution — a solution that would better protect our customers and bridge the gap of more traditional fingerprinting approaches. This solution came to be the Advanced DNS Protection system. Similar to the Advanced TCP Protection system, it is a software-defined system that we built, and it is powered by our stateful mitigation platform, flowtrackd (flow tracking daemon).

The Advanced DNS Protection system complements our existing suite of DDoS defense systems. Following the same approach as our other DDoS defense systems, the Advanced DNS Protection system is also a distributed system, and an instance of it runs on every Cloudflare server around the world. Once the system has been initiated, each instance can detect and mitigate attacks autonomously without requiring any centralized regulation. Detection and mitigation is instantaneous (zero seconds). Each instance also communicates with other instances on other servers in a data center. They gossip and share threat intelligence to deliver a comprehensive mitigation within each data center.

Screenshots from the Cloudflare dashboard showcasing a DNS-based DDoS attack that was mitigated by the Advanced DNS Protection system 

Together, our fingerprinting-based systems (the DDoS protection managed rulesets) and our stateful mitigation systems provide a robust multi-layered defense strategy to defend against the most sophisticated and randomized DNS-based DDoS attacks. The system is also customizable, allowing Cloudflare customers to tailor it for their needs. Review our documentation for more information on configuration options.

Diagram of Cloudflare’s DDoS protection systems

We’ve also added new DNS-centric data points to help customers better understand their DNS traffic patterns and attacks. These new data points are available in a new “DNS Protection” tab within the Cloudflare Network Analytics dashboard. The new tab provides insights about which DNS queries are passed and dropped, as well as the characteristics of those queries, including the queried domain name and the record type. The analytics can also be fetched by using the Cloudflare GraphQL API and by exporting logs into your own monitoring dashboards via Logpush.

To protect against sophisticated and highly randomized DNS-based DDoS attacks, we needed to get better at deciding which DNS queries are likely to be legitimate for our customers. However, it’s not easy to infer what’s legitimate and what’s likely to be a part of an attack just based on the query name. We can’t rely solely on fingerprint-based detection mechanisms, since sometimes seemingly random queries, like abc123.example.com, can be legitimate. The opposite is true as well: a query for mailserver.example.com might look legitimate, but can end up not being a real subdomain for a customer.

To make matters worse, our Layer 3 packet routing-based mitigation service, Magic Transit, uses direct server return (DSR), meaning we can not see the DNS origin server’s responses to give us feedback about which queries are ultimately legitimate.

Diagram of Magic Transit with Direct Server Return (DSR)

We decided that the best way to combat these attacks is to build a data model of each customer’s expected DNS queries, based on a historical record that we build. With this model in hand, we can decide with higher confidence which queries are likely to be legitimate, and drop the ones that we think are not, shielding our customer’s DNS servers.

This is the basis of Advanced DNS Protection. It inspects every DNS query sent to our Magic Transit customers, and passes or drops them based on the data model and each customer’s individual settings.

To do so, each server at our global network continually sends certain DNS-related data such as query type (for example, A record) and the queried domains (but not the source of the query) to our core data centers, where we periodically compute DNS query traffic profiles for each customer. Those profiles are distributed across our global network, where they are consulted to help us more confidently and accurately decide which queries are good and which are bad. We drop the bad queries and pass on the good ones, taking into account a customer's tolerance for unexpected DNS queries based on their configurations.

In building this system, we faced several specific technical challenges:

We process tens of millions of DNS queries per day across our global network for our Magic Transit customers, not counting Cloudflare’s suite of other DNS products, and use the DNS-related data mentioned above to build custom query traffic profiles. Analyzing this type of data requires careful treatment of our data pipelines. When building these traffic profiles, we use sample-on-write and adaptive bitrate technologies when writing and reading the necessary data, respectively, to ensure that we capture the data with a fine granularity while protecting our data infrastructure, and we drop information that might impact the privacy of end users.

Some of our customers see tens of millions of DNS queries per day alone. This amount of data would be prohibitively expensive to store and distribute in an uncompressed format. To solve this challenge, we decided to use a counting Bloom filter for each customer’s traffic profile. This is a probabilistic data structure that allows us to succinctly store and distribute each customer’s DNS profile, and then efficiently query it at packet processing time.

We periodically need to recompute and redistribute every customer’s DNS traffic profile between our data centers to each server in our fleet. We used our very own R2 storage service to greatly simplify this task. With regional hints and custom domains enabled, we enabled caching and used only a handful of R2 buckets. Each time we need to update the global view of the customer data models across our edge fleet, 98% of the bits transferred are served from cache.

When new domain names are put into service, our data models will not immediately be aware of them because queries with these names have never been seen before. This and other reasons for potential false positives mandate that we need to build a certain amount of tolerance into the system to allow through potentially legitimate queries. We do so by leveraging token bucket algorithms. Customers can configure the size of the token buckets by changing the sensitivity levels of the Advanced DNS Protection system. The lower the sensitivity, the larger the token bucket — and vice versa. A larger token bucket provides more tolerance for unexpected DNS queries and expected DNS queries that deviate from the profile. A high sensitivity level translates to a smaller token bucket and a stricter approach.

At the end of the day, these are the types of challenges that Cloudflare is excellent at solving. Our customers trust us with handling their traffic, and ensuring their Internet properties are protected, available and performant. We take that trust extremely seriously.

The Advanced DNS Protection system leverages our global infrastructure and data processing capabilities alongside intelligent algorithms and data structures to protect our customers.

If you are not yet a Cloudflare customer, let us know if you’d like to protect your DNS servers. Existing Cloudflare customers can enable the new systems by contacting their account team or Cloudflare Support.