Fine-tuning isn’t an opaque process: customers have to get some data points and then decide what works for them. This post explains the technologies we offer to enable customers to see why the WAF takes certain actions — and the improvements that have been made to reduce noise and increase signal.

Cloudflare’s WAF protects origin servers from different kinds of layer 7 attacks, which are attacks that target the application layer. Protection is provided with various tools like:

• Managed rules, which security analysts at Cloudflare write to address common vulnerabilities and exposures (CVE), OWASP security risks, and vulnerabilities like Log4Shell.

• Custom rules, where customers can write rules with the expressive Rules language.

• Rate limiting rules, malicious uploads detection, leaked credentials detection, etc.

These tools are built on the Rulesets engine. When there is a match on a Rule expression, the engine executes an action.

The Log action is used to simulate the behaviour of rules. This action proves that a rule expression is matched by the engine and emits a log event which can be accessed via Security Analytics, Security Events, Logpush or Edge Log Delivery.

Logs are great at validating a rule works as expected on the traffic it was expected to match, but showing that the rule matches isn’t sufficient, especially when a rule expression can take many code paths. In pseudocode, an expression can look like:

If any of the http request headers contains an “authorization” key OR the lowercased representation of the http host header starts with “cloudflare” THEN log The rules language syntax will be:

any(http.request.headers[*] contains "authorization") or starts_with(lower(http.host), "cloudflare")

Debugging this expression poses a couple of problems. Is it the left-hand side (LHS) or right-hand side (RHS) of the OR expression above that matches? Functions such as Base64 decoding, URL decoding, and in this case lowercasing can apply transformations to the original representation of these fields, which leads to further ambiguity as to which characteristics of the request led to a match.

To further complicate this, many rules in a ruleset can register matches. Rulesets like Cloudflare OWASP use a cumulative score of different rules to trigger an action when the score crosses a set threshold. 

Additionally, the expressions of the Cloudflare Managed and OWASP rules are private. This increases our security posture – but it also means that customers can only guess what these rules do from their titles, tags and descriptions. For instance, one might be labeled “SonicWall SMA - Remote Code Execution - CVE:CVE-2025-32819.”

Which raises questions: What part of my request led to a match in the Rulesets engine? Are these false positives? 

This is where payload logging shines. It can help us drill down to the specific fields and their respective values, post-transformation, in the rule that led to a match. 

Payload logging is a feature that logs which fields in the request are associated with a rule that led to the WAF taking an action. This reduces ambiguity and provides useful information that can help spot check false positives, guarantee correctness, and aid in fine-tuning of these rules for better performance.

From the example above, a payload log entry will contain either the LHS or RHS of the expression, but not both. 

The payload logging and Rulesets engines are built on Wirefilter, which has been explained extensively.

Fundamentally, these engines are objects written in Rust which implement a compiler trait. This trait drives the compilation of the abstract syntax trees (ASTs) derived from these expressions.

struct PayloadLoggingCompiler {
     regex_cache HashMap<String, Arc<Regex>>
}

impl wirefilter::Compiler for PayloadLoggingCompiler {
	type U = PayloadLoggingUserData
	
	fn compile_logical_expr(&mut self, node: LogicalExpr) -> CompiledExpr<Self::U> {
		// ...
		let regex = self.regex_cache.entry(regex_pattern)
		.or_insert_with(|| Arc::new(regex))
		// ...
	}

}

The Rulesets Engine executes an expression and if it evaluates to true, the expression and its execution context are sent to the payload logging compiler for re-evaluation. The execution context provides all the runtime values needed to evaluate the expression.

After re-evaluation is done, the fields involved in branches of the expression that evaluate to true are logged.

The structure of the log is a map of wirefilter fields and their values Map<Field, Value>

{

	“http.host”: “cloudflare.com”,
	“http.method”: “get”,
	“http.user_agent”: “mozilla”

}

Note: These logs are encrypted with the public key provided by the customer. 

These logs go through our logging pipeline and can be read in different ways. Customers can configure a Logpush job to write to a custom Worker we built that uses the customer’s private key to automatically decrypt these logs. The Payload logging CLI tool, Worker, or the Cloudflare dashboard can also be used for decryption.

In wirefilter, some fields are array types. The field http.request.headers.names is an array of all the header names in a request. For example:

[“content-type”, “content-length”, “authorization”, "host"]

An expression that reads any(http.request.headers.names[*] contains “c”) will evaluate to true because at least one of the headers contains the letter “c”. With the previous version of the payload logging compiler, all the headers in the “http.request.headers.names” field will be logged since it's a part of the expression that evaluates to true.  

Payload log (previous)

http.request.headers.names[*] = [“content-type”, “content-length”, “authorization”, "host"]

Now, we partially evaluate the array fields and log the indexes that match the expressions constraint. In this case, it’ll be just the headers that contain a “c”!

Payload log (new)

http.request.headers.names[0,1] = [“content-type”, “content-length”]

This brings us to operators in wirefilter. Some operators like “eq” result in exact matches, e.g. http.host eq “a.com”. There are other operators that result in “partial” matches – like “in”, “contains”, “matches” – that work alongside regexes. The expression in this example: `any(http.request.headers[*] contains “c”)` uses a “contains” operator which produces a partial match. It also uses the “any” function which we can say produces a partial match, because if at least one of the headers contains a “c”, then we should log that header – not all the headers as we did in the previous version.

With the improvements to the payload logging compiler, when these expressions are evaluated, we log just the partial matches. In this case, the new payload logging compiler handles the “contains” operator similarly to the “find” method for bytes in the Rust standard library. This improves our payload log to:

http.request.headers.names[0,1] = [“c”, “c”]

This makes things a lot clearer. It also saves our logging pipeline from processing millions of bytes. For example, a field that is analyzed a lot is the request body — http.request.body.raw — which can be tens of kilobytes in size. Sometimes the expressions are checking for a regex pattern that should match three characters. In this case we’ll be logging 3 bytes instead of kilobytes!

I know, I know, [“c”, “c”] doesn’t really mean much. Even if we’ve provided the exact reason for the match and are significantly saving on the volume of bytes written to our customers storage destinations, the key goal is to provide useful debugging information to the customer. As part of the payload logging improvements, the compiler now also logs a “before” and "after” (if applicable) for partial matches. The size for these buffers are currently 15 bytes each. This means our payload log now looks like:

http.request.headers[0,1] = [
    {
        before: null, // isnt included in the final log
        content: “c”, 
        after: “ontent-length”
    },
    {
        before: null, // isnt included in the final log
        content: “c”, 
        after:”ontent-type”
    }
]

Example of payload log (previous)

Example of payload log (new)

In the previous log, we have all the header values. In the new log, we have the 8th index which is a malicious script in a HTTP header. The match is on the “<script>” tag and the rest is the context which is the text in gray.      

Managed rules rely heavily on regular expressions to fingerprint malicious requests. Parsing and compiling these expressions are CPU-intensive tasks. As managed rules are written once and deployed across millions of zones, we benefit from compiling these regexes and caching them in memory. This saves us CPU cycles as we don’t have to re-compile these until the process restarts.

The Payload logging compiler uses a lot of dynamically sized arrays or vectors to store the intermediate state for these logs. Crates like smallvec are also used to reduce heap allocations.  

Sometimes, customers see “truncated” in their payload logs. This is because every firewall event has a size limit in bytes. When this limit is exceeded, the payload log is truncated. 

Payload log (previous)

Payload log (new)

We have seen the p50 byte size of the payload logs shrink from 1.5 Kilobytes to 500 bytes – a 67% reduction! That means way fewer truncated payload logs.

We’re currently using a lossy representation of utf-8 strings to represent values. This means that non-valid utf-8 strings like multimedia are represented as U+FFFD unicode replacement characters. For rules that will work on binary data, the integrity of these values should be preserved with byte arrays or with a different serialization format.

The storage format for payload logging is JSON. We’ll be benchmarking this alongside other binary formats like CBOR, Cap'n Proto, Protobuf, etc., to see how much processing time this saves our pipeline. This will help us deliver logs to our customers faster, with the added advantage that binary formats can also help with maintaining a defined schema that will be backward compatible. 

Finally, payload logging only works with Managed rules. It will be rolled out to other Cloudflare WAF products like custom rules, WAF attack score, content scanning, Firewall for AI, and more. An example of payload logging showing prompts containing PII, detected by Firewall for AI:

Visibility into the actions taken by the WAF will give customers assurance that their rules or configurations are doing exactly what they expect. Improvements to the specificity of payload logging is a step in this direction — and in the pipeline are further improvements to reliability, latency, and expansion to more WAF products.

As this was a breaking change to the JSON schema, we’ve rolled this out slowly to customers with adequate documentation.

To get started and enable payload logging, visit our developer documentation.

Around three years ago, we brought multiple features into the Firewall tab in our dashboard navigation, with the motivation “to make our products and services intuitive.” With our hard work in expanding capabilities offerings in the past three years, we want to take another opportunity to evaluate the intuitiveness of Cloudflare WAF (Web Application Firewall).

The security landscape is moving fast; types of web applications are growing rapidly; and within the industry there are various approaches to what a WAF includes and can offer. Cloudflare not only proxies enterprise applications, but also millions of personal blogs, community sites, and small businesses stores. The diversity of use cases are covered by various products we offer; however, these products are currently scattered and that makes visibility of active protection rules unclear. This pushes us to reflect on how we can best support our customers in getting the most value out of WAF by providing a clearer offering that meets expectations.

A few months ago, we reached out to our customers to answer a simple question: what do you consider to be part of WAF? We employed a range of user research methods including card sorting, tree testing, design evaluation, and surveys to help with this. The results of this research illustrated how our customers think about WAF, what it means to them, and how it supports their use cases. This inspired the product team to expand scope and contemplate what (Web Application) Security means, beyond merely the WAF.

Based on what hundreds of customers told us, our user research and product design teams collaborated with product management to rethink the security experience. We examined our assumptions and assessed the effectiveness of design concepts to create a structure (or information architecture) that reflected our customers’ mental models.

This new structure consolidates firewall rules, managed rules, and rate limiting rules to become a part of WAF. The new WAF strives to be the one-stop shop for web application security as it pertains to differentiating malicious from clean traffic.

As of today, you will see the following changes to our navigation:

1. Firewall is being renamed to Security.

2. Under Security, you will now find WAF.

3. Firewall rules, managed rules, and rate limiting rules will now appear under WAF.

From now on, when we refer to WAF, we will be referring to above three features.

Further, some important updates are coming for these features. Advanced rate limiting rules will be launched as part of Security Week, and every customer will also get a free set of managed rules to protect all traffic from high profile vulnerabilities. And finally, in the next few months, firewall rules will move to the Ruleset Engine, adding more powerful capabilities thanks to the new Ruleset API. Feeling excited?

Almost 500 customers participated in this user research study that helped us learn about needs and context of use. We employed four research methods, all of which were conducted in an unmoderated manner; this meant people around the world could participate remotely at a time and place of their choosing.

• Card sorting involved participants grouping navigational elements into categories that made sense to them.

• Tree testing assessed how well or poorly a proposed navigational structure performed for our target audience.

• Design evaluation involved a task-based approach to measure effectiveness and utility of design concepts.

• Survey questions helped us dive deeper into results, as well as painting a picture of our participants.

Results of this four-pronged study informed changes to both WAF and Security that are detailed below.

The final result reveals the WAF as part of a broader Security category, which also includes Bots, DDoS, API Shield and Page Shield. This destination enables you to create your rules (a.k.a. firewall rules), deploy Cloudflare managed rules, set rate limit conditions, and includes handy tools to protect your web applications.

All customers across all plans will now see the WAF products organized as below:

1. Firewall rules allow you to create custom, user-defined logic by blocking or allowing traffic that leverages all the components of the HTTP requests and dynamic fields computed by Cloudflare, such as Bot score.

2. Rate limiting rules include the traditional IP-based product we launched back in 2018 and the newer Advanced Rate Limiting for ENT customers on the Advanced plan (coming soon).

3. Managed rules allows customers to deploy sets of rules managed by the Cloudflare analyst team. These rulesets include a “Cloudflare Free Managed Ruleset” currently being rolled out for all plans including FREE, as well as Cloudflare Managed, OWASP implementation, and Exposed Credentials Check for all paying plans.

4. Tools give access to IP Access Rules, Zone Lockdown and User Agent Blocking. Although still actively supported, these products cover specific use cases that can be covered using firewall rules. However, they remain a part of the WAF toolbox for convenience.

Gestalt design principles suggest that “elements which are close in proximity to each other are perceived to share similar functionality or traits.” This principle in addition to the input from our customers informed our design decisions.

After reviewing the responses of the study, we understood the importance of making it easy to find the security products in the Dashboard, and the need to make it clear how particular products were related to or worked together with each other.

Crucially, the page needed to:

• Display each type of rule we support, i.e. firewall rules, rate limiting rules and managed rules

• Show the usage amount of each type

• Give the customer the ability to add a new rule and manage existing rules

• Allow the customer to reprioritise rules using the existing drag and drop behavior

• Be flexible enough to accommodate future additions and consolidations of WAF features

We iterated on multiple options, including predominantly vertical page layouts, table based page layouts, and even accordion based page layouts. Each of these options, however, would force us to replicate buttons of similar functionality on the page. With the risk of causing additional confusion, we abandoned these options in favor of a horizontal, tabbed page layout.

As of today, we are launching this new design of WAF to everyone! In the meantime, we are updating documentation to walk you through how to maximize the power of Cloudflare WAF.

This is a starting point of our journey to make Cloudflare WAF not only powerful but also easy to adapt to your needs. We are evaluating approaches to empower your decision-making process when protecting your web applications. Among growing intel information and more rules creation possibilities, we want to shorten your path from a possible threat detection (such as by security overview) to setting up the right rule to mitigate such threat. Stay tuned!

Today, we’re excited to announce new capabilities to help customers make the switch from hardware firewall appliances to a true cloud-native firewall built for next-generation networks. Cloudflare One provides a secure, performant, and Zero Trust-enabled platform for administrators to apply consistent security policies across all of their users and resources. Best of all, it’s built on top of our global network, so you never need to worry about scaling, deploying, or maintaining your edge security hardware.

As part of this announcement, Cloudflare launched the Oahu program today to help customers leave legacy hardware behind; in this post we’ll break down the new capabilities that solve the problems of previous firewall generations and save IT teams time and money.

In order to understand where we are today, it’ll be helpful to start with a brief history of IP firewalls.

The first generation of network firewalls were designed mostly to meet the security requirements of private networks, which started with the castle and moat architecture we defined as Generation 1 in our post yesterday. Firewall administrators could build policies around signals available at layers 3 and 4 of the OSI model (primarily IPs and ports), which was perfect for (e.g.) enabling a group of employees on one floor of an office building to access servers on another via a LAN.

This packet filtering capability was sufficient until networks got more complicated, including by connecting to the Internet. IT teams began needing to protect their corporate network from bad actors on the outside, which required more sophisticated policies.

Firewall hardware evolved to include stateful packet inspection and the beginnings of deep packet inspection, extending basic firewall concepts by tracking the state of connections passing through them. This enabled administrators to (e.g.) block all incoming packets not tied to an already present outgoing connection.

These new capabilities provided more sophisticated protection from attackers. But the advancement came at a cost: supporting this higher level of security required more compute and memory resources. These requirements meant that security and network teams had to get better at planning the capacity they’d need for each new appliance and make tradeoffs between cost and redundancy for their network.

In addition to cost tradeoffs, these new firewalls only provided some insight into how the network was used. You could tell users were accessing 198.51.100.10 on port 80, but to do a further investigation about what these users were accessing would require you to do a reverse lookup of the IP address. That alone would only land you at the front page of the provider, with no insight into what was accessed, reputation of the domain/host, or any other information to help answer “Is this a security event I need to investigate further?”. Determining the source would be difficult here as well, it would require correlating a private IP address handed out via DHCP with a device and then subsequently a user (if you remembered to set long lease times and never shared devices).

To accommodate these challenges, the industry introduced the Next Generation Firewall (NGFW). These were the long reigning, and in some cases are still the industry standard, corporate edge security device. They adopted all the capabilities of previous generations while adding in application awareness to help administrators gain more control over what passed through their security perimeter. NGFWs introduced the concept of vendor-provided or externally-provided application intelligence, the ability to identify individual applications from traffic characteristics. Intelligence which could then be fed into policies defining what users could and couldn’t do with a given application.

As more applications moved to the cloud, NGFW vendors started to provide virtualized versions of their appliances. These allowed administrators to no longer worry about lead times for the next hardware version and allowed greater flexibility when deploying to multiple locations.

Over the years, as the threat landscape continued to evolve and networks became more complex, NGFWs started to build in additional security capabilities, some of which helped consolidate multiple appliances. Depending on the vendor, these included VPN Gateways, IDS/IPS, Web Application Firewalls, and even things like Bot Management and DDoS protection. But even with these features, NGFWs had their drawbacks — administrators still needed to spend time designing and configuring redundant (at least primary/secondary) appliances, as well as choosing which locations had firewalls and incurring performance penalties from backhauling traffic there from other locations. And even still, careful IP address management was required when creating policies to apply pseudo identity.

As firewall vendors added more sophisticated controls, in parallel, a paradigm shift for network architecture was introduced to address the security concerns introduced as applications and users left the organization’s “castle” for the Internet. Zero Trust security means that no one is trusted by default from inside or outside the network, and verification is required from everyone trying to gain access to resources on the network. Firewalls started incorporating Zero Trust principles by integrating with identity providers (IdPs) and allowing users to build policies around user groups — “only Finance and HR can access payroll systems” — enabling finer-grained control and reducing the need to rely on IP addresses to approximate identity.

These policies have helped organizations lock down their networks and get closer to Zero Trust, but CIOs are still left with problems: what happens when they need to integrate another organization’s identity provider? How do they safely grant access to corporate resources for contractors? And these new controls don’t address the fundamental problems with managing hardware, which still exist and are getting more complex as companies go through business changes like adding and removing locations or embracing hybrid forms of work. CIOs need a solution that works for the future of corporate networks, instead of trying to duct tape together solutions that address only some aspects of what they need.

Cloudflare is helping customers build the future of their corporate networks by unifying network connectivity and Zero Trust security. Customers who adopt the Cloudflare One platform can deprecate their hardware firewalls in favor of a cloud-native approach, making IT teams’ lives easier by solving the problems of previous generations.

Rather than managing different devices for different use cases, all traffic across your network — from data centers, offices, cloud properties, and user devices — should be able to flow through a single global firewall. Cloudflare One enables you to connect to the Cloudflare network with a variety of flexible on-ramp methods including network-layer (GRE or IPsec tunnels) or application-layer tunnels, direct connections, BYOIP, and a device client. Connectivity to Cloudflare means access to our entire global network, which eliminates many of the challenges with physical or virtualized hardware:

• No more capacity planning: The capacity of your firewall is the capacity of Cloudflare’s global network (currently >100Tbps and growing).

• No more location planning: Cloudflare’s Anycast network architecture enables traffic to connect automatically to the closest location to its source. No more picking regions or worrying about where your primary/backup appliances are — redundancy and failover are built in by default.

• No maintenance downtimes: Improvements to Cloudflare’s firewall capabilities, like all of our products, are deployed continuously across our global edge.

• DDoS protection built in: No need to worry about DoS attacks overwhelming your firewalls; Cloudflare’s network automatically blocks attacks close to their source and sends only the clean traffic on to its destination.

Cloudflare One policies can be used to secure and route your organizations traffic across all the various traffic ramps. These policies can be crafted using all the same attributes available through a traditional NGFW while expanding to include Zero Trust attributes as well. These Zero Trust attributes can include one or more IdPs or endpoint security providers.

When looking strictly at layers 3 through 5 of the OSI model, policies can be based on IP, port, protocol, and other attributes in both a stateless and stateful manner. These attributes can also be used to build your private network on Cloudflare when used in conjunction with any of the identity attributes and the Cloudflare device client.

Additionally, to help relieve the burden of managing IP allow/block lists, Cloudflare provides a set of managed lists that can be applied to both stateless and stateful policies. And on the more sophisticated end, you can also perform deep packet inspection and write programmable packet filters to enforce a positive security model and thwart the largest of attacks.

Cloudflare’s intelligence helps power our application and content categories for our Layer 7 policies, which can be used to protect your users from security threats, prevent data exfiltration, and audit usage of company resources. This starts with our protected DNS resolver, which is built on top of our performance leading consumer 1.1.1.1 service. Protected DNS allows administrators to protect their users from navigating or resolving any known or potential security risks. Once a domain is resolved, administrators can apply HTTP policies to intercept, inspect, and filter a user's traffic. And if those web applications are self-hosted or SaaS enabled you can even protect them using a Cloudflare access policy, which acts as a web based identity proxy.

Last but not least, to help prevent data exfiltration, administrators can lock down access to external HTTP applications by utilizing remote browser isolation. And coming soon, administrators will be able to log and filter which commands a user can execute over an SSH session.

Traditional firewalls required deploying policies on each device or configuring and maintaining an orchestration tool to help with this process. In contrast, Cloudflare allows you to manage policies across our entire network from a simple dashboard or API, or use Terraform to deploy infrastructure as code. Changes propagate across the edge in seconds thanks to our Quicksilver technology. Users can get visibility into traffic flowing through the firewall with logs, which are now configurable.

Firewalls need to cover a myriad of traffic flows to satisfy different security needs, including blocking bad inbound traffic, filtering outbound connections to ensure employees and applications are only accessing safe resources, and inspecting internal (“East/West”) traffic flows to enforce Zero Trust. Different hardware often covers one or multiple use cases at different locations; we think it makes sense to consolidate these as much as possible to improve ease of use and establish a single source of truth for firewall policies. Let’s walk through some use cases that were traditionally satisfied with hardware firewalls and explain how IT teams can satisfy them with Cloudflare One.

Traditionally, IT teams needed to provision at least one hardware firewall per office location (multiple for redundancy). This involved forecasting the amount of traffic at a given branch and ordering, installing, and maintaining the appliance(s). Now, customers can connect branch office traffic to Cloudflare from whatever hardware they already have — any standard router that supports GRE or IPsec will work — and control filtering policies across all of that traffic from Cloudflare’s dashboard.

Step 1: Establish a GRE or IPsec tunnelThe majority of mainstream hardware providers support GRE and/or IPsec as tunneling methods. Cloudflare will provide an Anycast IP address to use as the tunnel endpoint on our side, and you configure a standard GRE or IPsec tunnel with no additional steps — the Anycast IP provides automatic connectivity to every Cloudflare data center.

Step 2: Configure network-layer firewall rulesAll IP traffic can be filtered through Magic Firewall, which includes the ability to construct policies based on any packet characteristic — e.g., source or destination IP, port, protocol, country, or bit field match. Magic Firewall also integrates with IP Lists and includes advanced capabilities like programmable packet filtering.

Step 3: Upgrade traffic for application-level firewall rulesAfter it flows through Magic Firewall, TCP and UDP traffic can be “upgraded” for fine-grained filtering through Cloudflare Gateway. This upgrade unlocks a full suite of filtering capabilities including application and content awareness, identity enforcement, SSH/HTTP proxying, and DLP.

Firewalls used for processing data at a high-traffic headquarters or data center location can be some of the largest capital expenditures in an IT team’s budget. Traditionally, data centers have been protected by beefy appliances that can handle high volumes gracefully, which comes at an increased cost. With Cloudflare’s architecture, because every server across our network can share the responsibility of processing customer traffic, no one device creates a bottleneck or requires expensive specialized components. Customers can on-ramp traffic to Cloudflare with BYOIP, a standard tunnel mechanism, or Cloudflare Network Interconnect, and process up to terabits per second of traffic through firewall rules smoothly.

In order to connect to corporate resources or get secure access to the Internet, users in traditional network architectures establish a VPN connection from their devices to a central location where firewalls are located. There, their traffic is processed before it’s allowed to its final destination. This architecture introduces performance penalties and while modern firewalls can enable user-level controls, they don’t necessarily achieve Zero Trust. Cloudflare enables customers to use a device client as an on-ramp to Zero Trust policies; watch out for more updates later this week on how to smoothly deploy the client at scale.

We can’t wait to keep evolving this platform to serve new use cases. We’ve heard from customers who are interested in expanding NAT Gateway functionality through Cloudflare One, who want richer analytics, reporting, and user experience monitoring across all our firewall capabilities, and who are excited to adopt a full suite of DLP features across all of their traffic flowing through Cloudflare’s network. Updates on these areas and more are coming soon (stay tuned).

Cloudflare’s new firewall capabilities are available for enterprise customers today. Learn more here and check out the Oahu Program to learn how you can migrate from hardware firewalls to Zero Trust — or talk to your account team to get started today.

At Cloudflare, we believe that you shouldn’t have to compromise privacy for security. Last year, we launched Cloudflare Gateway — a comprehensive, Secure Web Gateway with built-in Zero Trust browsing controls for your organization. Today, we’re excited to share the latest set of privacy features available to administrators to log and audit events based on your team’s needs.

Cloudflare Gateway helps organizations replace legacy firewalls while also implementing Zero Trust controls for their users. Gateway meets you wherever your users are and allows them to connect to the Internet or even your private network running on Cloudflare. This extends your security perimeter without having to purchase or maintain any additional boxes.

Organizations also benefit from improvements to user performance beyond just removing the backhaul of traffic to an office or data center. Cloudflare’s network delivers security filters closer to the user in over 250 cities around the world. Customers start their connection by using the world’s fastest DNS resolver. Once connected, Cloudflare intelligently routes their traffic through our network with layer 4 network and layer 7 HTTP filters.

To get started, administrators deploy Cloudflare’s client (WARP) on user devices, whether those devices are macOS, Windows, iOS, Android, ChromeOS or Linux. The client then sends all outbound layer 4 traffic to Cloudflare, along with the identity of the user on the device.

With proxy and TLS decryption turned on, Cloudflare will log all traffic sent through Gateway and surface this in Cloudflare’s dashboard in the form of raw logs and aggregate analytics. However, in some instances, administrators may not want to retain logs or allow access to all members of their security team.

The reasons may vary, but the end result is the same: administrators need the ability to control how their users' data is collected and who can audit those records.

Legacy solutions typically give administrators an all-or-nothing blunt hammer. Organizations could either enable or disable all logging. Without any logging, those services did not capture any personally identifiable information (PII). By avoiding PII, administrators did not have to worry about control or access permissions, but they lost all visibility to investigate security events.

That lack of visibility adds even more complications when teams need to address tickets from their users to answer questions like “why was I blocked?”, “why did that request fail?”, or “shouldn’t that have been blocked?”. Without logs related to any of these events, your team can’t help end users diagnose these types of issues.

Starting today, your team has more options to decide the type of information Cloudflare Gateway logs and who in your organization can review it. We are releasing role-based dashboard access for the logging and analytics pages, as well as selective logging of events. With role-based access, those with access to your account will have PII information redacted from their dashboard view by default.

We’re excited to help organizations build least-privilege controls into how they manage the deployment of Cloudflare Gateway. Security team members can continue to manage policies or investigate aggregate attacks. However, some events call for further investigation. With today’s release, your team can delegate the ability to review and search using PII to specific team members.

We still know that some customers want to reduce the logs stored altogether, and we’re excited to help solve that too. Now, administrators can now select what level of logging they want Cloudflare to store on their behalf. They can control this for each component, DNS, Network, or HTTP and can even choose to only log block events.

That setting does not mean you lose all logs — just that Cloudflare never stores them. Selective logging combined with our previously released Logpush service allows users to stop storage of logs on Cloudflare and turn on a Logpush job to their destination of choice in their location of choice as well.

To get started, any Cloudflare Gateway customer can visit the Cloudflare for Teams dashboard and navigate to Settings > Network. The first option on this page will be to specify your preference for activity logging. By default, Gateway will log all events, including DNS queries, HTTP requests and Network sessions. In the network settings page, you can then refine what type of events you wish to be logged. For each component of Gateway you will find three options:

1. Capture all

2. Capture only blocked

3. Don’t capture

Additionally, you’ll find an option to redact all PII from logs by default. This will redact any information that can be used to potentially identify a user including User Name, User Email, User ID, Device ID, source IP, URL, referrer and user agent.

We’ve also included new roles within the Cloudflare dashboard, which provide better granularity when partitioning Administrator access to Access or Gateway components. These new roles will go live in January 2022 and can be modified on enterprise accounts by visiting Account Home → Members.

If you’re not yet ready to create an account, but would like to explore our Zero Trust services, check out our interactive demo where you can take a self-guided tour of the platform with narrated walkthroughs of key use cases, including setting up DNS and HTTP filtering with Cloudflare Gateway.

Moving forward, we’re excited to continue adding more and more privacy features that will give you and your team more granular control over your environment. The features announced today are available to users on any plan; your team can follow this link to get started today.

Earlier today a cybersecurity advisory was published by international security agencies identifying widespread attacks against government and private sector targets worldwide. You can read the full report here, which discusses widespread, distributed, and anonymized brute force access attempts since mid-2019 and still active through early 2021.

Today, we have rolled out WAF mitigations to protect our customers against these types of attacks.

And we are making the exposed credential check feature of Account Takeover Protection available to all paid plans at no additional charge today. We had been planning to release these features later this month to a subset of our customers, but when we were informed of this ongoing attack we accelerated the release timeline and expanded those eligible to use the protections.

The attack which we are now protecting against was carried out in three main steps:

1. Initial account compromise performed via brute force attacks against authentication endpoints;

2. Once access was gained, network traversal was performed leveraging several publicly known vulnerabilities, including but not limited to CVE 2020-0688 and CVE 2020-17144 that widely affected Microsoft Exchange Servers;

3. Deployment of remote shells, such as a variant of the reGeorg web shell, and network reconnaissance to gather additional information;

The findings in the report highlight the increasing problem of password reuse and compromise that affects online applications, including government and large private sector online properties.

In March 2021, during Security Week, we launched a beta program for a new feature called Exposed Credential Checks. This feature allows website administrators to be notified whenever a login attempt is performed using a breached username and password credential pair. This is a very strong signal to enforce two factor authentication, a password reset, or simply increase user logging for the duration of the session.

Starting today, all paid plans (i.e., Pro and above) can enable the exposed credential check feature of Account Takeover Protection. We made the decision to give this to more customers due to the severity of the report and ongoing nature of the exploitation attempts.

While we work to accelerate the automatic deployment of the capability across these plans, you can file a support ticket with “Account Takeover Protections activation request” in the subject line to have it manually enabled today for your domains.

Customers who are not yet running the new WAF announced during Security Week will first be upgraded to this version; all accounts created after May 6, 2021, are already on the new version. The exposed credential managed ruleset can then be turned on with a single click, and supports the following applications out of the box:

• WordPress

• Joomla

• Drupal

• Ghost

• Magento

• Plone

• Microsoft Exchange

When turned on, whenever a compromised credential is detected the following header will be added to the request to the origin server:

Exposed-Credential-Check: 1

This header alone won’t provide additional security, but can be used by the origin server to enforce additional measures, for example forcing a two factor authentication or password reset flow. The feature can also be deployed in logging mode to easily identify brute force attacks targeting your application using the Firewall Analytics dashboard.

If your application is not in the default set of protected applications, as long as your login endpoints conform to one of our generic rules, the feature will work as expected. We currently have two options:

• A JSON endpoint (application/json) that submits credentials with 'email' and 'password' keys, for example {“email”:”user@example.com”, “password”:”pass”}

• A standard login HTML form (application/x-www-form-urlencoded), under a URL that contains “login”. The form fields should be named username and password respectively;

Developer documentation can be found here.

In addition to exposed credential checks, we have implemented improvements to the following WAF rules effective immediately:

• Improved rule 100197

• Added a new rule 100197B (default disabled)

These rules will match against request payloads that contain the reGeorg shell variant mentioned in the report. The rule improvements were based on, but not limited to, the Yara rule found in the security advisory. In summary the rule will block payloads which contain the following signatures and similar variations:

%@ Page Language=C#
StrTr
System.Net.IPEndPoint
Response.AddHeader
Socket

In addition to monitoring and defending against credential stuffing attacks using datasets of compromised credentials, security administrators should implement additional best practices for their authentication endpoints. For example, multi-factor authentication, account time-out and lock-out features, and stronger methods of authentication that require “having” something such as a hard token or client certificate—not just “knowing” something such as a username and password.

Cloudflare has a number of additional features that customers are also advised to deploy where possible on their environments to strengthen their security posture:

• Cloudflare Access can be used to provide strong, multi-factor authentication for both internal and external facing applications, and integrates directly with your organization’s SSO and identity providers (IdP);

• Where possible, implementing Mutual TLS rules (mTLS) in front of authentication endpoints will increase an application security posture considerably by avoiding the use of passwords. This can be done both as a Firewall Rule or as an option when setting up Cloudflare Access;

• We recently announced a Managed IP list that will contain Open Proxy endpoints identified by Cloudflare’s intelligence - this list can be used when creating Firewall Rules to protect authentication endpoints by issuing Captcha (or other) challenges;

• The use of our Bot Management detection has recently been expanded to all self-serve paid plans via our Super Bot Fight Mode product - this product allows customers to set up rules to challenge/block automated traffic, such as bots attempting brute force attacks, while letting verified bots access Internet properties normally.

Brute force attacks are a prevalent and successful means to gain initial access to private networks, especially when applications require only username and password pairs for authentication. The report released today reinforced the widespread use of these credential stuffing attacks to gain access and then pivot to additional sensitive resources and data using other vulnerabilities.

Cloudflare customers are protected against these automated attacks by two new WAF rules, and also through the exposed credential check feature of our Account Takeover Protection offering. We have made the exposed credential check feature available today, to all paid plans, in advance of our planned launch later this month. Reach out to our support team immediately if you would like this feature enabled as we work to turn it on for everyone.

The Cloudflare Web Application Firewall (WAF) blocks more than 57 billion cyber threats per day. That is 650k blocked HTTP requests per second. The original code that filters this traffic was written by Cloudflare’s now CTO and the WAF has since received many accolades including the highest score for ability to execute in the 2020 Gartner Magic Quadrant for WAF.

Because we value replacing code when it is no longer as maintainable, performant, or scalable as it once was, we regularly rewrite key parts of the Cloudflare stack. That’s necessary as our enormous growth makes yesterday’s solutions unworkable. For some time, we have been working on replacing that original LuaJIT code John wrote with new code, written in Rust, along with an improved UI.

We are now excited to announce a new Cloudflare Web Application Firewall.

Starting today, 10% of newly created accounts on Cloudflare will be given access to the new WAF whenever a Pro plan zone or above is added. This percentage will increase to 100% of new accounts over the month of April, after which migration efforts will commence for existing customers. Enterprise customers may migrate early by contacting their account team.

The Web Application Firewall (WAF) is a core component of the Cloudflare platform. As one of the most used products in the portfolio, we have gained a lot of feedback and experience from running it at scale, that helped guide us in this major iteration. The new WAF brings:

• Better rule browsing and configuration - easy one click deploy without losing the power tools: advanced filtering, bulk editing, rule tags and more. Turning on all WordPress rules, setting all Cloudflare Managed Rules to LOG or figuring out which rules are not running is now easy.

• A new matching engine - written in Rust and supporting the wirefilter syntax - the same syntax used by custom Firewall Rules. This engine will allow us to have faster managed rule deployments and the ability to scale to the next level by allowing the WAF to be deployed on even more traffic. All while improving performance and security.

• Updated Rulesets -__ the new WAF ships with updated rulesets that provide better control separating rule status from action. The Cloudflare OWASP Core Ruleset has also been improved based on the latest version of the OWASP Core Ruleset (v3.3 at time of writing), which adds paranoia levels and improves false positives rates compared to the current version available.

• Global configuration - deploy the same configuration across your entire account. Group rules as rulesets and make use of native versioning and rollback capabilities.

The list above is only a small subset of the things we are excited about and each point deserves its own post, but here are the highlights.

The Cloudflare Managed Ruleset, which includes the Cloudflare Specials¹ group, is one of the main components of the WAF. It has several hundred rules that are provided and maintained by Cloudflare. In its default configuration we aim to achieve a very low false positive rate, while providing a very good security baseline for any web application. For the best possible security stance though, you should enable as many rules as possible. This means that, sometimes it’s necessary to deep dive and customise the ruleset behaviour based on the underlying application.

With the new WAF, we wanted to make sure enabling the Managed Ruleset was a one click effort with the default configuration, while allowing a much better configuration experience for those interested.

The new WAF UI. One click to turn on the Cloudflare Managed Ruleset and the Cloudflare OWASP ModSecurity Core Ruleset.

Today, to turn on our Cloudflare Managed Ruleset, you need to enable a global WAF switch and configure any rule groups of interest. The ten rule groups, which include WordPress, Joomla, PHP, and similar, are displayed directly on the page with on/off toggles. This UI does not allow to easily filter or configure rules within those groups without checking each rule individually.

The current (old) Managed Ruleset WAF UI. Clicking on a group displays a list of rules in each group.

Although the UI was simple, it did not allow common tasks to be executed quickly. For example: show me all rules that are off or show me all rules mitigating XSS attacks. Now, all rules are displayed in a single table - but filtering by rule status, action and tag is one click away. Rule tags are also replacing groups, and a rule may have one or more tags, making the system a lot more flexible. Tags will be used for:

• Identifying if a rule is applicable to a specific software component

• Identifying the attack vector (e.g. XSS, SQLi, RCE)

• Identifying rules that are CVE specific

Finally, we are allowing for bulk editing controls in addition to inline single rule controls to allow for faster configuration changes based on specific use cases.

The new WAF ruleset browser. Bulk options, tags and filtering components in one place.‌‌

As we expect the number of rules available to increase, and for more users to adopt custom configurations, we have added a review screen when deploying configuration changes. From here you can easily see any changes from the defaults, and optionally revert them.

Ruleset deployment review screen. From here you can review any custom configuration overrides from defaults.

The current Cloudflare WAF, responsible for managed ruleset execution, is written in LuaJIT and is implemented as an NGINX module. The rule syntax followed a superset of the syntax implemented by ModSecurity with added features specific to the Cloudflare implementation.

By moving to a new engine we wanted to achieve:

• A safer, better and more performant environment consistent with other technologies used at Cloudflare

• Exposure of much better filtering and matching capabilities to allow for flexibility of deployment and easier exception handling

• Unified product feature set by adopting the wirefilter syntax as the basis for managed rulesets

The last point was especially important not only for us but also for our users because this syntax is already in use for our custom Firewall Rules, which even uses the same underlying Rust library to execute the filters!

The new engine is implemented in Rust, which we have talked about how much we love quite a few times before on this blog. We're also working on making sure that this new implementation not only comes with safety improvements, but speed improvements as well, the specifics of which we'll go into in future blog posts.

The Cloudflare rulesets have been updated and ported over to the new WAF. Notably the rulesets now use wirefilter syntax and rule status is separated from rule action, allowing you to configure both independently.

The Cloudflare OWASP Core Ruleset has also received a major update independently from the engine. The current Cloudflare WAF implements a 2.x version of the official OWASP ModSecurity Core Ruleset. In the new WAF the Cloudflare OWASP Core Ruleset is based directly on the latest 3.3 version available from the GitHub repository.

The new Cloudflare OWASP Core Ruleset, along with added engine features, brings several improvements over the existing one:

• Fewer false positives and more powerful application generic rules

• More control over the sensitivity score, with clear indication of how much each rule contributes to the score and what the total score was on triggered requests

• The addition of a paranoia level - to provide easy inclusion or exclusion or rule groups based on false positive risk

• Rule tags to allow deployment with pertinent rules based on the application

The Cloudflare OWASP Core Ruleset‌‌

The Cloudflare OWASP Core Ruleset review screen‌‌

As part of the efforts to convert the latest version of the OWASP ModSecurity Core Ruleset to its Cloudflare implementation, the team has also built a ModSecurity to wirefilter syntax converter. This will enable us to easily deploy and update the ruleset shortly after any upstream improvements to ensure customers have access to the latest version at all times. We also plan to open source and expose the converter in the UI in the future to make customer migrations from ModSecurity-based WAFs to Cloudflare easier.

Cloudflare has operated on a zone-based model for the Cloudflare WAF since the beginning. This serves well for simple use cases where customers are protecting a small number of applications, or a very diverse set of applications on a per-zone basis.

More complex or unified deployments across zones are normally implemented by leveraging the API or automation tools such as our Cloudflare Terraform Provider.

With the new WAF, ruleset deployments can be made across any arbitrary filter of traffic under a single account. For example:

• Deploy Cloudflare Managed Ruleset across all my zones.

• Deploy Cloudflare OWASP Core Ruleset on all traffic that does not contain /api/* in the path.

• Disable the Managed Rulesets across my account for traffic coming from my IP.

This enables powerful account wide WAF configurations with a couple of clicks.

_Three rulesets deployed on arbitrary applications on the account._‌‌

To achieve this, rulesets (group of rules) become a first class concept, and are natively versioned allowing for both rollback and diffing capabilities directly in the UI — features that we plan to start exposing in the coming months.

Account-based configurations will initially only be available to Enterprise customers who can now request early access by contacting their account team. Custom Firewall Rules themselves will soon be migrated onto the new engine, allowing for customers to also create their own custom firewall rulesets, and deploy them as needed on any traffic filter.

There is a lot more to the WAF than meets the eye and the team is already busy finalising a set of additional features built on the new WAF — these include improvements to the engine itself, better analytics and visibility into actionable events. The entire engine is in fact designed to be the basis for many of the Cloudflare rule-based products, with the aim of ultimately representing the entire Cloudflare configuration as a set of rules.

In the meantime, we look forward to your feedback and we are excited to see how far we can innovate.

.......

¹The Cloudflare Specials are rules written by the Cloudflare security team based on observing and protecting millions of web applications that sit behind the Cloudflare platform.

API traffic is growing fast. Last year alone it grew 300% faster at our edge than web traffic. Because APIs power mobile and web applications, transmitting instructions as diverse as “order a pizza from my favourite restaurant using this credit card” or “place a cryptocurrency trade and these are my personal details”, they are ripe for data theft and abuse. Data exposure is listed as one of the top threats for API traffic by OWASP; this includes data leaks. The increase in API traffic and more frequent data attacks call for new security solutions.

Cloudflare’s security toolkit had always been designed to protect web and API traffic. However, after talking to hundreds of customers we realised that there is a need for easily deployed and configured security tools for API traffic in a single interface. To meet this demand, in October 2020 we launched API Shield^TM, a new product aimed at bringing together all security solutions designed for API traffic. We started by providing mTLS authentication to all Cloudflare users free of charge, gRPC support and Schema Validation in Beta. During the launch we laid a plan for future releases with more advanced security functionality. We are now thrilled to be able to expand our offering with new features designed to protect your APIs from exposing sensitive data.

Today we are launching four features to help reduce the impact of exfiltration attacks: Schema Validation for all Enterprise customers, a managed IP List allowing you to block traffic from Open Proxies, more control over the certificate lifecycle, and a Data Loss Prevention solution. Later this week, we’ll also announce capabilities to help you discover APIs running on your network that you may not be aware of, and ways to identify anomalous requests that deviate from intended uses.

During the API Shield launch we introduced Schema Validation and we released it to a few selected customers. Over the last few months, we have been working with our early adopters to add more capabilities and build an easy-to-use interface directly into our dashboard. You can now navigate to the ‘API Shield’ tab where you can deploy API security products directly from the UI. The deployment flow will be expanded soon to include additional capabilities found elsewhere in the dashboard, such as mTLS and Rate Limiting. We will also be integrating new features such as API anomaly detection for an easy feedback loop.

Schema Validation works by creating a positive security model based on an API “schema”, which is a contract that defines the consumption of an API and guides developers to integrate it in their systems. Conversely to a negative security model (where rules define what characteristics a request must have to trigger an action, such as block), Schema Validation is designed to allow requests that have been verified as compliant while taking actions on everything else. Schema Validation accepts schemas that adhere to the OpenAPI v3 Specification (also known as Swagger Specification), which is the standard for defining RESTful interfaces (learn more at this page).

Schema Validation evaluates each request against an API Schema logging or blocking requests that do not comply with it.

API Shield offers an easy to use UI where users can upload their schemas to the Firewall and automatically create rules that validate each request against the API definition. If the request is compliant then it is forwarded to the origin. Conversely, if the format or data content of the request does not match what is expected by API Shield, the call is either logged or dropped protecting the origin from an invalid request or a malicious payload. Requests with extraneous input may not have been anticipated by the API developer, and they may trigger unforeseen application behaviour, such as a data leak.

API Shield with Schema Validation being deployed using the Cloudflare Rulesets OpenAPI schema.‌‌

The UI guides the user through different steps, including defining the hostname and base path of the API where API Shield will be deployed and uploading the schema file. Once API Shield with Schema Validation has been deployed, it is possible to inspect what endpoints have been parsed by the Firewall and what level of protection is being provided. In the review page, there are two groups of endpoints: protected and unprotected. The former lists all endpoints and methods whose schema is supported, the latter indicates any endpoint whose definition was either not supported or ambiguous. To avoid breaking traffic directed to endpoints not listed in the schema file, we include a final rule that matches traffic not directed to any of the protected endpoints.

Every time a non-compliant request is identified by Schema Validation and an action such as block or log is taken, a new event tagged with the source ‘API Shield’ is created and added to the Firewall logs. Users can access analytics and logs by visiting the Overview page where they can then drill down data using the flexibility of our GraphQL-powered dashboard.

Schema validation performs checks on the path, path variables, query parameters, headers, and cookies, and allows logging non-compliant traffic. Schema validation Beta is now broadly available to Enterprise customers — fill this form to get access.

Data loss is one of the biggest security concerns that affect small and large organisations, but it also has an impact on individuals and their privacy. Loss of sensitive data can have a massive impact on companies in terms of financial impact, brand value erosion, and compliance with the latest laws on data protection. Finally, loss of sensitive data belonging to individuals can have a detrimental effect in terms of monetary loss and privacy concerns.

Earlier today we announced our Data Loss Prevention (DLP) product suite; we’re now extending this to identify sensitive data leaving your origin in the response phase of an HTTP or API request. The solution evaluates the egress traffic, checking payloads against common patterns of sensitive data. These include personally identifiable information such as Social Security numbers and financial information, including credit card numbers, bank details, etc. For the first release, users will be able to log any match triggered by DLP. We are planning to add other actions such as obfuscating and blocking sensitive data leaving origins behind Cloudflare. Next, we intend to let customers customise the rules to identify sensitive data that are specific to their application.

We developed DLP with simplicity in mind so that every customer can be protected without requiring complex and time-consuming set up periods. We are releasing DLP as a managed ruleset that can be turned on through the Firewall Managed Rules tab. DLP can be used as part of the WAF of a reverse proxy, but it can also be used as part of Cloudflare for Teams integrating data protection in a Zero Trust configuration. This tight integration enables better control on who can access sensitive information within your organization.

DLP is in beta and we are releasing it to selected early adopter customers. Please fill this form to join the waitlist.

Data Loss Protection can be turned on as a Managed Ruleset.

We are launching our first Managed IP List which will be available for use within Firewall Rules. In July 2020 we released IP Lists, which gave customers the ability to upload large lists of IPs that can be used when writing Firewall Rules. Today we are launching a list that is curated by Cloudflare and that customers can use in their rules exactly as they use custom uploaded Lists.

‘Cloudflare Open Proxies’ contains the IPs of Open SOCKS and HTTP Proxies determined by Cloudflare by analysing traffic at its edge. This is not just limited to API requests; rules can apply to all types of traffic being evaluated by the Firewall.

The list is the first feed we are making public based on Cloudflare threat intelligence that leverages the scale and reach of our network. How do we populate this list? We see requests from every publicly routable IP address on the Internet. Cloudflare combines open source lists with its large network to identify open proxies. After verifying the proxies, Cloudflare determines the exit IPs and creates a list. We then feed this reputation data back into our security systems and make it available to our customers in the form of a managed IP list.

The list is available to all Enterprise plans and it can be used by selecting ‘Cloudflare Open Proxies’ in the drop-down menu collecting all available IP Lists (see picture below).

Cloudflare Open Proxies managed list can be used directly in the Firewall rule builder.

We launched mTLS with the first release of API Shield with mobile apps and IoT devices in mind. Enforcing strong authentication with client side certificates is a very effective measure to protect traffic from data exfiltration and abuse in general. However, in the event of an IoT device or mobile phone being stolen, lost or having control taken over by a malicious actor, Cloudflare users need a way to revoke the certificate that is considered a potential security risk. Having the ability to permanently exclude traffic from compromised devices is an effective way to prevent data loss and malicious attacks.

Many of our customers who have started embedding API Shield certificates in their apps have implemented a revocation solution using Workers with Workers KV. Although this solution allows granular control on certificates, it does require significant development effort from our customers and does not scale easily.

For Security Week, we are releasing a fully managed solution to revoke (and restore) certificates without the need to write a single line of code. We built a straightforward interface to manage the entire lifecycle of your certificates at our edge, from issuance to revocation. We take care of this for you, so you can focus on building your application without having to worry about setting up a complex and costly Public Key Infrastructure (PKI) and managing the revocation of potentially risky devices. The customer touchpoints are a new ‘Revoke’ and ‘Restore’ button in the client certificate tab, its supporting API calls and a new field for Firewall Rules.

Each request presenting a certificate to the Cloudflare’s edge will have two Firewall fields set: cf.tls_client_auth.cert_verified and cf.tls_client_auth.cert_revoked. The request is processed by the Firewall where users can combine these fields with all other Firewall functionality. This allows customers to set different behaviours based on whether the certificate was verified or verified but revoked. It also allows you to implement the required security policy while providing a good experience for end users. A classic configuration is to allow only requests with a verified certificate, while forwarding requests from revoked certificates to a different page or endpoint to handle the exception as required by your users’ journey.

Users can revoke certificates by visiting the Client Certificates tab.

The Cloudflare team is working on releasing additional features under the API Shield umbrella. We are talking to hundreds of customers who are using Cloudflare for API traffic and three features come up as high priority: in-depth API Analytics, a more flexible Rate Limiting tool, and integration with API Anomaly Detection.

Schema Validation and Data Loss Prevention are released today with full integration in our logs and analytics engine. Going forward, we are planning to expand the ability to analyse traffic and provide customers with tools to identify and manage attacks specifically directed to API endpoints.

Cloudflare’s rate limiting is designed to work best for web traffic where you can write rules based on URLs and request methods. We are now working on integrating the power of Firewall Rules with the control provided by rate limiting. This will allow users to segment their traffic leveraging the powerful logic available in the Firewall. We are also extending the counting mechanism to include the ability to rate limit based on API key and User ID.

When targeting API traffic, attack patterns can vary greatly, making traditional Bot Management solutions not the ideal candidate to identify suspicious behaviour. On Friday we will announce two major features that focus on further protecting your applications: API Discovery and Anomaly Detection. Discovery allows customers to map their endpoints and get visibility on the surface area of their APIs. Anomaly Detection is Cloudflare’s solution to autonomously separate good API traffic from malicious activity reliably and at scale. Customers will be able to set this up along with mTLS, Schema Validation, and Rate Limiting to maximise level of protection. Check out our blog on Friday to learn more about these new products.

Back in October 2020, we introduced Cloudflare One, our vision for the future of corporate networking and security. Since then, we’ve been laser-focused on delivering more pieces of this platform, and today we’re excited to announce two of its most foundational aspects: Magic WAN and Magic Firewall. Magic WAN provides secure, performant connectivity and routing for your entire corporate network, reducing cost and operational complexity. Magic Firewall integrates smoothly with Magic WAN, enabling you to enforce network firewall policies at the edge, across traffic from any entity within your network.

Enterprise networks have historically adopted one of a few models, which were designed to enable secure information flow between offices and data centers, with access to the Internet locked down and managed at office perimeters. As applications moved to the cloud and employees moved out of offices, these designs stopped working, and band-aid solutions like VPN boxes don’t solve the core problems with enterprise network architecture.

On the connectivity side, full mesh MPLS (multiprotocol label switching) networks are expensive and time consuming to deploy, challenging to maintain, exponentially hard to scale, and often have major gaps in visibility. Other architectures require backhauling traffic through central locations before sending it back to the source, which introduces unacceptable latency penalties and necessitates purchasing costly hub hardware for maximum capacity rather than actual utilization. And most customers we’ve talked to are struggling with the worst of both worlds: a combination of impossible-to-manage architectures stitched together over years or decades. Security architects also struggle with these models - they have to juggle a stack of security hardware boxes from different vendors and trade off cost, performance, and security as their network grows.

With Magic WAN, you can securely connect any traffic source - data centers, offices, devices, cloud properties - to Cloudflare’s network and configure routing policies to get the bits where they need to go, all within one SaaS solution. Magic WAN supports a variety of on-ramps including Anycast GRE tunnels, Cloudflare Network Interconnect, Argo Tunnel, WARP, and a variety of Network On-ramp Partners. No more MPLS expense or lead times, no more performance penalties from traffic trombones, and no more nightmare of managing a tangle of legacy solutions: instead, use Cloudflare’s global Anycast network as an extension of yours, and get better performance and visibility built-in.

Once your traffic is connected to Cloudflare, how do you control which entities within your network are allowed to interact with each other, or the Internet? This is where Magic Firewall comes in. Magic Firewall allows you to centrally manage policy across your entire network, all at the edge as a service. Magic Firewall gives you fine-grained control over which data is allowed in and out of your network, or inside your network. Even better, you get visibility into exactly how traffic is flowing through your network from a single dashboard.

Magic WAN provides the foundation for the broad suite of functions included in Cloudflare One, which were all built in software from the ground up to scale and integrate smoothly. Magic Firewall is available for Magic WAN out of the box, and customers can easily activate additional Zero Trust security and performance features such as our Secure Web Gateway with remote browser isolation, Intrusion Detection System, Smart Routing, and more.

“Our network team is excited by Magic WAN. Cloudflare has built a global network-as-a-service platform that will help network teams manage complex edge and multi-cloud environments much more efficiently. Operating a single global WAN with built-in security and fast routing functionality — regardless of the HQ, data center, branch office, or end user location — is a game-changer in WAN technology.”— Sander Petersson, Head of Infrastructure, FlightRadar24

What does this look like concretely? Let's explore some ways you can use Magic WAN and Magic Firewall to solve problems in enterprise networking and security with an example customer, Acme Corp.

Today, Acme Corp has offices around the world that are each connected to regional data centers and to each other with MPLS. Each data center, which hosts corporate applications and a stack of hardware boxes to keep them secure, also has leased line connectivity to at least one other data center. Acme is also migrating some applications to the cloud, and they’re planning to establish direct connections from their data centers to cloud providers to boost security and reliability.

As Acme has grown, one of their network team's most consistent pain points is managing and maintaining their MPLS connectivity. It's expensive and deployment requires long lead times, limiting Acme’s speed of expanding into new locations (especially internationally) or supporting offices added through acquisitions. Employees in Acme’s offices connecting to cloud providers and SaaS apps experience frustrating latency, since traffic hairpins through an Acme data center for security policies enforced through a stack of hardware boxes before being sent to its destination. Traffic between offices, such as IP telephony and video conferencing, doesn’t have any security policies applied to it, presenting a gap in Acme’s security posture.

Just as they're working on transitioning to the cloud for compute and storage, Acme wants to migrate away from these private links and instead leverage the Internet, securely, for connectivity. They considered establishing fully meshed site-to-site IPSec VPN tunnels over the Internet, but the complexity strained their networking team as well as their heterogeneous router deployments. Magic WAN is ready to meet them where they are today, simplifying network management and delivering immediate performance benefits, as well as enabling Acme's gradual transition away from MPLS.

In this example deployment with Magic WAN, Acme connects each office and VPC to Cloudflare with Anycast GRE tunnels. With this architecture, Acme only needs to set up a single tunnel for each site/Internet connection in order to automatically receive connectivity to Cloudflare's entire global network (200+ cities in 100+ countries around the world) - like a hub and spoke architecture, except the hub is everywhere. Acme also chooses to establish dedicated private connectivity from their data centers with Cloudflare Network Interconnect, enabling even more secure and reliable traffic delivery and great connectivity to other networks/cloud providers through Cloudflare’s highly interconnected network.

Once these tunnels are established, Acme can configure allowed routes for traffic within their private network (RFC 1918 space), and Cloudflare gets the traffic where it needs to go, providing resiliency and traffic optimization. With an easy setup process that takes only a few hours, Acme Corp can start their migration away from MPLS. And as new Cloudflare One capabilities like QoS and Argo for Networks are introduced, Acme’s network performance and reliability will continue to improve.

When Acme Corp's employees abruptly transitioned to working remotely last year due to the COVID-19 pandemic, Acme's IT organization scrambled to find short term fixes to maintain employee access to internal resources. Their legacy VPNs didn't hold up - Acme employees working from home struggled with connectivity, reliability, and performance issues as appliances were pushed beyond the limits they were designed for.

Thankfully, there's a better solution! Acme Corp can use Cloudflare for Teams and Magic WAN to provide a secure way for employees to access resources behind private networks from their devices, wherever they're working. Acme employees install the WARP client on their devices to send traffic to Cloudflare's network, where it can be authenticated and routed to private resources in data centers or VPCs that are connected to Cloudflare via GRE tunnel (shown in the previous example), Argo Tunnel, Cloudflare Network Interconnect, or IPSec (coming soon). This architecture solves the performance and capacity issues Acme experienced with their legacy VPNs - rather than sending all traffic through single choke point devices, it’s routed to the closest Cloudflare location where policy is applied at the edge before being sent along an optimized path to its destination.

Traffic from Acme Corp's employee devices, data centers and offices will also be able to be policed by Magic Firewall for powerful, granular policy control that's enforced across all "on-ramps." Whether employees are connecting from their phones and laptops or working from Acme offices, the same policies can be applied in the same place. This simplifies configuration and improves visibility for Acme's IT and security teams, who will be able to log into the Cloudflare dashboard to see and control policies in one place - a game changer compared to managing employee access across different VPNs, firewalls, and cloud services.

This solution allows Acme to transition away from their VPN, firewall, and secure web gateway appliances, improving performance and enabling easy policy management across traffic from wherever their employees are working.

Historically, Acme relied on stacks of hardware appliances in physical offices and data centers to enforce network security and get visibility into what’s happening on the network: specialized firewalls to inspect inbound or outbound traffic, intrusion detection systems and SIEMs. As their organization is moving to the cloud and rethinking the future of remote work, the Acme security team is looking for sustainable solutions that improve security, even beyond what used to be possible in their traditional castle-and-moat architecture.

Previously, we walked through how Acme could configure Magic WAN to send traffic from offices, data centers, cloud properties, and devices to Cloudflare’s network. Once this traffic is flowing through Cloudflare, it’s easy to add access controls and filtering functions to augment or replace on-prem security hardware, all delivered and administered through a single pane of glass.

With Magic Firewall, customers get a single firewall-as-a-service that runs at the edge, replacing clunky boxes they have installed at branch offices or data centers. Magic Firewall allows them to easily manage configuration, but also simplifies compliance auditing.

To control access, customers can put policies in place that determine exactly which traffic is allowed to go where. For example, Acme wants traffic to flow from the Internet to their web servers inside data centers on port 80 and 443, but wants to lock down SSH access to only certain private networks inside branch offices.

If Acme wants to go further in locking down their network, they can adopt a Zero Trust access model  with Access and Gateway to control who can reach what, and how, across all their traffic flowing through the Cloudflare network. As Cloudflare releases new filtering and control functions, like our upcoming IDS/IPS and DLP solutions, Acme can enable them to further increase security with only a few clicks.

Acme’s long-term goal is to transition all security and performance functions to the cloud, consumed as a service. Magic WAN enables a smooth path for this migration, allowing them to gradually deepen their security posture as they retire legacy hardware.

Increased cloud adoption along with the recent pivot to remote workers has increased the volume of Internet, SaaS, and IaaS traffic straining traditional network architectures such as MPLS. WAN architectures that offer a global scale, integrated enterprise network security functions, and direct, secure connectivity to remote users are key to organizations looking to increase their operational agility and lower total costs of ownership.— Ghassan Abdo, IDC Research VP, WW Telecom, Virtualization & CDN

Like many of our products, Cloudflare One started as a collection of solutions to problems that we experienced in growing and securing our own network. Magic WAN and Magic Firewall allow us to extend the benefits of our careful architecture decisions over the past ten years to customers:

• Global scale & close to eyeballs: wherever your offices, data centers, and users are, we’re close by. We’ve worked hard to establish great connectivity to eyeball networks because of our CDN business, which pays dividends for remote workers who need great connectivity from their homes back to your network. It also means we can stop threats at the edge, close to their source, rather than running the risk of malicious traffic overwhelming capacity-limited on prem appliances.

• Hardware and carrier-agnostic: use whatever hardware you have today to connect to us, and get the benefits of resiliency afforded by our diverse carrier connectivity.

• Built from scratch to work together: we’ve developed our products in software, from the ground up, to integrate easily. We think constantly about how our products can integrate to make each other better as they’re created and evolved.

Magic WAN is available in limited beta, and Magic Firewall is generally available for all Magic Transit customers and included out of the box with Magic WAN. If you’re interested in testing out Magic WAN or want to learn more about how Cloudflare can help your organization replace legacy MPLS architecture, secure access for remote workers, and deepen your security posture while reducing total cost of ownership, please get in touch.

The Managed Rules team was recently given the task of allowing Enterprise users to debug Firewall Rules by viewing the part of a request that matched the rule. This makes it easier to determine what specific attacks a rule is stopping or why a request was a false positive, and what possible refinements of a rule could improve it.

The fundamental problem, though, was how to securely store this debugging data as it may contain sensitive data such as personally identifiable information from submissions, cookies, and other parts of the request. We needed to store this data in such a way that only the user who is allowed to access it can do so. Even Cloudflare shouldn't be able to see the data, following our philosophy that any personally identifiable information that passes through our network is a toxic asset.

This means we needed to encrypt the data in such a way that we can allow the user to decrypt it, but not Cloudflare. This means public key encryption.

Now we needed to decide on which encryption algorithm to use. We came up with some questions to help us evaluate which one to use:

• What requirements do we have for the algorithm?

• What language do we implement it in?

• How do we make this as secure as possible for users?

Here's how we made those decisions.

While we knew we needed to use public key encryption, we also needed to keep an eye on performance. This led us to select Hybrid Public Key Encryption (HPKE) early on as it has a best-of-both-worlds approach to using symmetric as well as public-key cryptography to increase performance. While these best-of-both-worlds schemes aren’t new [1][2][3], HPKE aims to provide a single, future-proof, robust, interoperable combination of a general key encapsulation mechanism and a symmetric encryption algorithm.

HPKE is an emerging standard developed by the Crypto Forum Research Group (CFRG), the research body that supports the development of Internet standards at the IETF. The CFRG produces specifications called RFCs (such as RFC 7748 for elliptic curves) that are then used in higher level protocols including two we talked about previously: ODoH and ECH. Cloudflare has long been a supporter of Internet standards, so HPKE was a natural choice to use for this feature. Additionally, HPKE was co-authored by one of our colleagues at Cloudflare.

HPKE combines an asymmetric algorithm such as elliptic curve Diffie-Hellman and a symmetric cipher such as AES. One of the upsides of HPKE is that the algorithms aren't dictated to the implementer, but making a combination that’s provably secure and meets the developer’s intuitive notions of security is important. All too often developers reach for a scheme without carefully understanding what it does, resulting in security vulnerabilities.

HPKE solves these problems by providing a high level of security in a generic manner and providing necessary hooks to tie messages to the context in which they are generated. This is the application of decades of research into the correct security notions and schemes.

HPKE is built in stages. First it turns a Diffie-Hellman key agreement into a Key Encapsulation Mechanism. A key encapsulation mechanism has two algorithms: Encap and Decap. The Encap algorithm creates a symmetric secret and wraps it in a public key, so that only the holder of the private key can unwrap it. An attacker with the encapsulation cannot recover the random key. Decap takes the encapsulation and the private key associated to the public key, and computes the same random key. This translation gives HPKE the flexibility to work almost unchanged with any kind of public key encryption or key agreement algorithm.

HPKE mixes this key with an optional info argument, as well as information relating to the cryptographic parameters used by each side. This ensures that attackers cannot modify messages’ meaning by taking them out of context. A postcard marked “So happy to see you again soon” is ominous from the dentist and endearing from one’s grandmother.

The specification for HPKE is open and available on the IETF website. It is on its way to becoming an RFC after passing multiple rounds of review and analysis by cryptography experts at the CFRG. HPKE is already gaining adoption in IETF protocols like ODoH, ECH, and the new Messaging Layer Security (MLS) protocol. HPKE is also designed with the post-quantum future since it is built to work with any KEM, including all the NIST finalists for post-quantum public-key encryption.

Once we had an encryption scheme selected, we needed to settle on an implementation. HPKE is still fairly new, so the libraries aren't quite mature yet. There is a reference implementation, and we’re in the process of developing an implementation in Go as part of CIRCL. However, in the absence of a clear "go to" that is widely known to be the best, we decided to go with an implementation leveraging the same language already powering much of the Firewall code running at the Cloudflare edge - Rust.

Aside from this, the language benefits from features like native primitives, and crucially the ability to easily compile to WebAssembly (WASM).

As we mentioned in a previous blog post, customers are able to generate a key pair and decrypt payloads either from the dashboard UI or from a CLI. Instead of writing and maintaining two different codebases for these, we opted to reuse the same implementation across the edge component that encrypts the payloads and the UI and CLI that decrypt them. To achieve this we compile our library to target WASM so it can be used in the dashboard UI code that runs in the browser. While this approach may yield a slightly larger JavaScript bundle size and relatively small computational overhead, we found it preferable to spending a significant amount of time securely re-implementing HPKE using JavaScript WebCrypto primitives.

The HPKE implementation we decided on comes with the caveat of not yet being formally audited, so we performed our own internal security review. We analyzed the cryptography primitives being used and the corresponding libraries. Between the composition of said primitives and secure programming practices like correctly zeroing memory and safe usage of random number generators, we found no security issues.

To encrypt on behalf of users, we need them to provide us with a public key. To make this as easy as possible, we built a CLI tool along with the ability to do it right in the browser. Either option allows the user to generate a public/private key pair without needing to talk to Cloudflare servers at all.

In our API, we specifically do not accept the private key of the key pair — we don't want it! We don't need and don't want to be able to decrypt the data we're storing.

For the dashboard, once the user provides the private key for decryption, the key is held in a temporary JavaScript variable and used for the in-browser decryption. This allows the user to not constantly have to provide the key while browsing the Firewall event logs. The private key is also not persisted in any way in the browser, so any action that refreshes the page such as refreshing or navigating away will require the user to provide the key again. We believe this is an acceptable usability compromise for better security.

After deciding how to encrypt the data, we just had to figure out the rest of the feature: what data to encrypt, how to store and transmit it, and how to allow users to decrypt it.

When an HTTP request reaches the L7 Firewall, it is evaluated against a set of rulesets. Each of these rulesets contain several rules written in the wirefilter syntax.

An example of one such rule would be:

http.request.version eq "HTTP/1.1"
and
(
    http.request.uri.path matches "\n+."
    or
    http.request.uri.query matches "\x00+."
)

This expression evaluates to a boolean “true” for HTTP/1.1 requests that either contain one or more newlines followed by a character in the request path or one or more NULL bytes followed by a character in the query string.

Say we had the following request that would match the rule above:

GET /cms/%0Aadmin?action=%00post HTTP/1.1
Host: example.com

If matched data logging is enabled, the rules that match would be executed again in a special context that tags all fields that are accessed during execution. We do this second execution because this tagging adds a noticeable computational overhead, and since the vast majority of requests don't trigger a rule at all we would be unnecessarily adding overhead to each request. Requests that do match any rules will only match a few rules as well, so we don't need to re-execute a large portion of the ruleset.

You may notice that although http.request.uri.query matches "\x00+." evaluates to true for this request, it won’t be executed, because the expression short-circuits with the first or condition that also matches. This results in only http.request.version and http.request.uri.path being tagged as accessed:

http.request.version -> HTTP/1.1
http.request.uri.path -> /cms/%0Aadmin

Having gathered the fields that were accessed, the Firewall engine does some post-processing; removing fields that are a subset of others (e.g., the query string and the full URI), or truncating fields that are beyond a certain character length.

Finally, these get serialized as JSON, encrypted with the customer's public key, serialized again as a set of bytes, and prefixed with a version number should we need to change/update it in the future. To simplify consumption of these blobs, our APIs display a base64 encoded version of the bytes:

Now that we have encrypted the data at the edge and persisted it in ClickHouse, we need to allow users to decrypt it. As part of the setup of turning this feature on, users generated a key-pair: the public key which was used to encrypt the payloads and a private key which is used to decrypt them. Decryption is done completely offline via either the command line using cloudflare/matched-data-cli:

$ MATCHED_DATA=AkjQDktMX4FudxeQhwa0UPNezhkgLAUbkglNQ8XVCHYqPgAAAAAAAACox6cEwqWQpFVE2gCFyOFsSdm2hCoE0/oWKXZJGa5UPd5mWSRxNctuXNtU32hcYNR/azLjsGO668Jwk+qCdFvmKjEqEMJgI+fvhwLQmm4=
$ matched-data-cli decrypt -d $MATCHED_DATA -k $PRIVATE_KEY
{"http.request.version": "HTTP/1.1", "http.request.uri.path": "/cms/%0Aadmin"}

Or the dashboard UI:

Since our CLI tool is open-source and HPKE is interoperable, it can also be used in other tooling as part of a user's logging pipeline, for example in security information and event management (SIEM) software.

This was a team effort with help from our Research and Security teams throughout the process. We relied on them for recommendations on how best to evaluate the algorithms as well as vetting the libraries we wanted to use.

We're very pleased with how HPKE has worked out for us from an ease-of-implementation and performance standpoint. It was also an easy choice for us to make due to its impending standardization and best-of-both-worlds approach to security.

Taking the time to reflect on what we’ve delivered also helps to reinforce my belief in the Cloudflare approach to shipping product: release early, stay close to customers for feedback, and iterate quickly to deliver incremental value. To borrow a term from the investment world, this approach brings the benefits of compounded returns to our customers: we put new products that solve real-world problems into their hands as quickly as possible, and then reinvest the proceeds of our shared learnings immediately back into the product.

It is these sustained investments that allow us to release a flurry of small improvements over the course of a year, and be recognized by leading industry analyst firms for the capabilities we’ve accumulated and distributed to our customers. Today we’re excited to announce that Frost & Sullivan has named Cloudflare the Innovation Leader in their Frost Radar™: Global Holistic Web Protection Market Report. Frost & Sullivan’s view that this market “will gradually absorb the markets formed around legacy and point solutions” is consistent with our view of the world, and we’re leading the way in “the consolidation of standalone WAF, DDoS mitigation, and Bot Risk Management solutions” they believe is “poised to happen before 2025”.

Image © 2020 Frost & Sullivan from Frost Radar™: Global Holistic Web Protection Market Report

We are honored to receive this recognition, based on the analysis of 10 providers’ competitive strengths and opportunities as assessed by Frost & Sullivan. The rest of this post explains some of the capabilities that we shipped in 2020 across our Web Application Firewall (WAF), Bot Management, and Distributed Denial-of-Service product lines—the scope of Frost & Sullivan’s report. Get a copy of the Frost & Sullivan Frost Radar report to see why Cloudflare was named the Innovation Leader here.

Before jumping into specific product and feature launches, I want to briefly explain how we think about building and delivering our web security capabilities. The most important “product” by far that’s been built at Cloudflare over the past 10 years is the massive global network that moves bits securely around the world, as close to the speed of light as possible. Building our features atop this network allows us to reject the legacy tradeoff of performance or security. And equipping customers with the ability to program and extend the network with Cloudflare Workers and Firewall Rules allows us to focus on quickly delivering useful security primitives such as functions, operators, and ML-trained data—then later packaging them up in streamlined user interfaces.

We talk internally about building up the “toolbox” of security controls so customers can express their desired security posture, and that’s how we think about many of the releases over the past year that are discussed below. We begin by providing the saw, hammer, and nails, and let expert builders construct whatever defenses they see fit. By watching how these tools are put to use and observing the results of billions of attempts to evade the erected defenses, we learn how to improve and package them together as a whole for those less inclined to build from components. Most recently we did this with API Shield, providing a guided template to create “positive security” models within Firewall Rules using existing primitives plus new data structures for strong authentication such as Cloudflare-managed client SSL/TLS certificates. Each new tool added to the toolbox increases the value of the existing tools. Each new web request—good or bad—improves the models that our threat intelligence and Bot Management capabilities depend upon.

Last year we spoke with many customers about our plan to decouple configuration from the zone/domain model and allow rules to be set for arbitrary paths and groups of services across an account. In 4Q2020 we put this granular control in the hands of a few developers and some of our most sophisticated enterprise customers, and we’re currently collecting and incorporating feedback before defaulting the capabilities on for new customers.

Rules are great, especially with increased flexibility, but without data structures and request enrichment at the edge (such as the Bot Management techniques described below) they cannot act on anything beyond static properties of the request. In 3Q2020 we released our IP Lists capabilities and customers have been steadily uploading their home-grown and third-party subscription lists. These lists can be referenced anywhere in a customer’s account as named variables and then combined with all other attributes of the request, even Bot Management scores, e.g., http.request.uri.path contains “/login” and (not ip.src in $pingdom_probes and cf.bot_management.score < 30) is a Firewall Rule filter that blocks all bots except Pingdom from accessing the login endpoint.

Requests that are blocked or challenged need to find their way as quickly as possible to our customers’ SOCs for triage, investigation and, occasionally, incident response, so we upgraded our edge-logging framework in 2Q2020 to push real time security-specific logs directly to customer SIEMs. And in 4Q2020, we released the ability to encrypt sensitive payloads within these logs using customer-provided encryption keys and novel encryption algorithms termed “Hybrid Public Key Encryption” (HPKE), and a data localization suite to provide control over where our customers’ data is stored and protected.

Built predominantly in 4Q2020 and currently being tested in the Firewall Rules engine is a brand new implementation of our Rate Limiting engine. By moving this matching and enforcement logic from a standalone tool to a component within a performant, memory-safe, expressive engine built in Rust, we have increased the utility of existing functions. Additional examples of improving this library of capabilities include the work completed in 1Q2020 to add HMAC functions and regex-based HTTP header and body inspection to the engine.

In addition to making edge data sets accessible for request evaluation, we continued to invest heavily within our Bot Management team to provide actionable data so that our customers could decide what (if any) automated traffic they wanted to allow to interact with their applications. Our highest priority for Bot research and development has always been efficacy, and last year was no different. A significant portion of our engineering effort was dedicated to our detection engines — both updating and iterating on existing systems or creating entirely new detection engines from scratch.

In 1Q2020 we completed a total rewrite of our Machine Learning engine, and are continually focused on improving the efficacy of our ML engines. To do this, we draw on one of our major competitive advantages: the massive amount of data flowing through Cloudflare’s network. The early 2020 upgrade to our ML model nearly doubled the number of features we use to evaluate and score requests. And to help customers better understand why requests are flagged as bots, we have recently complemented the bot likelihood score in our logs with attribution to the specific engine that generated the score.

Also in 1Q2020, we upgraded our behavioral analysis engine to incorporate more features and increase overall accuracy. This engine conducts histogram-based outlier scoring and is now fully deployed to nearly all Bot Management zones.

In 2Q2020, we developed a lightweight JavaScript element that further advanced our browser fingerprinting capabilities and aids in detection. Specifically, we now silently challenge browsers and detect if a browser is misrepresenting its User Agent. This technique will be incorporated into our ML models and combined with our heuristics engine for more accurate browser fingerprinting. This feature is entirely optional and can be enabled or disabled by customers through our UI and API. Customers with extremely performance sensitive zones or traffic types that are unsuitable for JavaScript (such as API or some mobile app traffic) can still be accurately scored by our Bot Management engine.

In addition to detection, we also spent (and will continue to spend) engineering effort on mitigation. Our entire JavaScript and CAPTCHA challenge platform was rewritten in the last year and deployed to our customer zones in a staged fashion in the second half of 2020. Our new platform is faster and more robust at detecting automated systems attempting to solve the challenges. More importantly, this platform allows us to further invest in new challenge types and modes as we enter 2021.

The biggest and most well received feature released in 2020 was our dedicated Bot Management analytics, released in 3Q2020. We now present informative graphs that double as diagnostic tools. Customers have found that analytics are far more than interesting charts and statistics: in the case of Bot Management, analytics are essential to spotting and subsequently eliminating false positives.

Last but definitely not least, we announced the deprecation of the __cfduid cookie in 4Q2020 which was used primarily to detect bots but caused confusion for some customers including questions about whether they needed to display a cookie banner because of what we do.

To get a sense of the Bot Attack trends we saw in the first half of 2020, take a read through this blog post. And if you’re curious about how our ML models and heuristic engines work to keep your properties safe, this deep dive by Alex Bocharov, Machine Learning Tech Lead on the Bots team, is an excellent guide.

At the beginning of 4Q2020, we released a product called API Shield that was purpose built to secure, protect, and accelerate API traffic — and will eventually provide much of the common functionality expected in traditional API Gateways. The UI for API Shield was built on top of Firewall Rules for maximum flexibility, and will serve as the jump-off point for configuring additional API security features we have planned this year.

As part of API Shield, every customer now gets a fully managed, domain-scoped private CA generated for each of their zones, and we plan to continue working closely with the SSL/TLS team to expand CA management options based on feedback. Since the release, we’ve seen great adoption from in particular IoT companies focused on locking down their APIs using short-lived client certificates distributed out to devices. Customers can also now upload OpenAPI schemas to be matched against incoming requests from these devices, with bad requests being dropped at the edge rather than passed on to origin infrastructure.

Another capability we released in 4Q2020 was support for gRPC-based API traffic. Since that release, customers have expressed significant interest in using Cloudflare as a secure API gateway between easy-to-use customer-facing JSON endpoints and internal-facing gRPC or GraphQL endpoints. Like most customer challenges at Cloudflare, early adopters are looking to solve these use cases initially with Cloudflare Workers, but we’re keeping an eye on whether there are aspects for which we’ll want to provide first-class feature support.

The application-layer security of a web application or API is of minimal importance if the service itself is not available due to a persistent DDoS attack at L3-L7. While mitigating such attacks has long been one of Cloudflare’s strengths, attack methodologies evolve and we continued to invest heavily in 2020 to drop attacks more quickly, more efficiently, and more precisely; as a result, automatic mitigation techniques are applied immediately and most malicious traffic is blocked in less than 3 seconds.

Early in 2020 we responded to a persistent increase in smaller, more localized attacks by fine-tuning a system that can autonomously detect attacks on any server in any datacenter. In the month prior to us first posting about this tool, it mitigated almost 300,000 network-layer attacks, roughly 55 times greater than the tool we previously relied upon. This new tool, dubbed “dosd”, leverages Linux’s eXpress Data Path (XDP) and allows our system to quickly — and automatically — deploy rules eBPF rules that run on each packet received. We further enhanced our edge mitigation capabilities in 3Q2020 by developing and releasing a protection layer that can operate even in environments where we only see one side of the TCP flow. These network layer protections help protect our customers who leverage both Magic Transit to protect their IP ranges and our WAF to protect their applications and APIs.

To document and provide visibility into these attacks, we released a GraphQL-backed interface in 1Q2020 called Network Analytics. Network Analytics extends the visibility of attacks against our customers’ services from L7 to L3, and includes detailed attack logs containing data such as top source and destination IPs and ports, ASNs, data centers, countries, bit rates, protocol and TCP flag distributions. A litany of improvements made to this graphical rendering engine over the course of 2020 have benefitted all analytics tools using the same front-end. In 4Q2020, Network Analytics was extended to provide traffic and attack insights into Cloudflare Spectrum-protected applications, which are terminated at L4 (TCP/UDP).

Towards the end of 4Q2020, we released real-time DDoS attack alerting capable of sending emails or pages via PagerDuty to alert security teams of ongoing attacks and mitigations. This capability was released just in time to assist with the onslaught of ransomware attacks that Cloudflare helped detect and defend against. For additional context on unique attacks we fought off in 2020, consider reading about an acoustics inspired attack, a 754 million packet-per-second, or a roundup of attacks from 1Q2020, 2Q2020, or 3Q2020.

2020 was a tough year around the world. Throughout what has also been, and continues to be, a period of heightened cyberattacks and breaches, we feel proud that our teams were able to release a steady flow of new and improved capabilities across several critical security product areas reviewed by Frost & Sullivan. These releases culminated in far greater protections for customers at the end of the year than the beginning, and a recognition for our sustained efforts.

We are pleased to have been named the Innovation Leader in their Frost Radar™: Global Holistic Web Protection Market Report, which “addresses organizations' demand for consolidated, single pane of glass solutions, which not only reduce the security gaps of legacy products but also provide simplified management capabilities”.

As we look towards 2021 we plan to continue releasing early and often, listening to feedback from our customers, and delivering incremental value along the way. If you have ideas on what additional capabilities you’d like to use to protect your applications and networks, we’d love to hear them below in the comments.

The Cloudflare Web Application Firewall (WAF) blocks more than 72B malicious requests per day from reaching our customers’ applications. Typically, our users can easily confirm these requests were not legitimate by checking the URL, the query parameters, or other metadata that Cloudflare provides as part of the security event log in the dashboard.

Sometimes investigating a WAF event requires a bit more research and a trial and error approach, as the WAF may have matched against a field that is not logged by default.

Not logging all parts of a request is intentional: HTTP headers and payloads often contain sensitive data, including personally identifiable information, which we consider a toxic asset. Request headers may contain cookies and POST payloads may contain username and password pairs submitted during a login attempt among other sensitive data.

We recognize that providing clear visibility in any security event is a core feature of a firewall, as this allows users to better fine tune their rules. To accomplish this, while ensuring end-user privacy, we built encrypted WAF matched payload logging. This feature will log only the specific component of the request the WAF has deemed malicious — and it is encrypted using a customer-provided key to ensure that no Cloudflare employee can examine the data*. Additionally, the crypto uses an exciting new standard — developed in part by Cloudflare — called Hybrid Public Key Encryption (HPKE).

*All Cloudflare logs are encrypted at rest. This feature implements a second layer of encryption for the specific matched fields so that only the customer can decrypt it.

To turn on this feature, you need to provide a public key, or generate a private-public key pair directly from the dashboard. Your data will then be encrypted using Hybrid Public Key Encryption (HPKE), which offers a great combination of both performance and security.

To simplify this process, we have built an easy-to-use command line utility to generate the key pair:

$ matched-data-cli generate-key-pair
{
  "private_key": "uBS5eBttHrqkdY41kbZPdvYnNz8Vj0TvKIUpjB1y/GA=",
  "public_key": "Ycig/Zr/pZmklmFUN99nr+taURlYItL91g+NcHGYpB8="
}

Cloudflare does not store the private key and it is our customers’ responsibility to ensure it is stored safely. Lost keys, and the data encrypted with them, cannot be recovered but customers can rotate keys to be used with future payloads.

Once encrypted, payloads will be available in the logs as encrypted base64 blobs within the metadata field:

"metadata": [
  {
    "key": "encrypted_matched_data",
    "Value": "AdfVn7odpamJGeFAGj0iW2oTtoXOjVnTFT2x4l+cHKJsEQAAAAAAAAB+zDygjV2aUI92FV4cHMkp+4u37JHnH4fUkRqasPYaCgk="
  }
]

Decrypting payloads can be done via the dashboard from the Security Events log, or by using the command line utility, as shown below. If done via the dashboard, the browser will decrypt the payload locally (i.e., client side) and will not send the private key to Cloudflare.

$ printf $PRIVATE_KEY | ./matched-data-cli decrypt -d AdfVn7odpamJGeFAGj0iW2oTtoXOjVnTFT2x4l+cHKJsEQAAAAAAAAB+zDygjV2aUI92FV4cHMkp+4u37JHnH4fUkRqasPYaCgk= --private-key-stdin

The command above returns:

{"REQUEST_HEADERS:REFERER":"https:\/\/example.com\/testkey.txt?a=<script>alert('xss');<\/script>"}

In the example above, the WAF matched against the REQUEST_HEADERS:REFERER field. Any other fields the WAF matched on would be similarly logged.

In the coming months, this feature will be available on our dashboard to our Enterprise customers. Enterprise customers who would like this feature enabled sooner should reach out to their account team. Only application owners who also have access to the Cloudflare dashboard as Super Administrators will be able to configure encrypted matched payload logging. Those who do not have access to the private key, including Cloudflare staff, are not able to decrypt the logs.

We are also excited for this feature to be one of our first to use Hybrid Public Key Encryption, and for Cloudflare to use this emerging standard developed by the Crypto Forum Research Group (CFRG), the research body that supports the development of Internet standards at the IETF. And stay tuned, we will publish a deep dive post with the technical details soon!

At Cloudflare, we’re constantly working on improving the performance of our edge — and that was exactly what my internship this summer entailed. I’m excited to share some improvements we’ve made to our popular Firewall Rules product over the past few months.

Firewall Rules lets customers filter the traffic hitting their site. It’s built using our engine, Wirefilter, which takes powerful boolean expressions written by customers and matches incoming requests against them. Customers can then choose how to respond to traffic which matches these rules. We will discuss some in-depth optimizations we have recently made to Wirefilter, so you may wish to get familiar with how it works if you haven’t already.

As a new member of the Firewall team, I quickly learned that performance is important — even in our security products. We look for opportunities to make our customers’ Internet properties faster where it’s safe to do so, maximizing both security and performance.

Our engine is already heavily used, powering all of Firewall Rules. But we have bigger plans. More and more products like our Web Application Firewall (WAF) will be running behind our Wirefilter-based engine, and it will become responsible for eating up a sizable chunk of our total CPU usage before long.

Measuring performance is a notoriously tricky task, and as you can probably imagine trying to do this in a highly distributed environment (aka Cloudflare’s edge) does not help. We’ve been surprised in the past by optimizations that look good on paper, but, when tested out in production, just don’t seem to do much.

Our solution? Performance measurement as a service — an isolated and reproducible benchmark for our Firewall engine and a framework for engineers to easily request runs and view results. It’s worth noting that we took a lot of inspiration from the fantastic Rust Compiler benchmarks to build this.

Our benchmarking framework, showing how performance during different stages of processing Wirefilter expressions has changed over time [1].

Our next challenge was to find some meaningful performance metrics. Some experimentation quickly uncovered that time was far too volatile a measure for meaningful comparisons, so we turned to hardware counters [2]. It’s not hard to find tools to measure these (perf and VTune are two such examples), although they (mostly) don’t allow control over which parts of the program are recorded. In our case, we wished to individually record measurements for different stages of filter processing — parsing, compilation, analysis, and execution.

Once again we took inspiration from the Rust compiler, and its self-profiling options, using the perf_event_open API to record counters from inside our binary. We then output something like the following, which our framework can easily ingest and store for later visualization.

Output of our benchmarks in JSON Lines format, showing a list of recordings for each combination of hardware counter and Wirefilter processing stage. We’ve used 10 repeats here for readability, but we use around 20, in addition to 5 warmup rounds, within our framework.

Whilst we mainly focussed on metrics relating to CPU usage, we also use a combination of getrusage and clear_refs to find the maximum resident set size (RSS). This is useful to understand the memory impact of particular algorithms in addition to CPU.

But the challenge was not over. Cloudflare’s standard CI agents use virtualization and sandboxing for security and convenience, but this makes accessing hardware counters virtually impossible. Running our benchmarks on a dedicated machine gave us access to these counters, and ensured more reproducible results.

Our benchmarks were designed from the outset to take an important place in our development process. For instance, we now perform a full benchmark run before releasing each new version to detect performance regressions.

But with our benchmarks in place, it quickly became clear that we had a problem. Our benchmarks simply weren’t fast enough — at least if we wanted to complete them in less than a few hours! The problem was we have a very large number  of filters. Since our engine would never usually execute requests against this many filters at once it was proving incredibly costly. We came up with a few tricks to cut this down…

• Deduplication. It turns out that only around a third of filters are structurally unique (something that is easy to check as Wirefilter can helpfully serialize to JSON). We managed to cut down a great deal of time by ignoring duplicate filters in our benchmarks.

• Sampling. Still, we had too many filters and random sampling presented an easy solution. A more subtle challenge was to make sure that the random sample was always the same to maintain reproducibility.

• Partitioning. We worried that deduplication and sampling would cause us to miss important cases that are useful to optimize. By first partitioning filters by Wirefilter language feature, we can ensure we’re getting a good range of filters. It also helpfully gives us more detail about where specifically the impact of a performance change is.

Most of these are trade-offs, but very necessary ones which allow us to run continual benchmarks without development speed grinding to a halt. At the time of writing, we’ve managed to get a benchmark run down to around 20 minutes using these ideas.

With a benchmarking framework in place, we were ready to begin testing optimizations. But how do you optimize an interpreter like Wirefilter? Just-in-time (JIT) compilation, selective inlining and replication were some ideas floating around in the word of interpreters that seemed attractive. After all, we previously wrote about the cost of dynamic dispatch in Wirefilter. All of these techniques aim to reduce that effect.

However, running some real filters through a profiler tells a different story. Most execution time, around 65%, is spent not resolving dynamic dispatch calls but instead performing operations like comparison and searches. Filters currently in production tend to be pretty light on functions, but throw in a few more of these and even less time would be spent on dynamic dispatch. We suspect that even a fair chunk of the remaining 35% is actually spent reading the memory of request fields.

By now, you shouldn’t be surprised that the contains operator was one of the first in line for optimization. If you’ve ever written a Firewall Rule, you’re probably already familiar with what it does — it checks whether a substring is present in the field you are matching against. For example, the following expression would match when the host is “example.com” or “www.example.net”, but not when it is “cloudflare.com”. In string searching algorithms, this is commonly referred to as finding a ‘needle’ (“example”) within a ‘haystack’ (“example.com”).

http.host contains “example”

How does this work under the hood? Ordinarily, we may have used Rust’s `String::contains` function but Wirefilter also allows raw byte expressions that don’t necessarily conform to UTF-8.

http.host contains 65:78:61:6d:70:6c:65

We therefore used the memmem crate which performs a two-way substring search algorithm on raw bytes.

Sounds good, right? It was, and it was working pretty well, although we’d noticed that rewriting `contains` filters using regular expressions could bizarrely often make them faster.

http.host matches “example”

Regular expressions are great, but since they’re far more powerful than the `contains` operator, they shouldn’t be faster than a specialized algorithm in simple cases like this one.

Something was definitely up. It turns out that Rust’s regex library comes equipped with a whole host of specialized matchers for what it deems to be simple expressions like this. The obvious question was whether we could therefore simply use the regex library. Interestingly, you may not have realized that the popular ripgrep tool does just that when searching for fixed-string patterns.

However, our use case is a little different. Since we’re building an interpreter (and we’re using dynamic dispatch in any case), we would prefer to dispatch to a specialized case for `contains` expressions, rather than matching on some enum deep within the regex crate when the filter is executed. What’s more, there are some pretty cool things being done to perform substring searching that leverages SIMD instruction sets. So we wired up our engine to some previous work by Wojciech Muła and the results were fantastic.

I encourage you to read more on “Algorithm 1”, which we used, but it works something like this (I’ve changed the order a little to help make it clearer). It’s worth reading up on SIMD instructions if you’re unfamiliar with them — they’re the essence behind what makes this algorithm fast.

1. We fill one SIMD register with the first byte of the needle being searched for, simply repeated over and over.

2. We load as much of our haystack as we can into another SIMD register and perform a bitwise equality operation with our previous register.

3. Now, any position in the resultant register that is 0 cannot be the start of the match since it doesn’t start with the same byte of the needle.

4. We now repeat this process with the last byte of the needle, offsetting the haystack, to rule out any positions that don’t end with the same byte as the needle.

5. Bitwise ANDing these two results together, we (hopefully) have now drastically reduced our potential matches.

6. Each of the remaining potential matches can be checked manually using a memcmp operation. If we find a match, then we’re done.

7. If not, we continue with the next part of our haystack and repeat until we’ve checked the entire thing.

You may be wondering what happens if our haystack doesn’t fit neatly into registers. In the original algorithm, nothing. It simply continues reading into the oblivion after the end of the haystack until the last register is full, and uses a bitmask to ignore potential false-positives from this additional region of memory.

As we mentioned, security is our priority when it comes to optimizations, so we could never deploy something with this kind of behaviour. We ended up porting Muła’s library to Rust (we’ve also open-sourced the crate!) and performed an overlapping registers modification found in ARM’s blog.

It’s best illustrated by example — notice the difference between how we would fill registers on an imaginary SIMD instruction-set with 4-byte registers.

Before modification

How registers are filled in the original implementation for the haystack “abcdefghij”, red squares indicate out of bounds memory.

After modification

How registers are filled with the overlapping modification for the same haystack, notice how ‘g’ and ‘h’ each appear in two registers.

In our case, repeating some bytes within two different registers will never change the final outcome, so this modification is allowed as-is. However, in reality, we found it was better to use a bitmask to exclude repeated parts of the final register and minimize the number of memcmp calls.

What if the haystack is too small to even fill a single register? In this case, we can’t use our overlapping trick since there’s nothing to overlap with. Our solution is straightforward: while we were primarily targeting AVX2, which can store 32-bytes in a lane, we can easily move down to another instruction set with smaller registers that the haystack can fit into. In reality, we don’t currently go any smaller than SSE2. Beyond this, we instead use an implementation of the Rabin-Karp searching algorithm which appears to perform well.

Choosing the first and last bytes of the needle to rule out potential matches is a great idea. It means that when it does come to performing a memcmp, we can ignore these, as we know they already match. Unfortunately, as Muła points out, this also makes the algorithm susceptible to a worst-case attack in some instances.

Let’s give an expression that a customer might write to illustrate this.

http.request.uri.path contains “/wp-admin/”

If we try to search for this within a very long sequence of ‘/’s, we will find a potential match in every position and make lots of calls to memcmp — essentially performing a slow bruteforce substring search.

Clearly we need to choose different bytes from the needle. But which ones should we choose? For each choice, an adversary can always find a slightly different, but equally troublesome, worst case. We instead use randomness to throw off our would-be adversary, picking the first byte of the needle as before, but then choosing another random byte to use.

Our new version is unsurprisingly slower than Muła’s, yet it still exhibits a great improvement over both the memmem and regex crates. Performance, but without sacrificing safety.

This is only a small taste of the performance work we’ve been doing, and we have much more yet to come. Nevertheless, none of this would have been possible without the support of my manager Richard and my mentor Elie, who contributed a lot of these ideas. I’ve learned so much over the past few months, but most of all that Cloudflare is an amazing place to be an intern!

[1] Since our benchmarks are not run within a production environment, results in this post do not represent traffic on our edge.

[2] We found instruction counts to be a particularly stable measure, and CPU cycles a particularly unstable one.

[3] Note that SWAR is technically not an instruction set, but instead uses regular registers like vector registers.

Building Cloudflare Bot Management platform is an exhilarating experience. It blends Distributed Systems, Web Development, Machine Learning, Security and Research (and every discipline in between) while fighting ever-adaptive and motivated adversaries at the same time.

This is the ongoing story of Bot Management at Cloudflare and also an introduction to a series of blog posts about the detection mechanisms powering it. I’ll start with several definitions from the Bot Management world, then introduce the product and technical requirements, leading to an overview of the platform we’ve built. Finally, I’ll share details about the detection mechanisms powering our platform.

Let’s start with Bot Management’s nomenclature.

Bot - an autonomous program on a network that can interact with computer systems or users, imitating or replacing a human user's behavior, performing repetitive tasks much faster than human users could.

Good bots - bots which are useful to businesses they interact with, e.g. search engine bots like Googlebot, Bingbot or bots that operate on social media platforms like Facebook Bot.

Bad bots - bots which are designed to perform malicious actions, ultimately hurting businesses, e.g. credential stuffing bots, third-party scraping bots, spam bots and sneakerbots.

Bot Management - blocking undesired or malicious Internet bot traffic while still allowing useful bots to access web properties by detecting bot activity, discerning between desirable and undesirable bot behavior, and identifying the sources of the undesirable activity.

WAF - a security system that monitors and controls network traffic based on a set of security rules.

Cloudflare has been stopping malicious bots from accessing websites or misusing APIs from the very beginning, at the same time helping the climate by offsetting the carbon costs from the bots. Over time it became clear that we needed a dedicated platform which would unite different bot fighting techniques and streamline the customer experience. In designing this new platform, we tried to fulfill the following key requirements.

• Complete, not complex - customers can turn on/off Bot Management with a single click of a button, to protect their websites, mobile applications, or APIs.

• Trustworthy - customers want to know whether they can trust the website visitor is who they say they are and provide a certainty indicator for that trust level.

• Flexible - customers should be able to define what subset of the traffic Bot Management mitigations should be applied to, e.g. only login URLs, pricing pages or sitewide.

• Accurate - Bot Management detections should have a very small error, e.g. none or very few human visitors ever should be mistakenly identified as bots.

• Recoverable - in case a wrong prediction was made, human visitors still should be able to access websites as well as good bots being let through.

Moreover, the goal for new Bot Management product was to make it work well on the following use cases:

Additionally, to the product requirements above, we engineers had a list of must-haves for the new Bot Management platform. The most critical were:

• Scalability - the platform should be able to calculate a score on every request, even at over 10 million requests per second.

• Low latency - detections must be performed extremely quickly, not slowing down request processing by more than 100 microseconds, and not requiring additional hardware.

• Configurability - it should be possible to configure what detections are applied on what traffic, including on per domain/data center/server level.

• Modifiability - the platform should be easily extensible with more detection mechanisms, different mitigation actions, richer analytics and logs.

• Security - no sensitive information from one customer should be used to build models that protect another customer.

• Explainability & debuggability - we should be able to explain and tune predictions in an intuitive way.

Equipped with these requirements, back in 2018, our small team of engineers got to work to design and build the next generation of Cloudflare Bot Management.

“Simplicity is the ultimate sophistication.”- Leonardo Da Vinci

Cloudflare operates on a vast scale. At the time of this writing, this means covering 26M+ Internet properties, processing on average 11M requests per second (with peaks over 14M), and examining more than 250 request attributes from different protocol levels. The key question is how to harness the power of such “gargantuan” data to protect all of our customers from modern day cyberthreats in a simple, reliable and explainable way?

Bot management is hard. Some bots are much harder to detect and require looking at multiple dimensions of request attributes over a long time, and sometimes a single request attribute could give them away. More signals may help, but are they generalizable?

When we classify traffic, should customers decide what to do with it or are there decisions we can make on behalf of the customer? What concept could possibly address all these uncertainty problems and also help us to deliver on the requirements from above?

As you might’ve guessed from the section title, we came up with the concept of Trusted Score or simply The Score - one thing to rule them all - indicating the likelihood between 0 and 100 whether a request originated from a human (high score) vs. an automated program (low score).

"One Ring to rule them all" by idreamlikecrazy, used under CC BY / Desaturated from original

Okay, let’s imagine that we are able to assign such a score on every incoming HTTP/HTTPS request, what are we or the customer supposed to do with it? Maybe it’s enough to provide such a score in the logs. Customers could then analyze them on their end, find the most frequent IPs with the lowest scores, and then use the Cloudflare Firewall to block those IPs. Although useful, such a process would be manual, prone to error and most importantly cannot be done in real time to protect the customer's Internet property.

Fortunately, around the same time we started worked on this system , our colleagues from the Firewall team had just announced Firewall Rules. This new capability provided customers the ability to control requests in a flexible and intuitive way, inspired by the widely known Wireshark®  language. Firewall rules supported a variety of request fields, and we thought - why not have the score be one of these fields? Customers could then write granular rules to block very specific attack types. That’s how the cf.bot_management.score field was born.

Having a score in the heart of Cloudflare Bot Management addressed multiple product and technical requirements with one strike - it’s simple, flexible, configurable, and it provides customers with telemetry about bots on a per request basis. Customers can adjust the score threshold in firewall rules, depending on their sensitivity to false positives/negatives. Additionally, this intuitive score allows us to extend our detection capabilities under the hood without customers needing to adjust any configuration.

So how can we produce this score and how hard is it? Let’s explore it in the following section.

What is powering the Bot Management score? The short answer is a set of microservices. Building this platform we tried to re-use as many pipelines, databases and components as we could, however many services had to be built from scratch. Let’s have a look at overall architecture (this overly simplified version contains Bot Management related services):

In a nutshell our systems process data received from the edge data centers, produce and store data required for bot detection mechanisms using the following technologies:

• Databases & data stores - Kafka, ClickHouse, Postgres, Redis, Ceph.

• Programming languages - Go, Rust, Python, Java, Bash.

• Configuration & schema management - Salt, Quicksilver, Cap’n Proto.

• Containerization - Docker, Kubernetes, Helm, Mesos/Marathon.

Each of these services is built with resilience, performance, observability and security in mind.

All bot detection mechanisms are applied on every request in real-time during the request processing stage in the Bot Management module running on every machine at Cloudflare’s edge locations. When a request comes in we extract and transform the required request attributes and feed them to our detection mechanisms. The Bot Management module produces the following output:

Firewall fields - Bot Management fields- cf.bot_management.score - an integer indicating the likelihood between 0 and 100 whether a request originated from an automated program (low score) to a human (high score).- cf.bot_management.verified_bot - a boolean indicating whether such request comes from a Cloudflare allowlisted bot.- cf.bot_management.static_resource - a boolean indicating whether request matches file extensions for many types of static resources.

Cookies - most notably it produces cf_bm, which helps manage incoming traffic that matches criteria associated with bots.

JS challenges - for some of our detections and customers we inject into invisible JavaScript challenges, providing us with more signals for bot detection.

Detection logs - we log through our data pipelines to ClickHouse details about each applied detection, used features and flags, some of which are used for analytics and customer logs, while others are used to debug and improve our models.

Once the Bot Management module has produced the required fields, the Firewall takes over the actual bot mitigation.

The Cloudflare Firewall's intuitive dashboard enables users to build powerful rules through easy clicks and also provides Terraform integration. Every request to the firewall is inspected against the rule engine. Suspicious requests can be blocked, challenged or logged as per the needs of the user while legitimate requests are routed to the destination, based on the score produced by the Bot Management module and the configured threshold.

Firewall rules provide the following bot mitigation actions:

• Log - records matching requests in the Cloudflare Logs provided to customers.

• Bypass - allows customers to dynamically disable Cloudflare security features for a request.

• Allow - matching requests are exempt from challenge and block actions triggered by other Firewall Rules content.

• Challenge (Captcha) - useful for ensuring that the visitor accessing the site is human, and not automated.

• JS Challenge - useful for ensuring that bots and spam cannot access the requested resource; browsers, however, are free to satisfy the challenge automatically.

• Block - matching requests are denied access to the site.

Our Firewall Analytics tool, powered by ClickHouse and GraphQL API, enables customers to quickly identify and investigate security threats using an intuitive interface. In addition to analytics, we provide detailed logs on all bots-related activity using either the Logpull API and/or LogPush, which provides the easy way to get your logs to your cloud storage.

In case a customer wants more flexibility on what to do with the requests based on the score, e.g. they might want to inject new, or change existing, HTML page content, or serve incorrect data to the bots, or stall certain requests, Cloudflare Workers provide an option to do that. For example, using this small code-snippet, we can pass the score back to the origin server for more advanced real-time analysis or mitigation:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})
 
async function handleRequest(request) {
  request = new Request(request);
 
  request.headers.set("Cf-Bot-Score", request.cf.bot_management.score)
 
  return fetch(request);
}

Now let’s have a look into how a single score is produced using multiple detection mechanisms.

The Cloudflare Bot Management platform currently uses five complementary detection mechanisms, producing their own scores, which we combine to form the single score going to the Firewall. Most of the detection mechanisms are applied on every request, while some are enabled on a per-customer basis to better fit their needs.

Having a score on every request for every customer has the following benefits:

• Ease of onboarding - even before we enable Bot Management in active mode, we’re able to tell how well it’s going to work for the specific customer, including providing historical trends about bot activity.

• Feedback loop - availability of the score on every request along with all features has tremendous value for continuous improvement of our detection mechanisms.

• Ensures scaling - if we can compute for score every request and customer, it means that every Internet property behind Cloudflare is a potential Bot Management customer.

• Global bot insights - Cloudflare is sitting in front of more than 26M+ Internet properties, which allows us to understand and react to the tectonic shifts happening in security and threat intelligence over time.

Overall globally, more than third of the Internet traffic visible to Cloudflare is coming from bad bots, while Bot Management customers have the ratio of bad bots even higher at ~43%!

Let’s dive into specific detection mechanisms in chronological order of their integration with Cloudflare Bot Management.

The majority of decisions about the score are made using our machine learning models. These were also the first detection mechanisms to produce a score and to on-board customers back in 2018. The successful application of machine learning requires data high in Quantity, Diversity, and Quality, and thanks to both free and paid customers, Cloudflare has all three, enabling continuous learning and improvement of our models for all of our customers.

At the core of the machine learning detection mechanism is CatBoost  - a high-performance open source library for gradient boosting on decision trees. The choice of CatBoost was driven by the library’s outstanding capabilities:

• Categorical features support - allowing us to train on even very high cardinality features.

• Superior accuracy - allowing us to reduce overfitting by using a novel gradient-boosting scheme.

• Inference speed - in our case it takes less than 50 microseconds to apply any of our models, making sure request processing stays extremely fast.

• C and Rust API - most of our business logic on the edge is written using Lua, more specifically LuaJIT, so having a compatible FFI interface to be able to apply models is fantastic.

There are multiple CatBoost models run on Cloudflare’s Edge in the shadow mode on every request on every machine. One of the models is run in active mode, which influences the final score going to Firewall. All ML detection results and features are logged and recorded in ClickHouse for further analysis, model improvement, analytics and customer facing logs. We feed both categorical and numerical features into our models, extracted from request attributes and inter-request features built using those attributes, calculated and delivered by the Gagarin inter-requests features platform.

We’re able to deploy new ML models in a matter of seconds using an extremely reliable and performant Quicksilver configuration database. The same mechanism can be used to configure which version of an ML model should be run in active mode for a specific customer.

A deep dive into our machine learning detection mechanism deserves a blog post of its own and it will cover how do we train and validate our models on trillions of requests using GPUs, how model feature delivery and extraction works, and how we explain and debug model predictions both internally and externally.

Not all problems in the world are the best solved with machine learning. We can tweak the ML models in various ways, but in certain cases they will likely underperform basic heuristics. Often the problems machine learning is trying to solve are not entirely new. When building the Bot Management solution it became apparent that sometimes a single attribute of the request could give a bot away. This means that we can create a bunch of simple rules capturing bots in a straightforward way, while also ensuring lowest false positives.

The heuristics engine was the second detection mechanism integrated into the Cloudflare Bot Management platform in 2019 and it’s also applied on every request. We have multiple heuristic types and hundreds of specific rules based on certain attributes of the request, some of which are very hard to spoof. When any of the requests matches any of the heuristics - we assign the lowest possible score of 1.

The engine has the following properties:

• Speed - if ML model inference takes less than 50 microseconds per model, hundreds of heuristics can be applied just under 20 microseconds!

• Deployability - the heuristics engine allows us to add new heuristic in a matter of seconds using Quicksilver, and it will be applied on every request.

• Vast coverage - using a set of simple heuristics allows us to classify ~15% of global traffic and ~30% of Bot Management customers’ traffic as bots. Not too bad for a few if conditions, right?

• Lowest false positives - because we’re very sure and conservative on the heuristics we add, this detection mechanism has the lowest FP rate among all detection mechanisms.

• Labels for ML - because of the high certainty we use requests classified with heuristics to train our ML models, which then can generalize behavior learnt from from heuristics and improve detections accuracy.

So heuristics gave us a lift when tweaked with machine learning and they contained a lot of the intuition about the bots, which helped to advance the Cloudflare Bot Management platform and allowed us to onboard more customers.

Machine learning and heuristics detections provide tremendous value, but both of them require human input on the labels, or basically a teacher to distinguish between right and wrong. While our supervised ML models can generalize well enough even on novel threats similar to what we taught them on, we decided to go further. What if there was an approach which doesn’t require a teacher, but rather can learn to distinguish bad behavior from the normal behavior?

Enter the behavioral analysis detection mechanism, initially developed in 2018 and integrated with the Bot Management platform in 2019. This is an unsupervised machine learning approach, which has the following properties:

• Fitting specific customer needs - it’s automatically enabled for all Bot Management customers, calculating and analyzing normal visitor behavior over an extended period of time.

• Detects bots never seen before - as it doesn’t use known bot labels, it can detect bots and anomalies from the normal behavior on specific customer’s website.

• Harder to evade - anomalous behavior is often a direct result of the bot’s specific goal.

Please stay tuned for a more detailed blog about behavioral analysis models and the platform powering this incredible detection mechanism, protecting many of our customers from unseen attacks.

So far we’ve discussed how to detect bad bots and humans. What about good bots, some of which are extremely useful for the customer website? Is there a need for a dedicated detection mechanism or is there something we could use from previously described detection mechanisms? While the majority of good bot requests (e.g. Googlebot, Bingbot, LinkedInbot) already have low score produced by other detection mechanisms, we also need a way to avoid accidental blocks of useful bots. That’s how the Firewall field cf.bot_management.verified_bot came into existence in 2019, allowing customers to decide for themselves whether they want to let all of the good bots through or restrict access to certain parts of the website.

The actual platform calculating Verified Bot flag deserves a detailed blog on its own, but in the nutshell it has the following properties:

• Validator based approach - we support multiple validation mechanisms, each of them allowing us to reliably confirm good bot identity by clustering a set of IPs.

• Reverse DNS validator - performs a reverse DNS check to determine whether or not a bots IP address matches its alleged hostname.

• ASN Block validator - similar to rDNS check, but performed on ASN block.

• Downloader validator - collects good bot IPs from either text files or HTML pages hosted on bot owner sites.

• Machine learning validator - uses an unsupervised learning algorithm, clustering good bot IPs which are not possible to validate through other means.

• Bots Directory - a database with UI that stores and manages bots that pass through the Cloudflare network.

Bots directory UI sample‌‌

Using multiple validation methods listed above, the Verified Bots detection mechanism identifies hundreds of unique good bot identities, belonging to different companies and categories.

When it comes to Bot Management detection quality it’s all about the signal quality and quantity. All previously described detections use request attributes sent over the network and analyzed on the server side using different techniques. Are there more signals available, which can be extracted from the client to improve our detections?

As a matter of fact there are plenty, as every browser has unique implementation quirks. Every web browser graphics output such as canvas depends on multiple layers such as hardware (GPU) and software (drivers, operating system rendering). This highly unique output allows precise differentiation between different browser/device types. Moreover, this is achievable without sacrificing website visitor privacy as it’s not a supercookie, and it cannot be used to track and identify individual users, but only to confirm that request’s user agent matches other telemetry gathered through browser canvas API.

This detection mechanism is implemented as a challenge-response system with challenge injected into the webpage on Cloudflare’s edge. The challenge is then rendered in the background using provided graphic instructions and the result sent back to Cloudflare for validation and further action such as  producing the score. There is a lot going on behind the scenes to make sure we get reliable results without sacrificing users’ privacy while being tamper resistant to replay attacks. The system is currently in private beta and being evaluated for its effectiveness and we already see very promising results. Stay tuned for this new detection mechanism becoming widely available and the blog on how we’ve built it.

This concludes an overview of the five detection mechanisms we’ve built so far. It’s time to sum it all up!

Cloudflare has the unique ability to collect data from trillions of requests flowing through its network every week. With this data, Cloudflare is able to identify likely bot activity with Machine Learning, Heuristics, Behavioral Analysis, and other detection mechanisms. Cloudflare Bot Management integrates seamlessly with other Cloudflare products, such as WAF  and Workers.

All this could not be possible without hard work across multiple teams! First of all thanks to everybody on the Bots Team for their tremendous efforts to make this platform come to life. Other Cloudflare teams, most notably: Firewall, Data, Solutions Engineering, Performance, SRE, helped us a lot to design, build and support this incredible platform.

Bots team during Austin team summit 2019 hunting bots with axes :)

Lastly, there are more blogs from the Bots series coming soon, diving into internals of our detection mechanisms, so stay tuned for more exciting stories about Cloudflare Bot Management!

The highest trafficked sites using Cloudflare receive billions of requests per day. But only about 5% of those requests typically trigger security rules, whether they be “managed” rules such as our WAF and DDoS protections, or custom rules such as those configured by customers using our powerful Firewall Rules and Rate Limiting engines.

When enforcement is taken on a request that interrupts the flow of malicious traffic, a Firewall Event is logged with detail about the request including which rule triggered us to take action and what action we took, e.g., challenged or blocked outright.

Previously, if you wanted to ingest all of these events into your SIEM or logging platform, you had to take the whole firehose of requests—good and bad—and then filter them client side. If you’re paying by the log line or scaling your own storage solution, this cost can add up quickly. And if you have a security team monitoring logs, they’re being sent a lot of extraneous data to sift through before determining what needs their attention most.

As of today, customers using Cloudflare Logs can create Logpush jobs that send only Firewall Events. These events arrive much faster than our existing HTTP requests logs: they are typically delivered to your logging platform within 60 seconds of sending the response to the client.

In this post we’ll show you how to use Terraform and Sumo Logic, an analytics integration partner, to get this logging set up live in just a few minutes.

The steps below take you through the process of configuring Cloudflare Logs to push security events directly to your logging platform. For purposes of this tutorial, we’ve chosen Sumo Logic as our log destination, but you’re free to use any of our analytics partners, or any logging platform that can read from cloud storage such as AWS S3, Azure Blob Storage, or Google Cloud Storage.

To configure Sumo Logic and Cloudflare we make use of Terraform, a popular Infrastructure-as-Code tool from HashiCorp. If you’re new to Terraform, see Getting started with Terraform and Cloudflare for a guided walkthrough with best practice recommendations such as how to version and store your configuration in git for easy rollback.

Once the infrastructure is in place, you’ll send a malicious request towards your site to trigger the Cloudflare Web Application Firewall, and watch as the Firewall Events generated by that request shows up in Sumo Logic about a minute later.

First you’ll need to install Terraform. See our Developer Docs for instructions.

Next you’ll need to install Go. The easiest way on macOS to do so is with Homebrew:

$ brew install golang
$ export GOPATH=$HOME/go
$ mkdir $GOPATH

Go is required because the Sumo Logic Terraform Provider is a "community" plugin, which means it has to be built and installed manually rather than automatically through the Terraform Registry, as will happen later for the Cloudflare Terraform Provider.

The official installation instructions for installing the Sumo Logic provider can be found on their GitHub Project page, but here are my notes:

$ mkdir -p $GOPATH/src/github.com/terraform-providers && cd $_
$ git clone https://github.com/SumoLogic/sumologic-terraform-provider.git
$ cd sumologic-terraform-provider
$ make install

While not strictly necessary, the livetail tool from Sumo Logic makes it easy to grab the Cloudflare Logs challenge token we’ll need in a minute, and also to view the fruits of your labor: seeing a Firewall Event appear in Sumo Logic shortly after the malicious request hit the edge.

On macOS:

$ brew cask install livetail
...
==> Verifying SHA-256 checksum for Cask 'livetail'.
==> Installing Cask livetail
==> Linking Binary 'livetail' to '/usr/local/bin/livetail'.
?  livetail was successfully installed!

This step assumes you already have a Sumo Logic account. If not, you can sign up for a free trial here.

1. Browse to https://service.$ENV.sumologic.com/ui/#/security/access-keys where $ENV should be replaced by the environment you chose on signup.

2. Click the "+ Add Access Key" button, give it a name, and click "Create Key"

3. In the next step you'll save the Access ID and Access Key that are provided as environment variables, so don’t close this modal until you do.

1. Log in to the Cloudflare Dashboard

2. Click on the profile icon in the top-right corner and then select "My Profile"

3. Select "API Tokens" from the nav bar and click "Create Token"

4. Click the "Get started" button next to the "Create Custom Token" label

On the Create Custom Token screen:

1. Provide a token name, e.g., "Logpush - Firewall Events"

2. Under Permissions, change Account to Zone, and then select Logs and Edit, respectively, in the two drop-downs to the right

3. Optionally, change Zone Resources and IP Address Filtering to restrict restrict access for this token to specific zones or from specific IPs

Click "Continue to summary" and then "Create token" on the next screen. Save the token somewhere secure, e.g., your password manager, as it'll be needed in just a minute.

Rather than add sensitive credentials to source files (that may get submitted to your source code repository), we'll set environment variables and have the Terraform modules read from them.

$ export CLOUDFLARE_API_TOKEN="<your scoped cloudflare API token>"
$ export CF_ZONE_ID="<tag of zone you wish to send logs for>"

We'll also need your Sumo Logic environment, Access ID, and Access Key:

$ export SUMOLOGIC_ENVIRONMENT="eu"
$ export SUMOLOGIC_ACCESSID="<access id from previous step>"
$ export SUMOLOGIC_ACCESSKEY="<access key from previous step>"

We'll create a directory to store our Terraform project in and build it up as we go:

$ mkdir -p ~/src/fwevents && cd $_

Then we'll create the Collector and HTTP source that will store and provide Firewall Events logs to Sumo Logic:

$ cat <<'EOF' | tee main.tf
##################
### SUMO LOGIC ###
##################
provider "sumologic" {
    environment = var.sumo_environment
    access_id = var.sumo_access_id
}

resource "sumologic_collector" "collector" {
    name = "CloudflareLogCollector"
    timezone = "Etc/UTC"
}

resource "sumologic_http_source" "http_source" {
    name = "firewall-events-source"
    collector_id = sumologic_collector.collector.id
    timezone = "Etc/UTC"
}
EOF

Then we'll create a variables file so Terraform has credentials to communicate with Sumo Logic:

$ cat <<EOF | tee variables.tf
##################
### SUMO LOGIC ###
##################
variable "sumo_environment" {
    default = "$SUMOLOGIC_ENVIRONMENT"
}

variable "sumo_access_id" {
    default = "$SUMOLOGIC_ACCESSID"
}
EOF

With our Sumo Logic configuration set, we’ll initialize Terraform with terraform init and then preview what changes Terraform is going to make by running terraform plan:

$ terraform init

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # sumologic_collector.collector will be created
  + resource "sumologic_collector" "collector" {
      + destroy        = true
      + id             = (known after apply)
      + lookup_by_name = false
      + name           = "CloudflareLogCollector"
      + timezone       = "Etc/UTC"
    }

  # sumologic_http_source.http_source will be created
  + resource "sumologic_http_source" "http_source" {
      + automatic_date_parsing       = true
      + collector_id                 = (known after apply)
      + cutoff_timestamp             = 0
      + destroy                      = true
      + force_timezone               = false
      + id                           = (known after apply)
      + lookup_by_name               = false
      + message_per_request          = false
      + multiline_processing_enabled = true
      + name                         = "firewall-events-source"
      + timezone                     = "Etc/UTC"
      + url                          = (known after apply)
      + use_autoline_matching        = true
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Assuming everything looks good, let’s execute the plan:

$ terraform apply -auto-approve
sumologic_collector.collector: Creating...
sumologic_collector.collector: Creation complete after 3s [id=108448215]
sumologic_http_source.http_source: Creating...
sumologic_http_source.http_source: Creation complete after 0s [id=150364538]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Success! At this point you could log into the Sumo Logic web interface and confirm that your Collector and HTTP Source were created successfully.

Before we’ll start sending logs to your collector, you need to demonstrate the ability to read from it. This validation step prevents accidental (or intentional) misconfigurations from overrunning your logs.

In a new shell window—you should keep the current one with your environment variables set for use with Terraform—we'll start tailing Sumo Logic for events sent from the firewall-events-source HTTP source.

The first time that you run livetail you'll need to specify your Sumo Logic Environment, Access ID and Access Key, but these values will be stored in the working directory for subsequent runs:

$ livetail _source=firewall-events-source
### Welcome to Sumo Logic Live Tail Command Line Interface ###
1 US1
2 US2
3 EU
4 AU
5 DE
6 FED
7 JP
8 CA
Please select Sumo Logic environment: 
See http://help.sumologic.com/Send_Data/Collector_Management_API/Sumo_Logic_Endpoints to choose the correct environment. 3
### Authenticating ###
Please enter your Access ID: <access id>
Please enter your Access Key <access key>
### Starting Live Tail session ###

Before requesting a challenge token, we need to figure out where Cloudflare should send logs.

We do this by asking Terraform for the receiver URL of the recently created HTTP source. Note that we modify the URL returned slightly as Cloudflare Logs expects sumo:// rather than https://.

$ export SUMO_RECEIVER_URL=$(terraform state show sumologic_http_source.http_source | grep url | awk '{print $3}' | sed -e 's/https:/sumo:/; s/"//g')

$ echo $SUMO_RECEIVER_URL
sumo://endpoint1.collection.eu.sumologic.com/receiver/v1/http/<redacted>

With URL in hand, we can now request the token.

$ curl -sXPOST -H "Content-Type: application/json" -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -d '{"destination_conf":"'''$SUMO_RECEIVER_URL'''"}' https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/logpush/ownership

{"errors":[],"messages":[],"result":{"filename":"ownership-challenge-bb2912e0.txt","message":"","valid":true},"success":true}

Back in the other window where your livetail is running you should see something like this:

{"content":"eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4R0NNIiwidHlwIjoiSldUIn0..WQhkW_EfxVy8p0BQ.oO6YEvfYFMHCTEd6D8MbmyjJqcrASDLRvHFTbZ5yUTMqBf1oniPNzo9Mn3ZzgTdayKg_jk0Gg-mBpdeqNI8LJFtUzzgTGU-aN1-haQlzmHVksEQdqawX7EZu2yiePT5QVk8RUsMRgloa76WANQbKghx1yivTZ3TGj8WquZELgnsiiQSvHqdFjAsiUJ0g73L962rDMJPG91cHuDqgfXWwSUqPsjVk88pmvGEEH4AMdKIol0EOc-7JIAWFBhcqmnv0uAXVOH5uXHHe_YNZ8PNLfYZXkw1xQlVDwH52wRC93ohIxg.pHAeaOGC8ALwLOXqxpXJgQ","filename":"ownership-challenge-bb2912e0.txt"}

Copy the content value from above into an environment variable, as you'll need it in a minute to create the job:

$ export LOGPUSH_CHALLENGE_TOKEN="<content value>"

With challenge token in hand, we'll use Terraform to create the job.

First you’ll want to choose the log fields that should be sent to Sumo Logic. You can enumerate the list by querying the dataset:

$ curl -sXGET -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/logpush/datasets/firewall_events/fields | jq .
{
  "errors": [],
  "messages": [],
  "result": {
    "Action": "string; the code of the first-class action the Cloudflare Firewall took on this request",
    "ClientASN": "int; the ASN number of the visitor",
    "ClientASNDescription": "string; the ASN of the visitor as string",
    "ClientCountryName": "string; country from which request originated",
    "ClientIP": "string; the visitor's IP address (IPv4 or IPv6)",
    "ClientIPClass": "string; the classification of the visitor's IP address, possible values are: unknown | clean | badHost | searchEngine | whitelist | greylist | monitoringService | securityScanner | noRecord | scan | backupService | mobilePlatform | tor",
    "ClientRefererHost": "string; the referer host",
    "ClientRefererPath": "string; the referer path requested by visitor",
    "ClientRefererQuery": "string; the referer query-string was requested by the visitor",
    "ClientRefererScheme": "string; the referer url scheme requested by the visitor",
    "ClientRequestHTTPHost": "string; the HTTP hostname requested by the visitor",
    "ClientRequestHTTPMethodName": "string; the HTTP method used by the visitor",
    "ClientRequestHTTPProtocol": "string; the version of HTTP protocol requested by the visitor",
    "ClientRequestPath": "string; the path requested by visitor",
    "ClientRequestQuery": "string; the query-string was requested by the visitor",
    "ClientRequestScheme": "string; the url scheme requested by the visitor",
    "Datetime": "int or string; the date and time the event occurred at the edge",
    "EdgeColoName": "string; the airport code of the Cloudflare datacenter that served this request",
    "EdgeResponseStatus": "int; HTTP response status code returned to browser",
    "Kind": "string; the kind of event, currently only possible values are: firewall",
    "MatchIndex": "int; rules match index in the chain",
    "Metadata": "object; additional product-specific information. Metadata is organized in key:value pairs. Key and Value formats can vary by Cloudflare security product and can change over time",
    "OriginResponseStatus": "int; HTTP origin response status code returned to browser",
    "OriginatorRayName": "string; the RayId of the request that issued the challenge/jschallenge",
    "RayName": "string; the RayId of the request",
    "RuleId": "string; the Cloudflare security product-specific RuleId triggered by this request",
    "Source": "string; the Cloudflare security product triggered by this request",
    "UserAgent": "string; visitor's user-agent string"
  },
  "success": true
}

Then you’ll append your Cloudflare configuration to the main.tf file:

$ cat <<EOF | tee -a main.tf

##################
### CLOUDFLARE ###
##################
provider "cloudflare" {
  version = "~> 2.0"
}

resource "cloudflare_logpush_job" "firewall_events_job" {
  name = "fwevents-logpush-job"
  zone_id = var.cf_zone_id
  enabled = true
  dataset = "firewall_events"
  logpull_options = "fields=RayName,Source,RuleId,Action,EdgeResponseStatusDatetime,EdgeColoName,ClientIP,ClientCountryName,ClientASNDescription,UserAgent,ClientRequestHTTPMethodName,ClientRequestHTTPHost,ClientRequestHTTPPath&timestamps=rfc3339"
  destination_conf = replace(sumologic_http_source.http_source.url,"https:","sumo:")
  ownership_challenge = "$LOGPUSH_CHALLENGE_TOKEN"
}
EOF

And add to the variables.tf file:

$ cat <<EOF | tee -a variables.tf

##################
### CLOUDFLARE ###
##################
variable "cf_zone_id" {
  default = "$CF_ZONE_ID"
}

Next we re-run terraform init to install the latest Cloudflare Terraform Provider Module. You’ll need to make sure you have at least version 2.6.0 as this is the version in which we added Logpush job support:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "cloudflare" (terraform-providers/cloudflare) 2.6.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

With the latest Terraform installed, we check out the plan and then apply:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

sumologic_collector.collector: Refreshing state... [id=108448215]
sumologic_http_source.http_source: Refreshing state... [id=150364538]

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # cloudflare_logpush_job.firewall_events_job will be created
  + resource "cloudflare_logpush_job" "firewall_events_job" {
      + dataset             = "firewall_events"
      + destination_conf    = "sumo://endpoint1.collection.eu.sumologic.com/receiver/v1/http/(redacted)"
      + enabled             = true
      + id                  = (known after apply)
      + logpull_options     = "fields=RayName,Source,RuleId,Action,EdgeResponseStatusDatetime,EdgeColoName,ClientIP,ClientCountryName,ClientASNDescription,UserAgent,ClientRequestHTTPMethodName,ClientRequestHTTPHost,ClientRequestHTTPPath&timestamps=rfc3339"
      + name                = "fwevents-logpush-job"
      + ownership_challenge = "(redacted)"
      + zone_id             = "(redacted)"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

$ terraform apply --auto-approve
sumologic_collector.collector: Refreshing state... [id=108448215]
sumologic_http_source.http_source: Refreshing state... [id=150364538]
cloudflare_logpush_job.firewall_events_job: Creating...
cloudflare_logpush_job.firewall_events_job: Creation complete after 3s [id=13746]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Success! Last step is to test your setup.

The following step assumes that you have the Cloudflare WAF turned on. Alternatively, you can create a Firewall Rule to match your request and generate a Firewall Event that way.

First make sure that livetail is running as described earlier:

$ livetail "_source=firewall-events-source"
### Authenticating ###
### Starting Live Tail session ###

Then in a browser make the following request https://example.com/<script>alert()</script>. You should see the following returned:

And a few moments later in livetail:

{"RayName":"58830d3f9945bc36","Source":"waf","RuleId":"958052","Action":"log","EdgeColoName":"LHR","ClientIP":"203.0.113.69","ClientCountryName":"gb","ClientASNDescription":"NTL","UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"upinatoms.com"}
{"RayName":"58830d3f9945bc36","Source":"waf","RuleId":"958051","Action":"log","EdgeColoName":"LHR","ClientIP":"203.0.113.69","ClientCountryName":"gb","ClientASNDescription":"NTL","UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"upinatoms.com"}
{"RayName":"58830d3f9945bc36","Source":"waf","RuleId":"973300","Action":"log","EdgeColoName":"LHR","ClientIP":"203.0.113.69","ClientCountryName":"gb","ClientASNDescription":"NTL","UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"upinatoms.com"}
{"RayName":"58830d3f9945bc36","Source":"waf","RuleId":"973307","Action":"log","EdgeColoName":"LHR","ClientIP":"203.0.113.69","ClientCountryName":"gb","ClientASNDescription":"NTL","UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"upinatoms.com"}
{"RayName":"58830d3f9945bc36","Source":"waf","RuleId":"973331","Action":"log","EdgeColoName":"LHR","ClientIP":"203.0.113.69","ClientCountryName":"gb","ClientASNDescription":"NTL","UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"upinatoms.com"}
{"RayName":"58830d3f9945bc36","Source":"waf","RuleId":"981176","Action":"drop","EdgeColoName":"LHR","ClientIP":"203.0.113.69","ClientCountryName":"gb","ClientASNDescription":"NTL","UserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36","ClientRequestHTTPMethodName":"GET","ClientRequestHTTPHost":"upinatoms.com"}

Note that for this one malicious request Cloudflare Logs actually sent 6 separate Firewall Events to Sumo Logic. The reason for this is that this specific request triggered a variety of different Managed Rules: #958051, 958052, 973300, 973307, 973331, and 981176.

Here's a demo of  launching livetail, making a malicious request in a browser, and then seeing the result sent from the Cloudflare Logpush job:

Our Firewall Analytics tool enables customers to quickly identify and investigate security threats using an intuitive interface. Until now, this tool had only been available to our Enterprise customers, who have been using it to get detailed insights into their traffic and better tailor their security configurations. Today, we are excited to make Firewall Analytics available to all paid plans and share details on several recent improvements we have made.

All paid plans are now able to take advantage of these capabilities, along with several important enhancements we’ve made to improve our customers’ workflow and productivity.

Previously, Enterprise customers could view 14 days of Firewall Analytics for their domains. Today we’re increasing that retention to 30 days, and again to 90 days in the coming months. Business and Professional plan zones will get 30 and 3 days of retention, respectively.

In addition to the extended retention, we are introducing adaptive sampling to guarantee that Firewall Analytics results are displayed in the Cloudflare Dashboard quickly and reliably, even when you are under a massive attack or otherwise receiving a large volume of requests.

Adaptive sampling works similar to Netflix: when your internet connection runs low on bandwidth, you receive a slightly downscaled version of the video stream you are watching. When your bandwidth recovers, Netflix then upscales back to the highest quality available.

Firewall Analytics does this sampling on each query, ensuring that customers see the best precision available in the UI given current load on the zone. When results are sampled, the sampling rate will be displayed as shown below:

As adoption of our expressive Firewall Rules engine has grown, one consistent ask we’ve heard from customers is for a more streamlined way to see all Firewall Events generated by a specific rule. Until today, if a malicious request matched multiple rules, only the last one to execute was shown in the Activity Log, requiring customers to click into the request to see if the rule they’re investigating was listed as an “Additional match”.

To streamline this process, we’ve changed how the Firewall Analytics UI interacts with the Activity Log. Customers can now filter by a specific rule (or any other criteria) and see a row for each event generated by that rule. This change also makes it easier to review all requests that would have been blocked by a rule by creating it in Log mode first before changing it to Block.

When our customers write rules to block undesired, automated traffic they want to make sure they’re not blocking or challenging desired traffic, e.g., humans wanting to make a purchase should be allowed but not bots scraping pricing.

To help customers determine what percent of CAPTCHA challenges returned to users may have been unnecessary, i.e., false positives, we are now showing the Challenge Solve Rate (CSR) for each rule. If you’re seeing rates higher than expected, e.g., for your Bot Management rules, you may want to relax the rule criteria. If the rate you see is 0% indicating that no CAPTCHAs are being solved, you may want to change the rule to Block outright rather than challenge.

Hovering over the CSR rate will reveal the number of CAPTCHAs issued vs. solved:

Business and Enterprise customers can now export a set of 500 events from the Activity Log. The data exported are those events that remain after any selected filters have been applied.

Sometimes the columns shown in the Activity Log do not contain the details you want to see to analyze the threat. When this happens, you can now click “Edit Columns” to select the fields you want to see. For example, a customer diagnosing a Bot related issue may want to also view the User-Agent and the source country whereas a customer investigating a DDoS attack may want to see IP addresses, ASNs, Path, and other attributes. You can now customize what you’d like to see as shown below.

We would love to hear your feedback and suggestions, so feel free to reach out to us via our Community forums or through your Customer Success team.

If you’d like to receive more updates like this one directly to your inbox, please subscribe to our Blog!

We have done away with those pesky modals, providing a much smoother user experience. To review more detailed information about an event, you simply click anywhere on the event list row.

In the expanded view, you are provided with all the information you may need to identify or diagnose issues with your Firewall or find more details about a potential threat to your application.

Cloudflare has several Firewall features to give customers granular control of their security. With this control comes some complexity when debugging why a request was stopped by the Firewall. To help clarify what happened, we have provided an “Additional matches” count at the bottom for events triggered by multiple services or rules for the same request. Clicking the number expands a list showing each rule and service along with the corresponding action.

This is one of my favourite parts of our new Firewall Event Log. Many of our customers have expressed their frustration with the difficulty of pinpointing specific events. This is where our new search capabilities come into their own. Customers can now filter and freeform search for any field that is visible in a Firewall Event!

Let’s say you want to find all the requests originating from a specific ISP or country where your Firewall Rules issued a JavaScript challenge. There are two different ways to do this in the UI.

Firstly, when in the detail view, you can create an include or exclude filter for that field value.

Secondly, you can create a freeform filter using the “+ Add Filter” button at the top, or edit one of the already filtered fields:

As illustrated above, with our WAF Managed Rules enabled in log only, we can see all the rules which would have triggered if this was a legitimate attack. This allows you to confirm that your configuration is working as expected.

In our old Firewall Event Log, to find an event, users had to traverse through many pages to find Events from a specific date. The last major change we have added is the capability to select a time window to view events between two points in time over the last 2 weeks. In the time selection window, Free and Pro customers can choose a 24 hour time window and our Business customers can view up to 72 hours.

We need your help! Please feel free to leave any feedback on our Community forums, or open a Support ticket with any problems you find. Your feedback is critical to our product improvement process, and we look forward to hearing from you.

Project Galileo sites are protected by the Cloudflare Firewall and Advanced DDoS Protection which contain a number of features that can be used to detect and mitigate different types of attack and suspicious traffic. The following table shows how each of these features contributed to the protection of sites on Project Galileo.

Although not the most impressive in terms of blocked requests, the WAF is the most interesting as it identifies and blocks malicious requests, based on heuristics and rules that are the result of seeing attacks across all of our customers and learning from those. The WAF is available to all of our paying customers, protecting them against 0-days, SQL/XSS exploits and more. For the Project Galileo customers the WAF rules blocked more than 4.5 million requests in the month that we looked at, matching over 130 WAF rules and approximately 150k requests per day.

Heat map showing the attacks seen on customer sites (rows) per day (columns)

This heat map may initially appear confusing but reading one is easy once you know what to expect so bear with us! It is a table where each line is a website on Project Galileo and each column is a day. The color represents the number of requests triggering WAF rules - on a scale from 0 (white) to a lot (dark red). The darker the cell, the more requests were blocked on this day.

We observe malicious traffic on a daily basis for most websites we protect. The average Project Galileo site saw malicious traffic for 27 days in the 1 month observed, and for almost 60% of the sites we noticed daily events.

Fortunately, the vast majority of websites only receive a few malicious requests per day, likely from automated scanners. In some cases, we notice a net increase in attacks against some websites - and a few websites are under a constant influx of attacks.

Heat map showing the attacks blocked for each WAF rule (rows) per day (columns)

This heat map shows the WAF rules that blocked requests by day. At first, it seems some rules are useless as they never match malicious requests, but this plot makes it obvious that some attack vectors become active all of a sudden (isolated dark cells). This is especially true for 0-days, malicious traffic starts once an exploit is published and is very active on the first few days. The dark active lines are the most common malicious requests, and these WAF rules protect against things like XSS and SQL injection attacks.

A DoS attack prevents legitimate visitors from accessing a website by flooding it with bad traffic.  Due to the way Cloudflare works, websites protected by Cloudflare are immune to many DoS vectors, out of the box. We block layer 3 and 4 attacks, which includes SYN floods and UDP amplifications. DNS nameservers, often described as the Internet’s phone book, are fully managed by Cloudflare, and protected - visitors know how to reach the websites.

Line plot - requests per second to a website under DoS attack

Can you spot the attack?

As for layer 7 attacks (for instance, HTTP floods), we rely on Gatebot, an automated tool to detect, analyse and block DoS attacks, so you can sleep. The graph shows the requests per second we received on a zone, and whether or not it reached the origin server. As you can see, the bad traffic was identified automatically by Gatebot, and more than 1.6 million requests were blocked as a result.

For websites with specific requirements we provide tools to allow customers to block traffic to precisely fit their needs. Customers can easily implement complex logic using Firewall Rules to filter out specific chunks of traffic, block IPs / Networks / Countries using Access Rules and Project Galileo sites have done just that. Let’s see a few examples.

Firewall Rules allows website owners to challenge or block as much or as little traffic as they desire, and this can be done as a surgical tool “block just this request” or as a general tool “challenge every request”.

For instance, a well-known website used Firewall Rules to prevent twenty IPs from fetching specific pages. 3 of these IPs were then used to send a total of 4.5 million requests over a short period of time, and the following chart shows the requests seen for this website. When this happened Cloudflare, mitigated the traffic ensuring that the website remains available.

Cumulative line plot. Requests per second to a website

Another website, built with WordPress, is using Cloudflare to cache their webpages. As POST requests are not cacheable, they always hit the origin machine and increase load on the origin server - that’s why this website is using firewall rules to block POST requests, except on their administration backend. Smart!

Website owners can also deny or challenge requests based on the visitor’s IP address, Autonomous System Number (ASN) or Country. Dubbed Access Rules, it is enforced on all pages of a website - hassle-free.

For example, a news website is using Cloudflare’s Access Rules to challenge visitors from countries outside of their geographic region who are accessing their website. We enforce the rules globally even for cached resources, and take care of GeoIP database updates for them, so they don’t have to.

The Zone Lockdown utility restricts a specific URL to specific IP addresses. This is useful to protect an internal but public path being accessed by external IP addresses. A non-profit based in the United Kingdom is using Zone Lockdown to restrict access to their WordPress’ admin panel and login page, hardening their website without relying on non official plugins. Although it does not prevent very sophisticated attacks, it shields them against automated attacks and phishing attempts - as even if their credentials are stolen, they can’t be used as easily.

Cloudflare acts as a CDN, caching resources and happily serving them, reducing bandwidth used by the origin server … and indirectly the costs. Unfortunately, not all requests can be cached and some requests are very expensive to handle. Malicious users may abuse this to increase load on the server, and website owners can rely on our Rate Limit to help them: they define thresholds, expressed in requests over a time span, and we make sure to enforce this threshold. A non-profit fighting against poverty relies on rate limits to protect their donation page, and we are glad to help!

Last but not least, one of Cloudflare’s greatest assets is our threat intelligence. With such a wide lens of the threat landscape, Cloudflare uses our Firewall data, combined with machine learning to curate our IP Reputation databases. This data is provided to all Cloudflare customers, and is configured through our Security Level feature. Customers then may define their threshold sensitivity, ranging  from Essentially Off to I’m Under Attack. For every incoming request, we ask visitors to complete a challenge if the score is above a customer defined threshold. This system alone is responsible for 25% of the requests we mitigated: it’s extremely easy to use, and it constantly learns from the other protections.

When taken together, the Cloudflare Firewall features provide our Project Galileo customers comprehensive and effective security that enables them to ensure their important work is available. The majority of security events were handled automatically, and this is our strength - security that is always on, always available, always learning.

Last year, we launched Spectrum. Spectrum brought the power of our DDoS and firewall features to all TCP ports and services. Spectrum for TCP allows you to protect your SSH services, gaming protocols, and as of last month, even FTP servers. We’ve seen customers running all sorts of applications behind Spectrum, such as Bitfly, Nicehash, and Hypixel.

This is great if you're running TCP services, but plenty of our customers also have workloads running over UDP. As an example, many multiplayer games prefer the low cost and lighter weight of UDP and don't care about whether packets arrive or not.

UDP applications have historically been hard to protect and secure, which is why we built Spectrum for UDP. Spectrum for UDP allows you to protect standard UDP services (such as RDP over UDP), but can also protect any custom protocol you come up with! The only requirement is that it uses UDP as an underlying protocol.

To configure on the dashboard, simply switch the application type from TCP to UDP:

With Spectrum, we terminate the connection and open a new one to your origin. But, what if you want to still see who's actually connecting to you? For TCP, there's Proxy Protocol. Whilst initially introduced by HAProxy, it has since been adopted by more parties, such as nginx. We added support late 2018, allowing you to easily read the client's IP and port from a header that precedes each data stream.

Unfortunately, there is no equivalent for UDP, so we're rolling our own. Due to the fact that UDP is connection-less, we can't get away with the Proxy Protocol approach for TCP, which prepends the entire stream with one header. Instead, we are forced to prepend each packet with a small header that specifies:

• the original client IP

• the Spectrum IP

• the original client port

• the Spectrum port

Schema representing a UDP packet prefaced with our Simple Proxy Protocol header.

0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Magic Number         |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                                                               |
+                                                               +
|                                                               |
+                         Client Address                        +
|                                                               |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                                                               |
+                                                               +
|                                                               |
+                         Proxy Address                         +
|                                                               |
+                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               |         Client Port           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Proxy Port          |          Payload...           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Simple Proxy Protocol is turned off by default, which means UDP packets will arrive at your origin as if they were sent from Spectrum. To enable, just enable it on your Spectrum app.

We're excited about launching this and and even more excited to see what you'll build and protect with it. In fact, what if you could build serverless services on Spectrum, without actually having an origin running? Stay tuned for some cool announcements in the near future.

Spectrum for UDP is currently an Enterprise-only feature. To get UDP enabled for your account, please reach out to your account team and we’ll get you set up.

One more thing... if you’re at GDC this year, say hello at booth P1639! We’d love to talk more and learn about what you’d like to do with Spectrum.

Recently we launched Firewall Rules, a new feature that allows you to construct expressions that perform complex matching against HTTP requests and then choose how that traffic is handled. As a Firewall feature you can, of course, block traffic. The expressions we support within Firewall Rules along with powerful control over the order in which they are applied allows complex new behaviour.

In this blog post I tell the story of Cloudflare’s Page Rules mechanism and how Firewall Rules came to be. Along the way I’ll look at the technical choices that led to us building the new matching engine in Rust.

Cloudflare offers two types of firewall for web applications, a managed firewall in the form of a WAF where we write and maintain the rules for you, and a configurable firewall where you write and maintain rules. In this article, we will focus on the configurable firewall.

One of the earliest Cloudflare firewall features was the IP Access Rule. It dates backs to the earliest versions of the Cloudflare Firewall and simply allows you to block traffic from specific IP addresses:

if request IP equals 203.0.113.1 then block the request

As attackers and spammers frequently launched attacks from a given network we also introduced the ASN matching capability:

if request IP belongs to ASN 64496 then block the request

We also allowed blocking ranges of addresses defined by CIDR notation when an IP was too specific and an ASN too broad:

if request IP is within 203.0.113.0/24 then block the request

Blocking is not the only action you might need and so other actions are available:

• Allowlist = apply no other firewall rules and allow the request to pass this part of the firewall

• Challenge = issue a CAPTCHA and if this is passed then allow the request but otherwise deny. This would be used to determine if the request came from a human operator

• JavaScript challenge = issue an automated JavaScript challenge and if this is passed then allow the request. This would be used to determine if the request came from a simple stateless bot (perhaps a wget or curl script)

• Block = deny the request

Cloudflare also has Page Rules. Page Rules allow you to match full URIs and then perform actions such as redirects or to raise the security level which can be considered firewall functions:

if request URI matches /nullroute then redirect to http://127.0.0.1

Cloudflare also added GeoIP information within an HTTP header and the firewall was extended to include that:

if request IP originates from county GB then CAPTCHA the request

All of the above existed in Cloudflare pre-2014, and then during 2016 we set about to identify the most commonly requested firewall features (according to Customer Support tickets and feedback from paying customers) and provide a self-service solution. From that analysis, we added three new capabilities during late 2016: Rate Limiting, User Agent Rules, and Zone Lockdown.

Whilst Cloudflare automatically stops very large denial of service attacks, rate limiting allowed customers to stop smaller attacks that were a real concern to them but were low enough volume that Cloudflare’s DDoS defences were not being applied.

if request method is POST and request URI matches /wp-admin/index.php and response status code is 403 and more than 3 requests like this occur in a 15 minute time period then block the traffic for 2 hours

User Agent rules are as simple as:

if request user_agent is `Fake User Agent` then CAPTCHA the request

Zone Lockdown, however was the first default deny feature. The Cloudflare Firewall could be thought of as “allow all traffic, except where a rule exists to block it”. Zone Lockdown is the opposite “for a given URI, block all traffic, except where a rule exists to allow it”.

Zone Lockdown allowed customers could to block access to a public website for all but a few IP addresses or IP ranges. For example, many customers wanted access to a staging website to only be available to their office IP addresses.

if request URI matches https://staging.example.com/ and request IP not in 203.0.113.0/24 then block the request

Finally, an Enterprise customer could also contact Cloudflare and have a truly bespoke rule created for them within the WAF engine.

The firewall worked well for simple mitigation, but it didn’t fully meet the needs of our customers.

Each of the firewall features had targeted a single attribute, and the interfaces and implementations reflected that. Whilst the Cloudflare Firewall had evolved to solve a problem as each problem arose, these did not work together. In late 2017 you could sum up the firewall capabilities as:

You can block any attack traffic on any criteria, so long as you only pick one of:

• IP

• CIDR

• ASN

• Country

• User Agent

• URI

We saw the problem, but how to fix it?

We match our firewall rules in two ways:

• Lookup matching

• String pattern matching

Lookup matching covers the IP, CIDR, ASN, Country and User Agent rules. We would create a key in our globally distributed key/value data store Quicksilver, and store the action in the value:

Key   = zone:www.example.com_ip:203.0.113.1
Value = block

When a request for www.example.com is received, we look at the IP address of the client that made the request, construct the key and perform the lookup. If the key exists in the store, then the value would tell us what action to perform, in this case if the client IP were 203.0.113.1 then we would block the request.

Lookup matching is a joy to work with, it is O(1) complexity meaning that a single request would only perform a single lookup for an IP rule regardless of how many IP rules a customer had. Whilst most customers had a few rules, some customers had hundreds of thousands of rules (typically created automatically by combining fail2ban or similar with a Cloudflare API call).

Lookups work well when you are only looking up a single value. If you need to combine an IP and a User Agent we would need to produce keys that composed these values together. This massively increases the number of keys that you need to publish.

String pattern matching occurs where URI matching is required. For our Page Rules feature this meant combining all of the Page Rules into a single regular expression that we would apply to the request URI whilst handling a request.

If you had Page Rules that said (in order):

• Match */wp-admin/index.php and then block

• Then match */xmlrpc.php and then block

These are converted into:

^(?<block__1>(?:.*/wp-admin/index.php))|(?<block__2>(?:.*/xmlrpc.php))$

Yes, you read that correctly. Each Page Rule was appended to a single regular expression in the order of execution, and the naming group is used as an overload for the desired action.

This works surprisingly well as regular expression matching can be simple and fast especially when the regular expression matches against a single value like the URI, but as soon as you want to match the URI plus an IP range it becomes less obvious how to extend this.

This is what we had, a set of features that worked really well providing you want to match a single property of a request. The implementation also meant that none of these features could be trivially extended to embrace multiple properties at a time. We needed something else, a fast way to compute if a request matches a rule that could contain multiple properties as well as pattern matching.

Over time Cloudflare engineers authored internal posts exploring how a new matching engine might work. The first thing that occurred to every engineer was that the matching must be an expression. These ideas followed a similar approach which we would construct an expression within JSON as a DSL (Domain Specific Language) of our expression language. This DSL could describe matching a request and a UI could render this, and a backend could process it.

Early proposals looked like this:

{
  "And": [
    {
      "Equals"{
        "host": "www.example.com"
      }
    },
    "Or": [
      {
        "Regex": {
          "path": "^(?: .*/wp-admin/index.php)$"
        }
      }{
        "Regex": {
          "path": "^(?: .*/xmlrpc.php)$"
        }
      }
    ]
  ]
}

The JSON describes an expression that computers can easily turn into a rule to apply, but people find this hard to read and work with.

As we did not wish to display JSON like this in our dashboard we thought about how we might summarise it for a UI:

if request host equals www.example.com and (request path matches ^(?:.*/wp-admin/index.php)$ or request path matches ^(?:.*/xmlrpc.php)$)

And there came an epiphany. As engineers working we’ve seen an expression language similar to this before, so may I introduce to you our old friend Wireshark®.

Wireshark is a network protocol analyzer. To use it you must run a packet capture to record network traffic from a capture device (usually a network card). This is then saved to disk as a .pcap file which you subsequently open in the Wireshark GUI. The Wireshark GUI has a display filter entry box, and when you fill in a display filter the GUI will dissect the saved packet capture such that it will determine which packets match the expression and then show those in the GUI.

But we don’t need to do that. In fact, for our scenario that approach does not work as we have a firewall and need to make decisions in real-time as part of the HTTP request handling rather than via the packet capture process.

For Cloudflare, we would want to use something like the expression language that is the Wireshark Display Filters but without the capture and dissection as we would want to do this potentially thousands of times per request without noticeable delay.

If we were able to use a Wireshark-style expression language then we can reduce the JSON encapsulated expression above to:

http.host eq "www.example.com" and (http.request.path ~ "wp-admin/index\.php" or http.request.path ~ "xmlrpc.php")

This is human readable, machine parseable, succinct.

It also benefits from being highly similar to Wireshark. For security engineers used to working with Wireshark when investigating attacks it offers a degree of portability from an investigation tool to a mitigation engine.

To make this work we would need to collect the properties of the request into a simple data structure to match the expressions against. Unlike the packet capture approach we run our firewall within the context of an HTTP server and the web server has already computed the request properties, so we can avoid dissection and populate the fields from the web server knowledge:

session=8521F670545D7865F79C3D7BED C29CCE;-background=light

www.example.com

GET

/articles/index?section=539061&expand=comments

/articles/index

section=539061&expand=comments

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36

203.0.113.1

64496

GB

true

With a table of HTTP request properties and an expression language that can provide a matching expression we were 90% of the way towards a solution! All we needed for the last 90% was the matching engine itself that would provide us with an answer to the question: Does this request match one of the expressions?

Enter wirefilter.

Wirefilter is the name of the Rust library that Cloudflare has created, and it provides:

• The ability for Cloudflare to define a set of fields of types, i.e. ip.src is a field of type IPAddress

• The ability to define a table of properties from all of the fields that are defined

• The ability to parse an expression and to say whether it is syntactically valid, whether the fields in the expression are valid against the fields defined, and whether the operators used for a field are valid for the type of the field

• The ability to apply an expression to a table and return a true|false response indicating whether the evaluated expression matches the request

It is named wirefilter as a hat tip towards Wireshark for inspiring our Wireshark-like expression language and also because in our context of the Cloudflare Firewall these expressions act as a filter over traffic.

The implementation of wirefilter allows us to embed this matching engine within our REST API which is written in Go:

// scheme stores the list of fields and their types that an expression can use
var scheme = filterexpr.Scheme{
	"http.cookie":                     filterexpr.TypeString,
	"http.host":                       filterexpr.TypeString,
	"http.referer":                    filterexpr.TypeString,
	"http.request.full_uri":           filterexpr.TypeString,
	"http.request.method":             filterexpr.TypeString,
	"http.request.uri":                filterexpr.TypeString,
	"http.request.uri.path":           filterexpr.TypeString,
	"http.request.uri.query":          filterexpr.TypeString,
	"http.user_agent":                 filterexpr.TypeString,
	"http.x_forwarded_for":            filterexpr.TypeString,
	"ip.src":                          filterexpr.TypeIP,
	"ip.geoip.asnum":                  filterexpr.TypeNumber,
	"ip.geoip.country":                filterexpr.TypeString,
	"ssl":                             filterexpr.TypeBool,
}

Later we validate expressions provided to the API:

// expression here is a string that may look like:
// `ip.src eq 203.0.113.1`
expressionHash, err := filterexpr.ValidateFilter(scheme, expression)
if fve, ok := err.(*filterexpr.ValidationError); ok {
	validationErrs = append(validationErrs, fve.Ascii)
} else if err != nil {
	return nil, stderrors.Errorf("failed to validate filter: %v", err)
}

This tells us whether the expression is syntactically correct and also whether the field operators and values match the field type. If the expression is valid then we can use the returned hash to determine uniqueness (the hash is generated inside wirefilter so that uniqueness can ignore blanks and minor differences).

The expressions are then published to our global network of PoPs and are consumed by Lua within our web proxy. The web proxy has the same list of fields that the API does, and is now responsible for building the table of properties from the context within the web proxy:

-- The `traits` table defines the mapping between the fields and
-- the corresponding values from the nginx evaluation context.
local traits = {
   ['http.host'] = field.str(function(ctx) return ctx.host end),
   ['http.cookie'] = field.str(function(ctx)
      local value = ctx.req_headers.cookie or ''
      if type(value) == 'table' then
         value = table.concat(value, ";")
      end
      return value
   end),
   ['http.referer'] = field.str(function(ctx) return ctx.req_headers.referer or '' end),
   ['http.request.method'] = field.str(function(ctx) return ctx.method end),
   ['http.request.uri'] = field.str(function(ctx)
      return ctx.rewrite_uri or ctx.request_uri
   end),
   ['http.request.uri.path'] = field.str(function(ctx)
      return ctx.uri or '/'
   end),
   ...

With this per-request table describing a request we can see test the filters. In our case what we’re doing here is:

• Fetch a list of all the expressions we would like to match against the request

• Check whether any expression, when applied via wirefilter to the table above, return true as having matched

• For all matched expressions check the associated actions and their priority

The actions are not part of the matching itself. Once we have a list of matched expressions we determine which action takes precedence and that is the one that we will execute.

Wirefilter then, is a generic library that provides this matching capability that we’ve plugged into our Go APIs and our Lua web proxy, and we use that to power the Cloudflare Firewall.

We chose Rust for wirefilter as early in the project we recognised that if we attempted to make implementations of this in Go and Lua, that it would result in inconsistencies that attackers may be able to exploit. We needed our API and edge proxy to behave exactly the same. For this needed a library, both could call and we could choose one of our existing languages at the edge like C, C++, Go, Lua or even implement this not as a library but as a worker in JavaScript. With a mixed set of requirements of performance, memory safety, low memory use, and the capability to be part of other products that we’re working on like Spectrum, Rust stood out as the strongest option.

With a library in place and the ability to now match all HTTP traffic, how to get that to a public API and UI without diluting the capability? The problems that arose related to specificity and mutual exclusion.

In the past all of our firewall rules had a single dimension to them: i.e. act on IP addresses. And this meant that we had a single property of a single type and whilst there were occasionally edge cases for the most part there were strategies to answer the question “Which is the most specific rule?”. I.e. an IP address is more specific then a /24 which is more specific than a /8. Likewise with URI matching an overly simplistic strategy is that the longer a URI the more specific it is. And if we had 2 IP rules, then only 1 could ever have matched as a request does not come from 2 IPs at once so mutual exclusion is in effect.

The old system meant that given 2 rules, we could implicitly and trivially say “this rule is most specific so use the action associated with this rule”.

With wirefilter powering Firewall Rules, it isn’t obvious that an IP address is more or less specific when compared to a URI. It gets even more complex when a rule can have negation, as a rule that matches a /8 is less specific than a rule that does not match a single IP (the whole address space except this IP - one of the gotchas of Firewall Rules is also a source of it’s power; you can invert your firewall into a positive security model.

As we couldn’t answer specificity using the expression alone, we needed another aspect of the Firewall Rule to provide us this guidance and we realised that customers already had a mechanism to tell us which rules were important… the action.

Given a set of rules, we logically have ordered them according to their action (Log has highest priority, Block has lowest):

1. Log

2. Allow

3. Challenge (CAPTCHA)

4. JavaScript Challenge

5. Block

For the vast majority of scenarios this proves to be good enough.

What about when that isn’t good enough though? Do we have examples of complex configuration that break that approach? Yes!

Because the expression language within Firewall Rules is so powerful, and we can support many Firewall Rules, it means that we can now create different firewall configuration for different parts of a web site. i.e. /blog could have wholly different rules than /shop, or for different audiences, i.e. visitors from your office IPs might be allowed on a given URI but everyone else trying to access that URI may be blocked.

In this scenario you need the ability to say “run all of these rules first, and then run the other rules”.

In single machine firewalls like iptables, OS X Firewall, or your home router firewall, the firewall rules were explicitly ordered so that when you match the first rule it terminates execution and you won’t hit the next rule. When you add a new rule the entire set of rules is republished and this helps to guarantee this behaviour. But this approach does not work well for a Cloud Firewall as a large website with many web applications typically also has a large number of firewall rules. Republishing all of these rules in a single transaction can be slow and if you are adding lots of rules quickly this can lead to delays to the final state being live.

If we published individual rules and supported explicit ordering, we risked race conditions where two rules that both were configured in position 4 might exist at the same time and the behaviour if they matched the request would be non-determinable.

We solved this by introducing a priority value, where 1 is the highest priority and as an int32 you can create low priority rules all the way down to priority = 2147483647. Not providing a priority value is the equivalent of “lowest” and runs after all rules that have a priority.

Priority does not have to be a unique value within Firewall Rules. If two rules are of equal priority then we resort to the order of precedence of the actions as defined earlier.

This provides us a few benefits:

1. Because priority allows rules that share a priority to exist we can publish rules 1 at a time… when you add a new rule the speed at which we deploy that globally is not affected by the number of rules you already have.

2. If you do have existing rules in a system that does sequentially order the rules, you can import those into Firewall Rules and preserve their order, i.e. this rule should always run before that rule.

3. But you don’t have to use priority exclusively for ordering as you can also use priority for grouping. For example you may say that all spammers are priority=10000 and all trolls are priority = 5000.

Finally… let’s look at those fields again, http.request.path notice that http prefix? By following the naming convention Wireshark has (see their Display Filter Reference) we have not limited this firewall capability solely to a HTTP web proxy. It is a small leap to imagine that if a Spectrum  application declares itself as running SMTP that we could also define fields that understand SMTP and allow filtering of traffic on other application protocols or even at layer 4.

What we have built in Firewall Rules gives us these features today:

• Rich expression language capable of targeting traffic precisely and in real-time

• Fast global deployment of individual rules

• A lot of control over the management and organisation of Firewall Rules

And in the future, we have a product that can go beyond HTTP and be a true Cloud Firewall for all protocols…the Cloudflare Firewall with Firewall Rules.

We have reorganised our features into meaningful pages: Events, Firewall Rules, Managed Rules, Tools, and Settings. Our customers will see an Overview tab, which contains our new Firewall Analytics, detailed below.

All the features you know and love are still available, and can be found in one of the four new tabs. Here is a breakdown of their new locations.

If the new sub navigation has not appeared, you may need to re-login to the dashboard or clear your browser’s cookies.

Insights into security events are critical for monitoring the health of your web applications. Furthermore, distinguishing between actual threats from false positives is essential for maintaining an optimal security configuration. Today, we are very pleased to announce our new Firewall Analytics which will help our Enterprise customers get detailed insights into firewall events, helping them to tailor their security configurations more effectively

Our new Firewall Analytics now enables our Enterprise customers to:

• visualise and analyse Firewall Events in one place to better understand their threat landscape

• identify, mitigate, and review attacks more effectively

After speaking with many of our customers, we learned a lot about their processes to identify and analyse attacks and the kinds of insights they needed to improve these processes. We then translated these learnings into useful features and charts that would help answer some of the most common questions such as ‘What kinds of security events occurred in a certain time frame?’ and ‘What caused a spike in a certain type of security event?’.  

Firewall Analytics and Firewall Configuration can be found together in the Firewall tab. A tight feedback loop between Firewall configuration and the resulting events allow for rapid iteration, ideal for security-focused teams.

To best demonstrate the power of Firewall Analytics, here’s a workflow that would answer a popular question our customers ask: “Why did I have a spike in threats?”. In the screenshot below, we can see a set of activity which triggered a number of ‘Blocks’ events:

To minimize the possibility of polluting our TopN statistics with event types other than ‘Block’ and get the most accurate diagnostic information, we will need to filter down to just ‘Block’ actions.

Now that only Block events are displayed, checking the Service Breakdowns will help us to identify which of our Firewall features was triggered.

From the Events breakdown, we can see that the Block events were triggered by a Country Block configured within Access Rules. Digging deeper and looking at our TopN breakdowns, we start to get a much more granular understanding of which Networks, IPs, User-Agents, Paths etc, were targeted.

Looking at our TopN breakdowns, we start to get a much more granular understanding of which Networks, IPs, User-Agents, Paths etc, were targeted.

From here, we can see that there are two specific IP addresses which were targeting my application to “/”.

To get the most detailed information, we can drill down further in the refreshed Firewall Event log, now controlled inline.

Whilst these TopNs and filters are great for clearly identifiable threats, they can also help identify false positives. Using the power of Cloudflare’s filters, it is possible to add a user-defined filter, which can be a RayID, User-Agent or IP address.

This is just one example of how the new Firewall Analytics can help expedite the process of identifying and mitigating threats. Firewall Analytics is now live for all Enterprise customers. Let us know your feedback by reaching out to your Enterprise Account Team.