Today we are launching Cloudflare’s paid public bug bounty program. We believe bug bounties are a vital part of every security team’s toolbox and have been working hard on improving and expanding our private bug bounty program over the last few years. The first iteration of our bug bounty was a pure vulnerability disclosure program without cash bounties. In 2018, we added a private bounty program and are now taking the next step to a public program.

Starting today, anyone can report vulnerabilities related to any Cloudflare product to our public bug bounty program, hosted on HackerOne’s platform.

Let's walk through our journey so far.

In 2014, when the company had fewer than 100 employees, we created a responsible disclosure policy to provide a safe place for security researchers to submit potential vulnerabilities to our security team, with some established rules of engagement. A vulnerability disclosure policy is an important first step for a company to take because it is an invitation to researchers to look at company assets without fear of repercussions, provided the researchers follow certain guidelines intended to protect everyone involved. We still stand by that policy and welcome reports related to all of our services through that program.

Over the years, we received many great reports through that program which led to improvements in our products. However, one early challenge we faced was that researchers struggled to understand our infrastructure and products. Unlike most of the public programs of that era when we launched, our services were not made up primarily of public facing web applications or even mobile applications; our products were primarily network security and performance solutions that operated as a proxy layer in front of customer resources.

Understanding where Cloudflare fits into the HTTP request/response pipeline can get very challenging with multiple products enabled. And because we did not provide much supporting documentation about how our products worked, and had scoped the program to broadly encompass everything, we left researchers to figure out our complicated products on their own. As a result, most of the reports we received over those early years came from people who saw something that seemed atypical to them, but in our view was not actually a vulnerability in need of repair. We dedicated a tremendous amount of time to triaging false positive reports and helping the researchers understand their errors.

Lesson #1 from that experience: we needed to provide much more detail about our products, so researchers could understand how to dig into our products and identify true vulnerabilities. For example, when a zone is being onboarded to Cloudflare, even before ownership is determined, Cloudflare will display the DNS records for the zone. These DNS records are public information, but researchers have filed reports claiming that this is an information leakage issue. The same results can be obtained with open-source tools. This does not affect existing Cloudflare zones since Cloudflare protects the actual origin IPs from being leaked.

We see the same types of issues come up regularly with the platforms that some companies use to assess the security of their vendors. Off-the-shelf scanners will inaccurately detect vulnerabilities in our platforms because our services not only sit in front of our environment, but the environments of many thousands of customers of all shapes and sizes. We encourage researchers not to use those tools and instead learn more about how our services work.

Leaving researchers to their own devices, and failing to invest in helping them understand how our product worked, led to a poor signal-to-noise ratio. At the time of writing, 1,197 reports have been submitted to our vulnerability disclosure program. Of those, only 158 resulted in points going to the researcher for a valid report. With only 13% of reports being valid, we needed to invest in improving the signal-to-noise ratio by helping our researchers before we could expand our program.

Early on, we rewarded researchers with a unique “Cloudflare bug hunter” T-shirt after we validated a vulnerability report. We thought it was a nice gesture at a low cost for people who found security bugs. In practice, when we factored in shipping issues, passing through customs, and wrong sizes, it was a nightmare. Shipping turned out to be such a challenge that we sometimes resorted to hand-delivering T-shirts to researchers when attending conferences. It was nice to meet the researchers in person, but not a scalable solution!

We have always felt that rewarding good security research deserved more than a T-shirt. We also believed that financially supporting researchers would incentivize higher quality reports and deeper security research.

Righhhhhhht. I think most of us would rather have the cash.

— Mrs. Y. (@MrsYisWhy) February 26, 2017

In order to learn the ropes of operating a paid bounty, in 2018 we opened a private bug bounty program and spent the past few years optimizing.

Our end goal has always been to reach a level of maturity that would allow us to operate a paid public program. To reach that goal we needed to learn how to best support the researchers and improve the signal-to-noise ratio of reports, while building our internal processes to track and remediate a stream of reported vulnerabilities with our engineering teams.

When we launched the private bug bounty we included all Cloudflare products eligible for rewards, and by mid-January 2022 we had paid out \$211,512 in bounties. We started the program by inviting a few researchers and slowly added more overtime. This helped us fine tune our policies and documentation and create a more scalable vulnerability management process internally. Our most prolific participant has earned \$54,800 in bounty rewards, and the signal-to-noise ratio has improved to 68%, with 292 of our 430 total reports receiving a reward.

The success of our private program has largely been the result of consistent effort from the members of our Product Security team to improve our internal handling of issues, and through projects to improve the researcher experience. All bug bounty reports are triaged and validated by members of the security team along with some initial support from HackerOne. Once triaged, the security issues flow through our vulnerability management program, where unique issues are tracked in a single ticket that can be shared outside the security team. To support a scaling program and company, automation was baked into this process and integrations were implemented with our ticketing system.

In this example of a valid report, a ticket was filed containing all the information the reporter provided to HackerOne. Once a report is acknowledged and a VULN ticket is filed, we pay out the security researcher to ensure they receive the reward in a timely fashion.

Each ticket is assigned to an Engineering Owner and Security Owner who share the responsibility for remediating the vulnerability. Early on in the process, a service-level agreement (SLA) and remediation timeline are determined by the severity of the issue. If the bug is determined to be of critical severity, it's all hands on deck to fix the issue.

After initial assignment and SLA determination, the open tickets are reviewed weekly by both Engineering and Security to ensure that problems are being addressed in line with the SLA.

We’ve been working hard to improve the researcher experience. We’ve seen that immediately paying researchers led to a huge improvement in satisfaction compared to waiting weeks or months for a T-shirt. Likewise, it was even more frustrating for security researchers to work hard on an issue and then find out it was out of scope. To address this we constantly update our scope section as we get more out-of-scope reports. Our policy page is now much clearer. We also have treasure maps for some products pointing at major risk areas, and even put together a test site where researchers can test theories.

Ultimately, the success of our private bug bounty came down to the researchers who put in the effort to look for issues. Cloudflare thanks all 419 researchers who have participated in our bug bounty program so far, with a special shout out to the top 10 researchers in the program:

• zeroxyele

• esswhy

• turla

• ginkoid

• albertspedersen

• ryotak

• base_64

• dward84

• ninetynine

• albinowax

Here’s how our total bounty amounts grew as we improved our program:

2018 - $4,500 2019 - $25,425 2020 - $78,877 2021 - $101,075

The current breakdown of bounty awards for primary targets based on issue severity is listed below. (All amounts in USD)

Because of how our services work, you need to have our products deployed in a test environment in order to explore their capabilities and limitations. And many of those products offer features that are not free to use. So, to make vulnerability research more accessible, we created CumulusFire to showcase Cloudflare features that typically require a paid level of service. We created this site for two reasons: to provide a standardized playground where researchers can test their exploits, and to make it easier for our teams to reproduce them while triaging.

CumulusFire has already helped us address the constant trickle of reports in which researchers would configure their origin server in an obviously insecure way, beyond default or expected settings, and then report that Cloudflare’s WAF does not block an attack. By policy, we will now only consider WAF bypasses a vulnerability if it is reproducible on CumulusFire.

As we expand our public program we will add additional services to the testing playground. Since we love dogfooding our own products, the entire sandbox is built on Cloudflare Workers.

Just as we grew our private program, we will continue to evolve our public bug bounty program to provide the best experience for researchers. We aim to add more documentation, testing platforms and a way to interact with our security teams so that researchers can be confident that their submissions represent valid security issues.

Look forward to us sharing more of our learnings as we grow the program.

We cover the details of how this vulnerability works in a separate blog post: Inside the Log4j2 vulnerability (CVE-2021-44228), but in summary, this vulnerability allows an attacker to execute code on a remote server. Because of the widespread use of Java and Log4j, this is likely one of the most serious vulnerabilities on the Internet since both Heartbleed and ShellShock. The vulnerability is listed as CVE-2021-44228. The CVE description states that the vulnerability affects Log4j2 <=2.14.1 and is patched in 2.15. The vulnerability additionally impacts all versions of log4j 1.x; however, it is End of Life and has other security vulnerabilities that will not be fixed. Upgrading to 2.15 is the recommended action to take. You can also read about how we updated our WAF rules to help protect our customers in this post: CVE-2021-44228 - Log4j RCE 0-day mitigation

One of the first things we do whenever we respond to an incident is start drafting a timeline of events we need to review and understand within the context of the situation. Some examples from our timeline here include:

• 2021-12-09 16:57 UTC - Hackerone report received regarding Log4j RCE on developers.cloudflare.com

• 2021-12-10 09:56 UTC - First WAF rule shipped to Cloudflare Specials ruleset

• 2021-12-10 10:00 UTC - Formal engineering INCIDENT is opened and work begins to identify areas we need to patch Log4j

• 2021-12-10 10:33 UTC - Logstash deployed with patch to mitigate vulnerability.

• 2021-12-10 10:44 UTC - Second WAF rule is live as part of Cloudflare managed rules

• 2021-12-10 10:50 UTC - ElasticSearch restart begins with patch to mitigate vulnerability

• 2021-12-10 11:05 UTC - ElasticSearch restart concludes and is no longer vulnerable

• 2021-12-10 11:45 UTC - Bitbucket is patched and no longer vulnerable

• 2021-12-10 21:22 UTC - Hackerone report closed as Informative after it was unable to be reproduced

An important question when dealing with any software vulnerability, and what may actually be the hardest question that every company has to answer in this particular case is: where are all the places that the vulnerable software is actually running?

If the vulnerability is in a proprietary piece of software licensed by one company to the rest of the world, that is easy to answer - you just find that one piece of software. But in this case that was much harder. Log4j is a widely used piece of software but not one that people who are not Java developers are likely to be familiar with. Our first action was to refamiliarize ourselves with all places in our infrastructure where we were running software on the JVM, in order to determine which software components could be vulnerable to this issue.

We were able to create an inventory of all software we have running on the JVM using our centralized code repositories. We used this information to research and determine each individual Java application we had, whether it contained Log4j, and which version of Log4j was compiled into it.

We discovered that our ElasticSearch, LogStash, and Bitbucket contained instances of the vulnerable Log4j package that was between versions 2.0 and 2.14.1. We were able to use the mitigation strategies described in the official Log4j security documentation to patch the issue. For each instance of Log4j we either removed the JndiLookup class from the classpath:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Or we set the mitigating system property in the log4j configuration:

log4j2.formatMsgNoLookups

We were able to quickly mitigate this issue in these packages using these strategies while waiting for new versions of the packages to be released.

Even before we were done making the list of internal places where the vulnerable software was running, we started by looking at external reports - from HackerOne, our bug bounty program, and a public post in GitHub suggesting that we might be at risk.

We identified at least two reports that seemed to indicate that Cloudflare was vulnerable and compromised. In one of the reports was the following screenshot:

This example is targeting our developer documentation hosted at https://developer.cloudflare.com. On the right-hand side, the attacker demonstrates that a DNS query was received for the payload he sent to our server. However, the IP address flagged here is 173.194.95.134, a member of a Google owned IPv4 subnet (173.194.94.0/23).

Cloudflare’s developer documentation is hosted as a Cloudflare Worker and only serves static assets. The repository is public. The Worker relies on Google’s analytics library as seen here, therefore, we hypothesize that the attacker was not receiving a request from Cloudflare but through Google's servers.

Our backend servers receive logging from Workers, but exploitation was also not possible in this instance as we leverage robust Kubernetes egress policies that prevent calling out to the Internet. The only communication allowed is to a curated set of internal services.

When we received a similar report in our vulnerability disclosure program while gathering more information, the researcher was unable to reproduce the issue. This further enforced our hypothesis that it was third party servers, and they may have patched the issue.

While we were running versions of the software as described above, thanks to our speed of response and defense in depth approach, we do not believe Cloudflare was compromised. We have invested significant efforts into validating this, and we will continue working on this effort until everything is known about this vulnerability. Here is a bit about that part of our efforts.

As we were working to evaluate and isolate all the contexts in which the vulnerable software might be running and remediate them, we started a separate workstream to analyze whether any of those instances had been exploited.  Our detection and response methodology follows industry standard Incident Response practices and was thoroughly deployed to validate whether any of our assets were indeed compromised. We followed a multi-pronged approach described next.

Our asset inventory and code scanning tooling allowed us to identify all applications and services reliant on Apache Log4j. While these applications were being reviewed and upgraded if needed, we were performing a thorough scan of these services and hosts. Specifically, the exploit for CVE-2021-44228 relies on particular patterns in log messages and parameters, for example \$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+. For each potentially impacted service, we performed a log analysis to expose any attempts at exploitation.

Our network analytics allow us to identify suspicious network behavior that may be indicative of attempted or actual exploitation of our infrastructure. We scrutinised our network data to identify the following:

1. Suspicious Inbound and Outbound ActivityBy analyzing suspicious inbound and outbound connections, we were able to sweep our environment and identify whether any of our systems were displaying signs of active compromise.

2. Targeted Systems & ServicesBy leveraging pattern analytics against our network data, we uncovered systems and services targeted by threat-actors. This allowed us to perform correlative searches against our asset inventory, and drill down to each host to determine if any of those machines were exposed to the vulnerability or actively exploited.

3. Network IndicatorsFrom the aforementioned analysis, we gained insight into the infrastructure of various threat actors and identified network indicators being utilized in attempted exploitation of this vulnerability. Outbound activity to these indicators was blocked in Cloudflare Gateway.

We were able to correlate our log analytics and network analytics workflows to supplement our endpoint analysis. From our findings from both of those analyses, we were able to craft endpoint scanning criteria to identify any additional potentially impacted systems and analyze individual endpoints for signs of active compromise. We utilized the following techniques:

We are in the process of deploying custom Yara detection rules to alert on exploitation of the vulnerability. These rules will be deployed in the Endpoint Detection and Response agent running on all of our infrastructure and our centralized Security Information and Events Management (SIEM) tool.

Cloudflare continuously collects and analyzes endpoint process events from our infrastructure. We used these events to search for post-exploitation techniques like download of second stage exploits, anomalous child processes, etc.

Using all of these approaches, we have found no evidence of compromise.

In the analysis above, we focused on reviewing code and data we generate ourselves. But like most companies, we also rely on software that we have licensed from third parties. When we started our investigation into this matter, we also partnered with the company’s information technology team to pull together a list of each and every primary third-party provider and all sub-processors to inquire about whether they were affected. We’re in the process of receiving and reviewing responses from the providers. Any providers who we deem critical that are impacted by this vulnerability will be disabled and blocked until they are fully remediated.

As we responded to this incident, we found several places where our defense in depth approach worked.

1. Restricting outbound traffic

Restricting the ability to call home is an essential part of the kill-chain to make exploitation of vulnerabilities much harder. As noted above, we leverage Kubernetes network policies to restrict egress to the Internet on our deployments. In this context, it prevents the next-stage of the attack, and the network connection to attacker controlled resources is dropped.

All of our externally facing services are protected by Cloudflare. The origin servers for these services are set up via authenticated origin pulls. This means that none of the servers are exposed directly to the Internet.

2.   Using Cloudflare to secure Cloudflare

All of our internal services are protected by our Zero-trust product, Cloudflare Access. Therefore, once we had patched the limited attack surface we had identified, any exploit attempts to Cloudflare’s systems or customers leveraging Access would have required the attacker to authenticate.

And because we have the Cloudflare WAF product deployed as part of our effort to secure Cloudflare using Cloudflare, we benefited from all the work being done to protect our customers. All new WAF rules written to protect against this vulnerability were updated with a default action of BLOCK. Like every other customer who has the WAF deployed, we are now receiving protection without any action required on our side.

While our response to this challenging situation continues, we hope that this outline of our efforts helps others. We are grateful for all the support we have received from within and outside of Cloudflare.

Thank you to Evan Johnson, Anjum Ahuja, Sourov Zaman, David Haynes, and Jackie Keith who also contributed to this blog.