Cloudflare’s commitment to the CISA pledge reflects our dedication to transparency and accountability to our customers, and to cybersecurity best practices. Furthermore, Cloudflare is committed to being a trusted partner by sharing our strategies to ensure the highest priority is placed on safeguarding our customers’ security. 

Cloudflare has successfully managed a public Vulnerability Disclosure Program (VDP) for years; our belief is that collaboration is the cornerstone of effective cybersecurity. We are excited to announce a major milestone in our journey to meet Goal #5 of the pledge: our program will now include a bug bounty VIP program in conjunction with our bug bounty public program.

Continuous investment in maturing our bug bounty program is a vital tool for the success of any security organization. By encouraging broader participation in vulnerability testing, we open the door to more diverse perspectives and expertise, ultimately leading to stronger, more resilient security measures. Additionally, the new VIP program will allow us greater flexibility in engaging security researchers on upcoming betas for Cloudflare products, and will allow us to award higher bounty payouts.

Our commitment to this effort underscores our belief that a safer Internet is achievable through shared responsibility and proactive engagement. The security team at Cloudflare is looking forward to implementing a more proactive approach to securing our products with the launch of the new bug bounty VIP program!

The new bug bounty VIP program is an exclusive hub for select security researchers who either have the specialized technical expertise in the niche areas Cloudflare is building products in (such as Cloudflare Workers) or have demonstrated a deep understanding of our products and platform by actively participating in the public program with meaningful security findings. As a VIP member, security researchers will have access to beta testing environments for Cloudflare products. This includes early access to our newest features and unannounced products before they go live.

The VIP program’s scope will be carefully modeled from Cloudflare's product release roadmap. Security researchers will have the opportunity to influence Cloudflare’s product and security development before release. VIP program participants also have the option to participate in external/gray box penetration testing activities (Spot Checks) for higher bounties related to security findings for upcoming product releases or critical infrastructure and services. 

We believe that exceptional contributions deserve exceptional rewards. As a result, we’ve restructured our bounty offerings for the VIP program with higher payouts. Recognizing the specialized skills and expertise required, VIP researchers will be eligible for significantly higher rewards. We have also introduced bonus rewards for high-impact findings, particularly those that address critical vulnerabilities in our beta projects through the aforementioned Spot Checks. To further incentivize meaningful contributions, security researchers in our public program will receive milestone bonuses and be invited to our VIP program based on the number and quality of their submissions over time.

The VIP bug bounty program’s focus is not only finding and fixing bugs, but it’s also aimed at fostering a deeper, more impactful relationship with our security researchers. Moreover, these outcomes align well with the CISA Vulnerability Disclosure Policy (VDP) goal. By offering exclusive access to beta software and enhanced rewards, our goals are as follows:

1. Elevate security standards: VIP researchers focusing on the most critical assets allows for further hardening of the overall security posture of Cloudflare’s products and services. 

2. Accelerate product development: Early identification of vulnerabilities allows the remediation of potential issues before they reach production, yielding faster, more secure, and more stable releases.

3. Foster innovation: Involving researchers in the development process creates an additional feedback loop that encourages innovative approaches to security challenges. 

4. Encourage collaboration: The bug bounty team will encourage collaborative blog posts for select reports as a way to disseminate security learnings and build partnerships with researchers.

This is a great professional growth opportunity for anyone in the technical research space as it gives participants the ability to work on cutting-edge technology with complex challenges, and can provide future opportunities for career/skill development.

The launch of the VIP program marks a new chapter in Cloudflare’s security journey. We are excited about the opportunity to partner more closely with our top security researchers to build safer products for customers. Together, we can achieve new heights in security excellence:

1. Stronger security: Security researchers with expertise in niche topics can help enhance Cloudflare's defenses against emerging and novel threats.

2. Proactive risk management: The new VIP program provides Cloudflare an additional avenue to identify and mitigate risks early in the product release cycle, reducing the likelihood of future security incidents.

3. Reinforced trust: Our commitment to security is central to our customer relationships and the trust they place in Cloudflare; by continuously improving our security posture, we seek to preserve that trust.

If you are a software manufacturer, we encourage you to familiarize yourself with CISA’s ‘Secure by Design’ principles and create a plan to implement them in your company.

As an individual, we encourage you to participate in the Cloudflare bug bounty program and promote cybersecurity awareness in your community.

Stay tuned for more updates, and if you’re part of our public program, keep submitting those reports — you might just earn an invitation to join the VIP ranks! You can also find more updates on our blog, as we build our roadmap to meet all seven CISA Secure by Design pledge goals by May 2025!

Let’s help build a better Internet together.

Configuration management is far from a solved problem. As organizations scale beyond a handful of administrators, having a secure, auditable, and self-service way of updating system settings becomes invaluable. Managing a Cloudflare account is no different. With dozens of products and hundreds of API endpoints, keeping track of current configuration and making bulk updates across multiple zones can be a challenge. While the Cloudflare Dashboard is great for analytics and feature exploration, any changes that could potentially impact users really should get a code review before being applied!

This is where Cloudflare's Terraform provider can come in handy. Built as a layer on top of the cloudflare-go library, the provider allows users to interface with the Cloudflare API using stateful Terraform resource declarations. Not only do we actively support this provider for customers, we make extensive use of it internally! In this post, we hope to provide some best practices we've learned about managing complex Cloudflare configurations in Terraform.

Unsurprisingly, we find Cloudflare's products to be pretty useful for securing and enhancing the performance of services we deploy internally. We use DNS, WAF, Zero Trust, Email Security, Workers, and all manner of experimental new features throughout the company. This dog-fooding allows us to battle-harden the services we provide to users and feed our desired features back to the product teams all while running the backend of Cloudflare. But, as Cloudflare grew, so did the complexity and importance of our configuration.

When we were a much smaller company, we only had a handful of accounts with designated administrators making changes on behalf of their colleagues. However, over time this handful of accounts grew into hundreds with each managed by separate teams. Independent accounts are useful in that they allow service-owners to make modifications that can't impact others, but it comes with overhead.

We faced the challenge of ensuring consistent security policies, up-to-date account memberships, and change visibility. While our  accounts were still administered by kind human stewards, we had numerous instances of account members not being removed after they transferred to a different team. While this never became a security incident, it demonstrated the shortcomings of manually provisioning account memberships. In the case of a production service migration, the administrator executing the change would often hop on a video call and ask for others to triple-check an IP address, ruleset, or access policy update. It was an era of looking through the audit logs to see what broke a service.

We wanted to make it easier for developers and users to make the changes they wanted without having to reach out to an administrator. Defining our configuration in code using Terraform has allowed us to keep tabs on the complexity of configuration while improving visibility and change management practices. By dogfooding the Cloudflare Terraform provider, we've been able to ensure:

• Modifications to accounts are peer reviewed by the team that owns an account.

• Each change is tied to a user, commit, and a ticket explaining the rationale for the change.

• API Tokens are tied to service accounts rather than individual human users, meaning they survive team changes and offboarding.

• Account configuration can be audited by anyone at the company for current state, accuracy, and security without needing to add everyone as a member of every account.

• Large changes, such as enforcing hard keys can be done rapidly– even in a single pull request.

• Configuration can be easily copied and reused across accounts to promote best practices and speed up development.

• We can use and iterate on our awesome provider and provide a better experience to other users (shoutout in particular to Jacob!).

Terraform has a fairly mature open source ecosystem, built from years of running-in-production experience. Thus, there are a number of ways to make interacting with the system feel as comfortable to developers as git. One of these tools is Atlantis.

Atlantis acts as continuous integration/continuous delivery (CI/CD) for Terraform; fitting neatly into version control workflows, and giving visibility into the changes being deployed in each code change. We use Atlantis to display Terraform plans (effectively a diff in configuration) within pull requests and apply the changes after the pull request has been approved. Having all the output from the terraform provider in the comments of a pull request means there's no need to fiddle with the state locally or worry about where a state lock is coming from. Using Terraform CI/CD like this makes configuration management approachable to developers and non-technical folks alike.

In this example pull request, I'm adding a user to the cloudflare-cool-account (see the code in the next section). Once the PR is opened, Bitbucket posts a webhook to Atlantis, telling it to run a `terraform plan` using this branch. The resulting comment is placed in the pull request. Notice that this pull request can't be applied or merged yet as it doesn't have an approval! Once the pull request is approved, I would comment "atlantis apply", wait for Atlantis to post a comment containing the output of the command, and merge the pull request if that output looks correct.

Our Terraforming Cloudflare architecture consists of a monorepo with one directory (and tfstate) for each internally-owned Cloudflare account. This keeps all of our Cloudflare configuration centralized for easier oversight while remaining neatly organized.

It will be possible in a future (as of this writing) release to manage multiple Cloudflare accounts in the same tfstate, but we've found that accounts in our use generally map fairly neatly onto teams. Teams can be configured as CODEOWNERS for a given directory and be tagged on any pull requests to that account. With teams owning separate accounts and each account having a separate tfstate, it's rare for pull requests to get stuck waiting for a lock on the tfstate. Team-account-sized states remain relatively small, meaning that they also build quickly. Later on, we'll share some of the other optimizations we've made to keep the repo user-friendly.

Each of our terraform states, given that they include secrets (including the API key!), is stored encrypted in an internal datastore. When a pull request is opened, Atlantis reaches out to a piece of middleware (that we may open source once it's cleaned up a bit) that retrieves and decrypts the state for processing. Once the pull request is applied, the state is encrypted and put away again.

We execute a daily Terraform apply across all tfstates to capture any unintended config drift and rotate certificates when they approach expiration. This prevents unrelated changes from popping up in pull request diffs and causing confusion. While we could run more frequent state applies to ensure Terraform remains firmly up to date, once-a-day rectification strikes a balance between code enforcement and avoiding state locks while users are running Terraform plans in pull requests.

One of the problems that we encountered during our transition to Terraform is that folks were in the habit of making updates to configuration in the Dashboard and were still able to edit settings there. Thus, we didn't always have a single source of truth for our configuration in code. It also meant the change would get mysteriously (to them) reverted the next day! So that's why I'm excited to share a new Zero Trust Dashboard toggle that we've been turning on for our accounts internally: API/Terraform read-only mode.

Easily one of my favorite new features

With this button, we're able to politely prevent manual changes to your Cloudflare account’s Zero Trust configuration without removing permissions from the set of users who can fix settings manually in a break-glass emergency scenario. Check out how you can enable this setting in your Zero Trust organization.

As our Terraform repository has matured, we've refined how we define Cloudflare resources in code. By finding a sweet spot between code reuse and readability, we've been able to minimize operational overhead and generally let users get their work done. Here's a couple of useful snippets that have been particularly valuable to us.

This allows for defining a fairly straightforward mapping of user emails to account privileges without code duplication or complex modules. We pull the list of human-friendly names of account roles from the API to show user permission assignments at a glance. Note: status is a new argument that allows for accounts to be added without sending an email to the user; perfect for when an organization is using SSO. (Thanks patrobinson for the feature request and mblackman for the PR!)

variables.tf
—-
data "cloudflare_account_roles" "my_account" {
	account_id = var.account_id
}

locals {
  roles = {
	for role in data.cloudflare_account_roles.my_account.roles :
  	role.name => role
  }
}

members.tf
—-
locals {
  users = {
    emerson = {
      roles = [
        local.roles["Administrator"].id
      ]
    }
    lucian = {
      roles = [
        local.roles["Super Administrator - All Privileges"].id
      ]
    }
    walruto = {
      roles = [
        local.roles_by_name["Audit Logs Viewer"].id,
        local.roles_by_name["Cloudflare Access"].id,
        local.roles_by_name["DNS"].id
      ]
  }
}

resource "cloudflare_account_member" "account_member" {
  for_each  	= local.users
  account_id	= var.account_id
  email_address = "${each.key}@cloudflare.com"
  role_ids  	= each.value.roles
  status            = "accepted"
}

The GitHub issue and provider change that enabled automatic Access service token refreshes actually came from a need inside Cloudflare. Here's how we ended up implementing it. We begin by defining a set of services that need to connect to our hostnames that are protected by Access. Each of these tokens are created and stored in a secret key value store. Next, we reference those access tokens by ID in the target Access policies. Once this has run, the service owner or the service itself can retrieve the credentials from the data store. (Note: we're using Vault here, but any storage provider could be used in its place).

tokens.tf
—
locals {
  service_tokens = toset([
    "customer-service",     # TICKET-120
    "full-service",               # TICKET-128
    "quality-of-service"      # TICKET-420 
    "room-service"            # TICKET-927
  ])
}

resource "cloudflare_access_service_token" "token" {
  for_each   = local.service_tokens
  account_id = var.account_id
  name   	= each.key
  min_days_for_renewal = 30
}

resource "vault_generic_secret" "access_service_token" {
  for_each   = local.service_tokens
  path = "kv/secrets/${each.key}/access_service_token"
  disable_read = true

  data_json = jsonencode({
	client_id        = cloudflare_access_service_token.token["${each.key}"].client_id,
client_secret = cloudflare_access_service_token.token["${each.key}"].client_secret
  })
}

super_cool_hostname.tf
—
resource "cloudflare_access_application" "super_cool_hostname" {
  account_id             	            = var.account_id
  name                   	            = "Super Cool Hostname"
  domain                 	            = "supercool.hostname.tld"
}

resource "cloudflare_access_policy" "super_cool_hostname_service_access" {
  application_id = cloudflare_access_application.super_cool_hostname.id
  zone_id    	= data.cloudflare_zone.hostname_tld.id
  name       	= "TICKET-927 Allow Room Service "
  decision   	= "non_identity"
  precedence 	= 1
  include {
	service_token = [cloudflare_access_service_token.token["room-service"].id]
  }
}

To further defense-in-depth objectives, we've been rolling out mTLS throughout our internal systems. One of the places where we can take advantage of our Terraform provider is in defining AOP (Authenticated Origin Pulls) certificates to lock down the Cloudflare-edge-to-origin connection. Anyone who has managed certificates of any kind can speak to the headaches they can cause. Having certificate configurations in Terraform takes out the manual work of rotation and expiration.

In this example we're defining hostname-level AOP as opposed to zone-level AOP. We start by cutting a certificate for each hostname. Once again we're using Vault for certificate creation, but other backends could be used just as well. This certificate is created with a (not-shown) 30 day expiration, but set to renew automatically. This means once the time-to-expiration is equal to min_seconds_remaining, the resource will be automatically tainted and replaced on the next Terraform run. We like to give this automation plenty of room before expiration to take into account holiday seasons and avoid sending alerts to humans when the alerts hit seven days to expiration. For the rest of this snippet, the certificate is uploaded to Cloudflare and the ID from that upload is then placed in the AOP configuration for the given hostname. The create_before_destroy meta-argument ensures that the replacement certificate is uploaded successfully before we remove the certificate that's currently in place.

locals {
  hostnames = toset([
	"supercool.hostname.tld",
            "thatsafinelooking.hostname.tld"
  ])
}

resource "vault_pki_secret_backend_cert" "vault_cert" {
  for_each          	      = local.hostnames
  backend           	      = "pki-aop"
  name              	      = "default"
  auto_renew         	      = true
  common_name       	      = "${each.key}.aop.pki.vault.cfdata.org"
  min_seconds_remaining = 864000 // renew when there are 10 days left before expiration
}

resource "cloudflare_authenticated_origin_pulls_certificate" "aop_cert" {
  for_each  = local.hostnames
  zone_id   = data.cloudflare_zone.hostname_tld.id
  type 	      = "per-hostname"

  certificate = vault_pki_secret_backend_cert.vault_cert["${each.key}"].certificate
  private_key = vault_pki_secret_backend_cert.vault_cert["${each.key}"].private_key

  lifecycle {
	create_before_destroy = true
  }
}

resource "cloudflare_authenticated_origin_pulls" "aop_config" {
  for_each                           	= local.hostnames
  zone_id    	                        = data.cloudflare_zone.hostname_tld.id
  authenticated_origin_pulls_certificate = cloudflare_authenticated_origin_pulls_certificate.aop_cert["${each.key}"].id
  hostname                           	= "${each.key}"
  enabled                            	= true
}

The comfortable automation that we've achieved thus far did not come without some hair-pulling. Below are a few of the learnings that have allowed us to maintain the repository as a side project run by two engineers (shoutout David).

It feels worth repeating that the tfstate contains secrets including any API keys you're using with providers and the default location of the tfstate is in the current working directory. It's very easy to accidentally commit this to source control. By defining a backend, the state can be stored with a cloud storage provider, in a secure location on a filesystem, in a database, or even Cloudflare Workers! Wherever the state is stored, make sure it is encrypted.

Modules are intended to reduce code repetition for well-defined chunks of systems such as "I want three clusters of whizz-bangs in locations A, C, and F." If cloud-computing was like Factorio, this would be amazing. However, financial, technical, and physical constraints mean subtle differences in systems develop over time such as "I want fewer whizz-bangs in C and the whizz-bangs in F should get a different network topology." In Terraform, implementation logic of these requirements is moved to the module code. HCL is absolutely not the place to write decipherable conditionals. While module versioning prevents having to make every change backwards-compatible, keeping module usage up-to-date becomes another chore for repository maintainers.

An understandable code base is a user-friendly codebase. It's rare that a deeply cryptic error will return from a misconfigured resource definition. Conversely, modules, especially custom ones, can lead users on a head-scratching adventure. This kind of system can't scale with confused users.

A few well-designed for_each loops (we're obviously fans) can achieve similar objectives as modules without the complexity. It's fine to use plain old resources too! Especially when there are more than a handful of varying arguments, it's more valuable for the configuration to be clear than to be eloquent. For example: an account_member resource makes sense to be in a for_loop, but a page_rule probably doesn't.

Maintaining quick pull-request-to-plan turnaround keeps Terraform from feeling like a burden on users' time. Furthermore, if a plan is taking 30 minutes to run, a rollback in the case of an issue would also take 30 minutes! This post describes our single-account-to-tfstate model.

However, after noticing slow-downs coming from the large number of AOP certificate configurations in a big zone, we moved that code to a separate tfstate. We were able to make this change because AOP configuration is fairly self-contained. To ensure there would be no fighting between the states, we kept the API token permissions for each tfstate mutually exclusive of each other. Our Atlantis Terraform plans typically finish under five minutes. If it feels impossible to keep the size of a tfstate down to a reasonable amount of time, it may be worth considering a different tool for that bit of configuration management.

Terraform isn't a panacea. We generally don't use Terraform to manage DNS records, for example. We use OctoDNS which integrates more neatly into our infrastructure automation. DNS records can quickly add up to long state-rendering times and are often dynamically generated from systems that Terraform doesn't know about. To avoid conflicts, there should only ever be one system publishing changes to DNS records.

We also haven't figured out a maintainable way of managing Workers scripts in Terraform. When a .js script in the Terraform directory changes, Terraform isn't aware of it. This means a change needs to occur somewhere else in a .tf file before the plan diff is generated. It likely isn't an unsolvable issue, but doesn't seem particularly worth cramming into Terraform when there are better options for Worker management like Wrangler.

We're continuing to invest in the Cloudflare Terraforming experience both for our own use and for the benefit of our users. With the provider, we hope to offer a comfortable and scalable method of interacting with Cloudflare products. Hopefully this post has presented some useful suggestions to anyone interested in adopting Cloudflare-configuration-as-code. Don't hesitate to reach out on the GitHub project for troubleshooting, bug reports, or feature requests. For more in depth documentation on using Terraform to manage your Cloudflare account, read on here. And if you don't have a Cloudflare account already, click here to get started.

Cloudflare customers do not have to take any action to protect themselves from these newly discovered vulnerabilities, and no Cloudflare customer data was ever at risk.

We have not seen any attempted exploitation of these vulnerabilities described in the advisory. We use OctoRPKI to perform Border Gateway Protocol (BGP) route validation so that our routers know where to direct IP packets at Layer 3 of the TCP/IP stack. TLS provides additional security at the TCP layer to ensure the integrity and confidentiality of customer data going over the Internet in the event of BGP hijacking.

Resource Public Key Infrastructure (RPKI) is a cryptographic method of signing records that associate a BGP route announcement with the correct originating Autonomous System (AS) number. In order to validate the records that contain that information we use an open source software called OctoRPKI that is part of the cfrpki toolkit.

OctoRPKI and GoRTR ecosystem diagram

OctoRPKI traverses a set of trusted certificate repositories, downloads all the records and manifests that they contain, and performs a set of validation checks on them. If they are valid, OctoRPKI will add their contents into a JSON file that is made available for GoRTR instances to consume.

RFC6481 further defines the role of certificate repositories:

  To validate attestations made in the context of the Resource Public
   Key Infrastructure (RPKI) [RFC6480], relying parties (RPs) need
   access to all the X.509/PKIX Resource Certificates, Certificate
   Revocation Lists (CRLs), and signed objects that collectively define
   the RPKI.

   Each issuer of a certificate, CRL, or a signed object makes it
   available for download to RPs through the publication of the object
   in an RPKI repository.

   The repository system is a collection of all signed objects that MUST
   be globally accessible to all RPs.  When certificates, CRLs and
   signed objects are created, they are uploaded to a repository
   publication point, from whence they can be downloaded for use by RPs.

The main list of trusted repositories that OctoRPKI uses can be found here. In general, OctoRPKI will attempt to process any file that it downloads from a repository. However, this leaves validation software open to processing malicious input. For example, OctoRPKI could be instructed to download and cache a file which contains a path that performs directory traversal, or it could be provided with a classic GZIP bomb attack leading to a crash. The RFC does not necessarily define limits on content within files returned by a repository and thus, a large number of undefined behaviors can occur.

Compounding this issue is the fact that any single repository in the chain of trust could introduce undefined behavior. Imagine a scenario where a malicious entity is able to compromise a single repository (there can be hundreds) within a trusted organization, or is able to introduce a malicious Trust Anchor Locator (TAL) file onto the host machine that is running OctoRPKI. In both cases, bad actors can attempt to trigger undefined behavior on machines running OctoRPKI by leveraging the fact that OctoRPKI will attempt to process arbitrary input. Our mitigations were primarily to fail closed whenever these events occurred as there is no other guidance in the RFC.

There were two classes of attacks disclosed in the NSCS advisory that affected OctoRPKI:

• CVE-2021-3907 - Arbitrary filepath traversal via URI injection

Impact

OctoRPKI does not escape a URI with a filename containing "..", which allows a malicious repository to create a file, for example rsync://example.org/repo/../../etc/cron.daily/evil.roa, which would then be written to disk outside the base cache folder. This could allow for remote code execution on the host machine OctoRPKI is running on.

Mitigation

In v1.4.0 we now filter URIs and force them to remain in the cache folder by overriding any upwards directory traversal.

• CVE-2021-3908 - Infinite certificate chain depth results in OctoRPKI running forever

• CVE-2021-3909 - Infinite open connection causes OctoRPKI to hang forever

• CVE-2021-3910 - NUL character in ROA causes OctoRPKI to crash

• CVE-2021-3911 - Misconfigured IP address field in ROA leads to OctoRPKI crash

• CVE-2021-3912 - OctoRPKI crashes when processing GZIP bomb returned via malicious repository

Impact

All of these trigger either a crash or infinite runtime by abusing the fact that OctoRPKI will process any file it ingests. For a production critical service it is imperative that undefined behavior is identified early, and either tossed away or caught and presented to the user as an error. Consistent crashes of OctoRPKI can lead to denial of service type attacks.

Mitigation

We implemented bounds checking across many components within OctoRPKI. These include adding instances of checking array length before attempting to index specific locations, or other cases where we utilized built in controls that Go provides when using an HTTP client. Repositories that attempt to abuse bounds checks are either skipped or included in an error message presented to the user.

We are ecstatic to see quality security research, like the vulnerabilities discovered by researchers from the University of Twente, being performed in the RPKI space. It is an incredible sign of progress in the deployment of RPKI, especially considering how recent widespread adoption has been. We are committed to ongoing support of RPKI and look forward to continuing to work with the security community to make the Internet safer and more secure for everyone.

When the security team at Cloudflare takes on new projects, we approach them with the goal of achieving the “builder first mindset” whereby we design, develop, and deploy solutions just as any standard engineering team would. Additionally, we aim to dogfood our products wherever possible. Cloudflare as a security platform offers a lot of functionality that is vitally important to us, including, but not limited to, our WAF, Workers platform, and Cloudflare Access. We get a lot of value out of using Cloudflare to secure Cloudflare. Not only does this allow us to test the security of our products; it provides us an avenue of direct feedback to help improve the roadmaps for engineering projects.

One specific product that we get a lot of use out of is our serverless platform, Cloudflare Workers. With it, we can have incredible flexibility in the types of applications that we are able to build and deploy to our edge. An added bonus here is that our team does not have to manage a single server that our code runs on.

Today, we’re launching support for the security.txt initiative through Workers to help give security researchers a common location to learn about how to communicate with our team. In this post, I’m going to focus on some of the benefits of managing security projects through Workers, detail how we built out the mechanism that we’re using to manage security.txt on cloudflare.com, and highlight some other use cases that our team has built on Workers in the past few months.

The Workers platform is designed to let anyone deploy code to our edge network which currently spans across 200 cities in 90 countries. As a result of this, applications deployed on the platform perform exceptionally well, while maintaining high reliability. You can implement nearly any logic imaginable on top of zones in your Cloudflare account with almost no effort.

One of the biggest immediate benefits of this is the idea that there are no servers for our team to maintain. Far too often I have seen situations where a long running application is deployed onto an instanced machine within a cloud environment and immediately forgotten about. In some cases, outdated software or poorly provisioned permissions can open up dangerous vectors for malicious actors if a compromise is occurring. Our team can deploy new projects with the confidence that they will remain secure. The Workers runtime environment has many wins such as the fact that it receives automatic security updates and that we don't need to maintain the software stack, just the application logic.

Another benefit is that since Workers executes JavaScript, we can dream up complex applications and rapidly deploy them as full applications in production without a huge investment in engineering time. This encourages rapid experimentation and iteration while achieving high impact with consistent performance. A couple of examples:

• Secure code review - On every pull request made to a code repository, a Worker runs a set of security curated rules and posts comments if there are matches.

• CSP nonces and HTML rewriting - A Worker generates random nonces and uses the HTMLRewriter API to mutate responses with dynamic content security policies on our dashboard.

• Authentication on legacy applications -  Using the Web Crypto API, a Worker sits in front of legacy origins and issues, validates, and signs JWTs to control access.

Stay tuned for future blog posts where we plan to dive deeper on these initiatives and more!

Our team regularly engages with external security researchers through our HackerOne program and since about 2014, the details of our program can be found on a dedicated page on our marketing site. This has worked quite well in starting up our program and allows anyone to contact our team if they have a vulnerability to report. However, every so often we’d see that there were issues with this approach. Specifically, there were cases of vulnerabilities being submitted to our support staff who would then tell the reporter about the disclosure page which then directs them to HackerOne resulting in an overall error prone experience for everyone involved. Other times, the team that manages our social media would direct researchers to us.

Security researchers, having done the hard work to find a vulnerability, face challenges to responsibly disclose it. They need to be able to quickly locate contact information, consume it and disclose without delay. The security.txt initiative addresses this problem by defining a common location and format for organizations to provide this information to researchers. A specification for this was submitted to the IETF with the description:

“When security vulnerabilities are discovered by researchers, proper reporting channels are often lacking.  As a result, vulnerabilities may be left unreported.  This document defines a format ("security.txt") to help organizations describe their vulnerability disclosure practices to make it easier for researchers to report vulnerabilities.”

Employing the guidance in security.txt doesn't solve the problem entirely but it is becoming common best practice amongst many other security conscious companies. Over time I expect that researchers will rely on security.txt for information gathering. In fact, some integrations are already being built by the community into security tools to automatically fetch information from a company’s security.txt!

Our security.txt can be found here: https://cloudflare.com/.well-known/security.txt

Cloudflare's security.txt

When scoping out the work for deploying security.txt we quickly realized that there were a few areas we wanted to address when deploying the service:

• Automation - Deploys should involve as few humans as possible.

• Ease of maintenance - Necessary changes (specification updates, key rotations, etc) should be a single commit and deploy away.

• Version control - security.txt is inherently a sensitive file, retaining attribution of every change made is valuable from an auditing perspective.

Certainly, we could manage this project through a manual process involving extensive documentation and coordination for manual deployments but we felt it was important to build this out as a full service that was easily maintainable. Manual maintenance and deployments do not necessarily scale well over time, especially for something that isn’t touched that often. We wanted to make that process as easy as possible since security.txt is meant to be a regularly maintained, living document.

We quickly turned to Workers for the task. Using the Wrangler CLI we can achieve the ease of deployment and automation requirements as well as track the project in git. In under half an hour I was able to build out a full prototype that addressed all of the requirements of the Internet-Draft and deploy it to a staging instance of our website. Since then, members from our team made some revisions to the text itself, updated the expiration on our PGP key, and bumped up to the latest draft version.

One cool decision we made is that the security.txt file is created from a template at build time which allows us to perform dynamic operations on our baseline security.txt. For example, Section 3.5.5 of the draft calls for a date and time after which the data contained in the "security.txt" file is considered stale and should not be used. To address this field, we wrote a short node.js script that automatically sets an expiration of 365 days after the point of deployment, encouraging regular updates:

const dayjs = require('dayjs')
const fs = require('fs')

const main = async () => {
  fs.appendFile(
    './src/txt/security.txt.temp',
    `\nExpires: ${dayjs()
      .add(365, 'day')
      // Thu, 31 Dec 2020 18:37:07 -0800
      .format('ddd, D MMM YYYY HH:mm:ss ZZ')}\n`,
    function(err) {
      if (err) throw err
      console.log('Wrote expiration field!')
    },
  )
}

We’ve also leveraged this benefit in our Make targets where at build time we ensure that the deployed security.txt is clearsigned with the security@cloudflare.com PGP key:

sign:
	gpg --local-user security@cloudflare.com -o src/txt/security.txt --clearsign src/txt/security.txt.temp
	rm src/txt/security.txt.temp

And finally, leveraging the multi-route support in wrangler 1.8 alongside the request object that gets passed into our Worker, we can serve our PGP public key and security.txt at the same time on two routes using one Worker, differentiated based on what the eyeball is asking for:

import pubKey from './txt/security-cloudflare-public-06A67236.txt'
import securityTxt from './txt/security.txt'

const handleRequest = async request => {
  const { url } = request
  if (url.includes('/.well-known/security.txt')) {
    return new Response(securityTxt, {
      headers: { 'content-type': 'text/plain; charset=utf-8' }, // security.txt
    })
  } else if (url.includes('/gpg/security-cloudflare-public-06A67236.txt')) {
    return new Response(pubKey, {
      headers: { 'content-type': 'text/plain; charset=utf-8' }, // GPG Public key
    })
  }
  return fetch(request) // Pass to origin
}

In the interest of transparency and to allow anyone to easily achieve the same wins we did, we’ve open sourced the Worker itself for anyone who wants to deploy this service onto their Cloudflare zone. It just takes a few minutes of time.

You can find the project on GitHub: https://github.com/cloudflare/securitytxt-worker

The security team at Cloudflare has grown a significant amount in the past year. This is most evident in terms of our headcount (and still growing!) but also in how we take on new projects. We’re hoping to share more stories like this in the future and open source some of the other security services that we are building on Workers to help others achieve the same security wins that we are achieving with the platform.