While Cloudflare One has been often cited as one of the easiest-to-use SASE platforms, there is no magic without proper configuration. And while Cloudflare has been striving to simplify complex networking concepts by creating products such as Cloudflare WAN, Magic Transit, and Cloudflare Network Firewall, which simplify and reduce the typical complexity associated with deploying comparable functions from other vendors, the breadth of capabilities provided by Cloudflare One require creation of best-practice policies and templates to achieve the most optimal outcomes.

To make it easy to start taking advantage of Cloudflare’s powerful SASE platform, we have developed a method that ensures customers get the right configuration quickly and easily. We call it Project Helix. 

In this post, we’ll dig into the problem of getting the correct customization, and how we built Project Helix to make it simple. That means our customers have access to the most powerful SASE platform out there — and the easiest to onboard.

Cloudflare One is the world’s largest composable platform, and we enable our product teams to release different capabilities when they are ready. That means customers get access to cutting-edge features as soon as possible, but sometimes these features require tweaking settings or attributes that are set in the platform by default. 

For example, Cloudflare One provides comprehensive DNS protection, Network Protection, Secure Web Gateway, and Zero Trust Access to any private application included in all of our comprehensive Interna packages. But deploying advanced security capabilities such as Secure Web Gateway, TLS inspection, DLP, AV scanning, etc. may be too disruptive right out of the gate — so a Cloudflare One tenant is typically provisioned with a blank slate. That means that there are many switches one must flip to enable the full power of Cloudflare One.

So we faced a dilemma: How can we help our customers get the right settings, right away?

We started by releasing guides to help administrators get started quickly, wherein they could select a scenario that matches their goals and outcomes.

But we soon realized that that approach did not accomplish the frictionless nirvana we were after. For example, customers who wanted to take advantage of all four scenarios described in the “Get Started” guide would need to step through each of those wizards individually. 

In another instance, we released a highly-anticipated capability to connect and secure any private app by hostname. But it was tricky to enable: in addition to flipping a switch in the Cloudflare One settings page, it required customers to change their default split tunnel configuration to include a specific CGNAT range designated for this functionality to be sent to Cloudflare via Cloudflare One Client. We couldn’t easily make this change a default Cloudflare One Client profile, as any change affecting traffic routing on a customer’s network could potentially break existing environments. 

For greenfield deployments, we want to be easily able to enable any customer to benefit from this capability without introducing a bunch of friction.

We needed a way to engage the knowledge we have, and use it to navigate the numerous knobs, switches, and policies on behalf of our customers — so they can take advantage of the full breadth of innovation.

To achieve this goal, we needed to find a reliable way of taking the amazing brainpower of our Solutions Engineers, Professional Service Engineers, and Partners and enable them to share the best practices they encountered deploying Cloudflare One, whether for production, demos, or proof-of-concepts. 

Sharing this knowledge had to be as easy as a push of a button and in a codified format — otherwise we knew it wouldn’t be done consistently. We decided to call it Project Helix, for the way in which it weaves together expertise and automation.

We kicked off the knowledge gathering by asking ourselves what we want customers to experience during the proof of concepts, and we documented all those outcomes. These included enabling baseline security best practice protections across DNS, Network, and HTTP protocols, enabling TLS inspection, QUIC/HTTP3 security for customers (a Cloudflare-exclusive capability for over 3 years now!), deploying Remote Browser Isolation for risky domain categories (such as newly-registered domains), deploying visibility and controls over AI applications the users can access, and elevating the visibility and configuration of the Tenant Control policies that allow customers to restrict their users to accessing only their own instance of SaaS applications such as Office 365, Google Workspace, Dropbox, Box, etc. 

We also noted that a frequent point of friction for our customers was splitting out traffic for popular real-time communication apps such as Zoom to go directly to the Internet. And for customers whose users are often traveling, the team assembled a list of widely used captive portals across airlines, hotels, etc., to help ensure a smoother experience for users accessing resources on those private networks in conjunction with the Cloudflare One client.

The old way — manual deployment — has significant drawbacks. Deploying all those policies and configurations manually on a brand-new tenant would take several hours. It would also require copious documentation that would need to be manually maintained and updated. And manual configuration and execution of all these steps is subject to human error, raising questions of consistency.

When we learned that our in-house Cloudflare teams had embraced Terraform to manage the ever-growing number of accounts used to support Cloudflare internal users, we decided to use a similar approach to solve our own dilemma.

We architected scalable and flexible Terraform templates that were programmed to deliver all these settings, configuration snippets, and policies. Once we saw how amazing that outcome was, we wanted to make this easier and more user-friendly for the broader user base.

So the team created a web-based user interface, hosted in Cloudflare Workers and leveraging Cloudflare Containers, to take input parameters and execute Terraform templates in an ephemeral fashion. As there’s no persistent storage used for this solution, it eliminates any potential security risk of storing logs or tokens used in the Terraform provisioning process. This allows anyone, from the most seasoned Solution Engineer to someone who is brand new to Cloudflare One, to deploy the full-functioning baseline configuration with a push a button. Within a couple of minutes of entering some basic information, the Cloudflare One tenant is fully configured and enabled with advanced security features and most optimal settings. Helix also surfaces a comprehensive list of security policies that we recommend the customer enable –- with a flip of the switch.

We start by deploying a set of robust DNS-based security settings, surfacing policies that allow corporate DNS for zero trust, while blocking security risks and questionable categories from ever being resolved by the DNS.  So when you log in to Cloudflare Dash interface, you will see the following DNS policies preconfigured:

We then layer it with robust network policies that protect users and stop malicious traffic across all ports and protocols that you can observe by going to the Network Policies tab in the Dash UI

And finally, we finish this with a broad set of robust HTTP security policies, featuring granular enterprise application tenant controls, securing of AI prompts, and isolating risky domains via Browser Isolation.

All of this is achieved in a matter of minutes, with 100% consistency and immunity to human data-entry errors. All you have to do is to turn these policies on or off to suit your particular needs.

To top it off, the deployment is optimized for maximum interoperability with leading captive portals across airlines and hotels, while also providing an option to easily break out traffic to Zoom to avoid performance issues of tunnelling. 

But wait — there was one more thing! Cloudflare internationalized its UI back in 2020, and we wanted to bring the same language-friendliness to all customers and partners across the globe. So we templatized all the object names, policy names, user interactions, etc., within Terraform, and delivered the ability to internationalize deployment of these core best practices and policies in any language.

The impact of this initiative has been massive. According to Bob Percciacante, a very seasoned Cloudflare One Solutions Engineer, using Helix for one of his proof-of-concepts saved 2–3 weeks of start-up and prep time to configure and verify all the necessary settings and features. He was able to demonstrate all the essential Cloudflare One features to the customer within 15 minutes of deploying a Helix-based configuration.

For the customer, it means they can start enjoying the security of Zero Trust from day one. 

Ready to go beyond the blank slate and accelerate your own Zero Trust deployment?

• Explore Cloudflare One: Learn more about the Cloudflare One platform and its comprehensive SASE capabilities on our Cloudflare One page.

• Contact your Cloudflare account team to experience the best of Cloudflare One deployment at lightning speed!

Since Cloudflare’s launch in 2010, we have collaborated with the National Center for Missing and Exploited Children (NCMEC), a US-based clearinghouse for reporting child sexual abuse material (CSAM), and are committed to doing what we can to support identification and removal of CSAM content.

Members of the public, customers, and trusted organizations can submit reports of abuse observed on Cloudflare’s network. A minority of these reports relate to CSAM, which are triaged with the highest priority by Cloudflare’s Trust & Safety team. We will also forward details of the report, along with relevant files (where applicable) and supplemental information to NCMEC.

The process to generate and submit reports to NCMEC involves multiple steps, dependencies, and error handling, which quickly became complex under our original queue-based architecture. In this blog post, we discuss how Cloudflare Workflows helped streamline this process and simplify the code behind it.

When we designed our latest NCMEC reporting system in early 2024, Cloudflare Workflows did not exist yet. We used the Workers platform Queues as a solution for managing asynchronous tasks, and structured our system around them.

Our goal was to ensure reliability, fault tolerance, and automatic retries. However, without an orchestrator, we had to manually handle state, retries, and inter-queue messaging. While Queues worked, we needed something more explicit to help debug and observe the more complex asynchronous workflows we were building on top of the messaging system that Queues gave us.

In our queue-based architecture each report would go through multiple steps:

1. Validate input: Ensure the report has all necessary details.

2. Initiate report: Call the NCMEC API to create a report.

3. Fetch impounded files (if applicable): Retrieve files stored in R2.

4. Upload files: Send files to NCMEC via API.

5. Finalize report: Mark the report as completed.

^{A diagram of our queue-based architecture} 

Each of these steps was handled by a separate queue, and if an error occurred, the system would retry the message several times before marking the report as failed. But errors weren’t always straightforward — for instance, if an external API call consistently failed due to bad input or returned an unexpected response shape, retries wouldn’t help. In those cases, the report could get stuck in an intermediate state, and we’d often have to manually dig through logs across different queues to figure out what went wrong.

Even more frustrating, when handling failed reports, we relied on a "Reaper" — a cron job that ran every hour to resubmit failed reports. Since a report could fail at any step, the Reaper had to deduce which queue failed and send a message to begin reprocessing. This meant:

• Debugging was a nightmare: Tracing the journey of a single report meant jumping between logs for multiple queues.

• Retries were unreliable: Some queues had retry logic, while others relied on the Reaper, leading to inconsistencies.

• State management was painful: We had no clear way to track whether a report was halfway through the pipeline or completely lost, except by looking through the logs.

• Operational overhead was high: Developers frequently had to manually inspect failed reports and resubmit them.

Queues gave us a solid foundation for moving messages around, but it wasn’t meant to handle orchestration. What we’d really done was build a bunch of loosely connected steps on top of a message bus and hoped it would all hold together. It worked, for the most part, but it was clunky, hard to reason about, and easy to break. Just understanding how a single report moved through the system meant tracing messages across multiple queues and digging through logs.

We knew we needed something better: a way to define workflows explicitly, with clear visibility into where things were and what had failed. But back then, we didn’t have a good way to do that without bringing in heavyweight tools or writing a bunch of glue code ourselves. When Cloudflare Workflows came along, it felt like the missing piece, finally giving us a simple, reliable way to orchestrate everything without duct tape.

Once Cloudflare Workflows was announced, we saw an immediate opportunity to replace our queue-based architecture with a more structured, observable, and retryable system. Instead of relying on a web of multiple queues passing messages to each other, we now have a single workflow that orchestrates the entire process from start to finish. Critically, if any step failed, the Workflow could pick back up from where it left off, without having to repeat earlier processing steps, re-parsing files, or duplicating uploads.

With Cloudflare Workflows, each report follows a clear sequence of steps:

1. Creating the report: The system validates the incoming report and initiates it with NCMEC.

2. Checking for impounded files: If there are impounded files associated with the report, the workflow proceeds to file collection.

3. Gathering files: The system retrieves impounded files stored in R2 and prepares them for upload.

4. Uploading files to NCMEC: Each file is uploaded to NCMEC using their API, ensuring all relevant evidence is submitted.

5. Adding file metadata: Metadata about the uploaded files (hashes, timestamps, etc.) is attached to the report.

6. Finalizing the report: Once all files are processed, the report is finalized and marked as complete.

Here’s a simplified version of the orchestrator:

import { WorkflowEntrypoint, WorkflowEvent, WorkflowStep } from 'cloudflare:workers';


export class ReportWorkflow extends WorkflowEntrypoint<Env, ReportType> {
  async run(event: WorkflowEvent<ReportType>, step: WorkflowStep) {
    const reportToCreate: ReportType = event.payload;
    let reportId: number | undefined;


    try {
      await step.do('Create Report', async () => {
        const createdReport = await createReportStep(reportToCreate, this.env);
        reportId = createdReport?.id;
      });


      if (reportToCreate.hasImpoundedFiles) {
        await step.do('Gather Files', async () => {
          if (!reportId) throw new Error('Report ID is undefined.');
          await gatherFilesStep(reportId, this.env);
        });


        await step.do('Upload Files', async () => {
          if (!reportId) throw new Error('Report ID is undefined.');
          await uploadFilesStep(reportId, this.env);
        });


        await step.do('Add File Metadata', async () => {
          if (!reportId) throw new Error('Report ID is undefined.');
          await addFilesInfoStep(reportId, this.env);
        });
      }


      await step.do('Finalize Report', async () => {
        if (!reportId) throw new Error('Report ID is undefined.');
        await finalizeReportStep(reportId, this.env);
      });
    } catch (error) {
      console.error(error);
      throw error;
    }
  }
}

Not only can tasks be broken into discrete steps, but the Workflows dashboard gives us real-time visibility into each report processed and the status of each step in the workflow!

This allows us to easily see active and completed workflows, identify which steps failed and where, and retry failed steps or terminate workflows. These features revolutionize how we troubleshoot issues, providing us with a tool to deep dive into any issues that arise and retry steps with a click of a button.

Below are two dashboard screenshots, one of our running workflows and the second of an inspection of the success and failures of each step in the workflow. Some workflows look slower or “stuck” — that’s because failed steps are retried with exponential backoff. This helps smooth over transient issues like flaky APIs without manual intervention.

^{Cloudflare Workflows Dashboard for our NCMEC Workflow}

^{Cloudflare Workflows Dashboard containing a breakout of the NCMEC Workflow Steps}

Cloudflare Workflows transformed how we handle NCMEC incident reports. What was once a complex, queue-based architecture is now a structured, retryable, and observable process. Debugging is easier, error handling is more robust, and monitoring is seamless. 

If you’re also building larger, multi-step applications, or have an existing Workers application that has started to approach what we ended up with for our incident reporting process, then you can typically wrap that code within a Workflow with minimal changes. Workflows can read from R2, write to KV, query D1 and call other APIs just like any other Worker, but are designed to help orchestrate asynchronous, long-running tasks.

To get started with Workflows, you can head to the Workflows developer documentation and/or pull down the starter project and dive into the code immediately:

$ npm create cloudflare@latest workflows-starter -- 
--template="cloudflare/workflows-starter"

Learn more about Cloudflare Workflows, and about using the Cloudflare CSAM Scanning Tool.

Cloudflare’s global network spans more than 310 cities in over 120 countries. That means thousands of servers geographically spread across different data centers, running services that protect and accelerate our customer’s Internet applications. Operating hardware at such a scale means that hardware can break anywhere and at any time. In such cases, our systems are engineered such that these failures cause little to no impact. However, detecting and managing server failure at scale requires automation. This blog aims to provide insights into the difficulties involved in handling broken servers and how we were able to simplify the process through automation.

When a server is found to have faulty hardware and needs to be removed from production, it is  considered broken and its state is set to Repair in the internal database where server status is tracked. In the past, our Data Center Operations team were essentially left to troubleshoot and diagnose broken servers on their own. They had to go through laborious tasks like performing queries to locate and repair servers, conducting diagnostics, reviewing results, evaluating if a server can be restored to production, and creating the necessary tickets for re-enabling servers and executing operations to put them back in production. Such effort can take hours for a single server alone, and can easily consume an engineer’s entire day.

As you can see, addressing server repairs was a labor-intensive process performed manually, Additionally, a lot of these servers remained powered on within the racks, wasting energy. With our fleet expanding rapidly, the attention of Data Center Operations is primarily devoted to supporting this growth, leaving less time to handle servers in need of repair.

It was clear that our infrastructure was growing too fast for us to be able to handle repairs and recovery, so we had to find a better way to handle these sorts of inefficiencies in our operations. This would allow our engineers to focus on the growth of our footprint while not abandoning repair and recovery – after all, these are still huge CapEx investments and wasted capacity that otherwise would have been fully utilized.

As members of the Infrastructure Software Systems and Automation team at Cloudflare, we primarily work on building tools and automation that help reduce excess work in order to ease the pressure on our operations teams, increase productivity, and enable people to execute operations with the highest efficiency.

Our team continuously strives to challenge our existing processes and systems, finding ways we can evolve them and make significant improvements – one of which is to build not just a typical automated system but an autonomous one. Building autonomous automations means creating systems that can operate independently, without the need for constant human intervention or oversight – a perfect example of this is Phoenix.

Phoenix is an autonomous diagnostics and recovery automation that runs at regular intervals to discover Cloudflare data centers with servers that are broken, performing diagnostics on detection, recovering those that pass diagnostics by re-provisioning, and ultimately re-enabling  those that have successfully been re-provisioned in the safest and most unobtrusive way possible – all without requiring any human intervention! Should a server fail at any point in the process, Phoenix will take care of updating relevant tickets, even pinpointing the cause of the failure, and reverting the state of the server accordingly when needed – again, all without any human intervention!

The image below illustrates the whole process:

To better understand exactly how Phoenix works, let’s dive into some details about its core functionality.

Discovery runs at a regular interval of 30 minutes, selecting a maximum of two Cloudflare data centers that have broken or repair state servers in its fleet, which are all configurable depending on business and operational needs, against which it can immediately execute diagnostics. At this rate, Phoenix is able to discover and operate on all broken servers in the fleet in about 3 days. On each run, it also detects data centers that may have broken servers already queued for recovery, and takes care of ensuring that the Recovery phase is executed immediately.

Diagnostics takes care of running various tests across the broken servers of a selected data center in a single run, verifying viability of the hardware components, and identifying the candidates for recovery.

A diagnostic operation includes running the following:

• Out-of-Band connectivity checkThis check determines the reachability of a device via out-of-band network. We employ IPMI (Intelligent Platform Management Interface) to ensure proper physical connectivity and accessibility of devices. This allows for effective monitoring and management of hardware components, enhancing overall system reliability and performance. Only devices that pass this check can progress to the Node Acceptance Testing phase.

• Node Acceptance TestsWe leverage an existing internally-built tool called [INAT](http://staging.blog.mrk.cfdata.org/redefining-fleet-management-at-cloudflare#:~:text=fleet%2C%20known%20as-,INAT,-(Integrated%20Node%20Acceptance) (Integrated Node Acceptance Testing) that runs various tests suites/cases (Hardware Validation, Performance, etc.).

  For every server that needs to be diagnosed, Phoenix will send relevant system instructions to have it boot into a custom Linux boot image, internally called INAT-image. Built into this image are the various tests that need to run when the server boots up, publishing the results to an internal resource in both human-readable (HTML) and machine-readable (JSON) formats, with the latter consumed and interpreted by Phoenix. Upon completion of the boot diagnostics, the server is powered off again to ensure it is not wasting energy.

Our node acceptance tests encompass a range of evaluations, including but not limited to benchmark testing, CPU/Memory/Storage checks, drive wiping, and various other assessments.  Look out for an upcoming in-depth blog post covering INAT.

A summarized diagnostics result is immediately added to the tracking ticket, including pinpointing the exact cause of a failure.

Recovery executes what we call an expansion operation, which in its first phase will provision the servers that pass diagnostics. The second phase is to re-enable the successfully provisioned servers back to production, where only those that have been re-enabled successfully will start receiving production traffic again.

Once the diagnostics are passed and the broken servers move on towards the first phase of recovery, we change their statuses from Repair to Pending Provision. If the servers don't fully recover, for example, because there are server configuration errors or issues enabling services, Phoenix assesses the situation. In such cases, it returns those servers to the Repair state for additional evaluation. Additionally, if the diagnostics indicate that the servers need any faulty components replaced, then Phoenix notifies our Data Center operation team for manual repairs as required, ensuring that the server is not repeatedly selected until the required part replacement is completed. This ensures any necessary human intervention can be applied promptly, making the server ready for Phoenix to rediscover in its next iteration.

An autonomous recovery operation requires infusing intelligence into the automated system so that we can fully trust that it’s able to execute an expansion operation in the safest way possible and handle situations on its own without any human interventions. To do this, we’ve made sure Phoenix is automation-aware – this means that it knows when there are other automations executing certain operations such as expansions, and will only execute an expansion when there are no ongoing provisioning operations in the target data center. This ability to execute only when it’s safe to do so is to ensure that the recovery operation will not interfere with any other ongoing operations in the data center. We’ve also adjusted its tolerance with faulty hardware – this means it’s able to gracefully deal with misbehaving servers by letting these quickly drop out of the recovery candidate list upon misbehavior that prevents blocking the operation.

While our autonomous system, Phoenix, seamlessly handles operations without human intervention, it doesn't mean we sacrifice visibility. Transparency is a key feature of Phoenix. It meticulously logs every operation, from executing tasks to providing progress updates, and shares this information in communication channels like chat rooms and Jira tickets. This ensures a clear understanding of what Phoenix is doing at all times.

Tracking of actions taken by automation as well as the state transitions of a server keeps us in the loop and gives us a better understanding of what these actions were and when they were executed, essentially giving us valuable insights that will help us improve not only the system but our processes as well. Having this operational data allows us to generate dashboards that let various teams monitor automation activities and measure their success. We are able to generate dashboards to guide business decisions and even answer common operational questions related to repair and recovery.

When we launched Phoenix, we were well aware that not every broken server can be re-enabled and successfully returned to production, and more importantly, there's no 100% guarantee that a recovered server will be as stable as the ones with no repair history – there's a risk that these servers could fail and end up back in Repair status again.

Although there's no guarantee that these recovered servers won't fail again, causing additional work for SRE’s due to the monitoring alerts that get triggered, what we can guarantee is that Phoenix immediately stops recoveries without any human intervention if a certain number of failures for a server are reached in a given time window – this is where we applied the concept of an Error Budget.

The Error Budget is the amount of error that automation can accumulate over a certain period of time before our SRE’s start being unhappy due to the excessive server failures or unreliability of the system. It is empathy embedded in automation.

In the figure above, the y-axis represents the error budget. In this context, the error budget applies to the number of recovered servers that failed and were moved back to Repair state again. The x-axis represents the time unit allocated to the error budget – in this case, 24 hours. To ensure that Phoenix is strict enough in mitigating possible issues, we divide the time unit into three consecutive buckets of the same duration – representing the three “follow the sun” SRE shifts in a day. With this, Phoenix can only execute recoveries if the number of server failures is no more than 2. Additionally, Phoenix will also have to compensate succeeding time buckets by deducting the error budget of any excess failures in a given time bucket.

Phoenix will immediately stop recoveries if it exhausts its error budget prematurely. In this context, prematurely means before the end of the time unit for which the error budget was granted. Regardless of the error budget depletion rate within a time unit, the error budget is fully replenished at the beginning of each time unit, meaning the budget resets every day.

The Error Budget has helped us define and manage our tolerance for hardware failures without causing significant harm to the system or too much noise for SREs, and gave us opportunities to improve our diagnostics system. It provides a common incentive that allows both the Infrastructure Engineering and SRE teams to focus on finding the right balance between innovation and reliability.

With Phoenix, we’ve not only witnessed the significant and far-reaching potential of having an autonomous automated system in our infrastructure, we’re actually reaping its benefits as well. It provides a win-win situation by successfully recovering hardware and ensuring that broken devices are powered off, thus preventing them from consuming unnecessary power while being idle in our racks. This not only reduces energy wastage but also contributes to sustainability efforts and cost savings. Automated processes that operate independently have not only freed our colleagues on various Infrastructure teams from doing mundane and repetitive tasks, allowing them to focus more on areas where they can use their skill sets for more interesting and productive work, but have also led us to evolving our old processes for handling hardware failures and repairs, making us much more efficient than ever.

Autonomous automation is a reality that is now beginning to shape the future of how we are building better and smarter systems here at Cloudflare, and we will continue to invest engineering time for these initiatives.

A huge thank you to Elvin Tan for his awesome work on INAT, and to Graeme, Darrel and David for INAT’s continuous improvements.