Building the large language models (LLMs) and diffusion models that power generative AI requires massive infrastructure. The most obvious component is compute – hundreds to thousands of GPUs – but an equally critical (and often overlooked) component is the data storage infrastructure. Training datasets can be terabytes to petabytes in size, and this data needs to be read in parallel by thousands of processes. In addition, model checkpoints need to be saved frequently throughout a training run, and for LLMs these checkpoints can each be hundreds of gigabytes!

To manage storage costs and scalability, many machine learning teams have been moving to object storage to host their datasets and checkpoints. Unfortunately, most object store providers use egress fees to “lock in” users to their platform. This makes it very difficult to leverage GPU capacity across multiple cloud providers, or take advantage of lower / dynamic pricing elsewhere, since the data and model checkpoints are too expensive to move. At a time when cloud GPUs are scarce, and new hardware options are entering the market, it’s more important than ever to stay flexible.

In addition to high egress fees, there is a technical barrier to object-store-centric machine learning training. Reading and writing data between object storage and compute clusters requires high throughput, efficient use of network bandwidth, determinism, and elasticity (the ability to train on different #s of GPUs). Building training software to handle all of this correctly and reliably is hard!

Today, we’re excited to show how MosaicML’s tools and Cloudflare R2 can be used together to address these challenges. First, with MosaicML’s open source StreamingDataset and Composer libraries, you can easily stream in training data and read/write model checkpoints back to R2. All you need is an Internet connection. Second, thanks to R2’s zero-egress pricing, you can start/stop/move/resize jobs in response to GPU availability and prices across compute providers, without paying any data transfer fees. The MosaicML training platform makes it dead simple to orchestrate such training jobs across multiple clouds.

Together, Cloudflare and MosaicML give you the freedom to train LLMs on any compute, anywhere in the world, with zero switching costs. That means faster, cheaper training runs, and no vendor lock in :)

“With the MosaicML training platform, customers can efficiently use R2 as the durable storage backend for training LLMs on any compute provider with zero egress fees. AI companies are facing outrageous cloud costs, and they are on the hunt for the tools that can provide them with the speed and flexibility to train their best model at the best price.” – Naveen Rao, CEO and co-founder, MosaicML

To read data from R2 efficiently and deterministically, you can use the MosaicML StreamingDataset library. First, write your training data (images, text, video, anything!) into .mds shard files using the provided Python API:

import numpy as np
from PIL import Image
from streaming import MDSWriter

# Local or remote directory in which to store the compressed output files
data_dir = 'path-to-dataset'

# A dictionary mapping input fields to their data types
columns = {
    'image': 'jpeg',
    'class': 'int'
}

# Shard compression, if any
compression = 'zstd'

# Save the samples as shards using MDSWriter
with MDSWriter(out=data_dir, columns=columns, compression=compression) as out:
    for i in range(10000):
        sample = {
            'image': Image.fromarray(np.random.randint(0, 256, (32, 32, 3), np.uint8)),
            'class': np.random.randint(10),
        }
        out.write(sample)

After your dataset has been converted, you can upload it to R2. Below we demonstrate this with the awscli command line tool, but you can also use `wrangler `or any S3-compatible tool of your choice. StreamingDataset will also support direct cloud writing to R2 soon!

$ aws s3 cp --recursive path-to-dataset s3://my-bucket/folder --endpoint-url $S3_ENDPOINT_URL

Finally, you can read the data into any device that has read access to your R2 bucket. You can fetch individual samples, loop over the dataset, and feed it into a standard PyTorch dataloader.

from torch.utils.data import DataLoader
from streaming import StreamingDataset

# Make sure that R2 credentials and $S3_ENDPOINT_URL are set in your environment    
# e.g. export S3_ENDPOINT_URL="https://[uid].r2.cloudflarestorage.com"

# Remote path where full dataset is persistently stored
remote = 's3://my-bucket/folder'

# Local working dir where dataset is cached during operation
local = '/tmp/path-to-dataset'

# Create streaming dataset
dataset = StreamingDataset(local=local, remote=remote, shuffle=True)

# Let's see what is in sample #1337...
sample = dataset[1337]
img = sample['image']
cls = sample['class']

# Create PyTorch DataLoader
dataloader = DataLoader(dataset)

StreamingDataset comes out of the box with high performance, elastic determinism, fast resumption, and multi-worker support. It also uses smart shuffling and distribution to ensure that download bandwidth is minimized. Across a variety of workloads such as LLMs and diffusion models, we find that there is no impact on training throughput (no dataloader bottleneck) when training from object stores like R2. For more information, check out the StreamingDataset announcement blog!

Streaming data into your training loop solves half of the problem, but how do you load/save your model checkpoints? Luckily, if you use a training library like Composer, it’s as easy as pointing at an R2 path!

from composer import Trainer
...

# Make sure that R2 credentials and $S3_ENDPOINT_URL are set in your environment
# e.g. export S3_ENDPOINT_URL="https://[uid].r2.cloudflarestorage.com"

trainer = Trainer(
        run_name='mpt-7b',
        model=model,
        train_dataloader=train_loader,
        ...
        save_folder=s3://my-bucket/mpt-7b/checkpoints,
        save_interval='1000ba',
        # load_path=s3://my-bucket/mpt-7b-prev/checkpoints/ep0-ba100-rank0.pt,
    )

Composer uses asynchronous uploads to minimize wait time as checkpoints are being saved during training. It also works out of the box with multi-GPU and multi-node training, and does not require a shared file system. This means you can skip setting up an expensive EFS/NFS system for your compute cluster, saving thousands of dollars or more per month on public clouds. All you need is an Internet connection and appropriate credentials – your checkpoints arrive safely in your R2 bucket giving you scalable and secure storage for your private models.

Using the above tools together with Cloudflare R2 enables users to run training workloads on any compute provider, with total freedom and zero switching costs.

As a demonstration, in Figure X we use the MosaicML training platform to launch an LLM training job starting on Oracle Cloud Infrastructure, with data streaming in and checkpoints uploaded back to R2. Part way through, we pause the job and seamlessly resume on a different set of GPUs on Amazon Web Services. Composer loads the model weights from the last saved checkpoint in R2, and the streaming dataloader instantly resumes to the correct batch. Training continues deterministically. Finally, we move again to Google Cloud to finish the run.

As we train our LLM across three cloud providers, the only costs we pay are for GPU compute and data storage. No egress fees or lock in thanks to Cloudflare R2!

Using the MosaicML training platform with Cloudflare R2 to run an LLM training job across three different cloud providers, with zero egress fees.

$ mcli get clusters
NAME            PROVIDER      GPU_TYPE   GPUS             INSTANCE                   NODES
mml-1            MosaicML   │  a100_80gb  8             │  mosaic.a100-80sxm.1        1    
                            │  none       0             │  cpu                        1    
gcp-1            GCP        │  t4         -             │  n1-standard-48-t4-4        -    
                            │  a100_40gb  -             │  a2-highgpu-8g              -    
                            │  none       0             │  cpu                        1    
aws-1            AWS        │  a100_40gb  ≤8,16,...,32  │  p4d.24xlarge               ≤4   
                            │  none       0             │  cpu                        1    
oci-1            OCI        │  a100_40gb  8,16,...,64   │  oci.bm.gpu.b4.8            ≤8  
                            │  none       0             │  cpu                        1    

$ mcli create secret s3 --name r2-creds --config-file path/to/config --credentials-file path/to/credentials
✔  Created s3 secret: r2-creds      

$ mcli create secret env S3_ENDPOINT_URL="https://[uid].r2.cloudflarestorage.com"
✔  Created environment secret: s3-endpoint-url      
               
$ mcli run -f mpt-125m-r2.yaml --follow
✔  Run mpt-125m-r2-X2k3Uq started                                                                                    
i  Following run logs. Press Ctrl+C to quit.                                                                            
                                                                                                                        
Cloning into 'llm-foundry'...

Using the MCLI command line tool to manage compute clusters, secrets, and submit runs.

### mpt-125m-r2.yaml ###
# set up secrets once with `mcli create secret ...`
# and they will be present in the environment in any subsequent run

integrations:
- integration_type: git_repo
  git_repo: mosaicml/llm-foundry
  git_branch: main
  pip_install: -e .[gpu]

image: mosaicml/pytorch:1.13.1_cu117-python3.10-ubuntu20.04

command: |
  cd llm-foundry/scripts
  composer train/train.py train/yamls/mpt/125m.yaml \
    data_remote=s3://bucket/path-to-data \
    max_duration=100ba \
    save_folder=s3://checkpoints/mpt-125m \
    save_interval=20ba

run_name: mpt-125m-r2

gpu_num: 8
gpu_type: a100_40gb
cluster: oci-1  # can be any compute cluster!

An MCLI job template. Specify a run name, a Docker image, a set of commands, and a compute cluster to run on.

The MosaicML platform is an invaluable tool to take your training to the next level, and in this post, we explored how Cloudflare R2 empowers you to train models on your own data, with any compute provider – or all of them. By eliminating egress fees, R2’s storage is an exceptionally cost-effective complement to MosaicML training, providing maximum autonomy and control. With this combination, you can switch between cloud service providers to fit your organization’s needs over time.

To learn more about using MosaicML to train custom state-of-the-art AI on your own data visit here or get in touch.

R2 is the ideal object storage platform to build data lakes. It’s infinitely scalable, highly durable (eleven 9's of annual durability), and has no egress fees. Zero egress fees mean zero vendor lock-in. You are free to use the tools you want to get the maximum value from your data.

Today we’re excited to announce our partnership with Snowflake so that you can use Snowflake to query data stored in your R2 data lake and load data from R2 into Snowflake. Organizations use Snowflake's Data Cloud to unite siloed data, discover, and securely share data, and execute diverse analytic workloads across multiple clouds.

One challenge of loading data into Snowflake database tables and querying external data lakes is the cost of data transfer. If your data is coming from a different cloud or even different region within the same cloud, this typically means you are paying an additional tax for each byte going into Snowflake. Pairing R2 and Snowflake lets you focus on getting valuable insights from your data, without having to worry about egress fees piling up.

If you haven’t already, you’ll need to sign up for R2 and create a bucket. You’ll also need to create R2 security credentials for Snowflake following the steps below.

1. In the Cloudflare dashboard, select R2.

2. Select Manage R2 API Tokens on the right side of the dashboard.

3. Select Create API token.

4. Optionally select the pencil icon or R2 Token text to edit your API token name.

5. Under Permissions, select Edit.

6. Select Create API Token.

You’ll need the Secret Access Key and Access Key ID to create an external stage in Snowflake.

In Snowflake, stages refer to the location of data files in object storage. To create an external stage, you’ll need your bucket name and R2 credentials. Find your Cloudflare account ID in the dashboard.

CREATE STAGE my_r2_stage
  URL = 's3compat://my_bucket/files/'
  ENDPOINT = 'cloudflare_account_id.r2.cloudflarestorage.com'
  CREDENTIALS = (AWS_KEY_ID = '1a2b3c...' AWS_SECRET_KEY = '4x5y6z...')

Note: You may need to contact your Snowflake account team to enable S3-compatible endpoints in Snowflake. Get more information here.

To load data from your R2 data lake into Snowflake, use the COPY INTO command.

COPY INTO t1
  FROM @my_r2_stage/load/;

You can flip the table and external stage parameters in the example above to unload data from Snowflake into R2.

You’ll first need to create an external table in Snowflake. Once you’ve done that you’ll be able to query your data stored in R2.

SELECT * FROM external_table;

For more information on how to use R2 and Snowflake together, refer to documentation here.

“Data is becoming increasingly the center of every application, process, and business metrics, and is the cornerstone of digital transformation. Working with partners like Cloudflare, we are unlocking value for joint customers around the world by helping save costs and helping maximize customers data investments,” – James Malone, Director of Product Management at Snowflake

We want to hear from you! If you have any feedback on the integration between Cloudflare R2 and Snowflake, please let us know by filling this form.

Be sure to check our Discord server to discuss everything R2!

As CIOs navigate the complexities of stitching together multiple solutions, we are extending our partnership with Microsoft to create one of the best Zero Trust solutions available. Today, we are announcing four new integrations between Azure AD and Cloudflare Zero Trust that reduce risk proactively. These integrated offerings increase automation allowing security teams to focus on threats versus implementation and maintenance.

Zero Trust is an overused term in the industry and creates a lot of confusion. So, let's break it down. Zero Trust architecture emphasizes the “never trust, always verify” approach. One way to think about it is that in the traditional security perimeter or “castle and moat” model, you have access to all the rooms inside the building (e.g., apps) simply by having access to the main door (e.g., typically a VPN).  In the Zero Trust model you would need to obtain access to each locked room (or app) individually rather than only relying on access through the main door. Some key components of the Zero Trust model are identity e.g., Azure AD (who), apps e.g., a SAP instance or a custom app on Azure (applications), policies e.g. Cloudflare Access rules (who can access what application), devices e.g. a laptop managed by Microsoft Intune (the security of the endpoint requesting the access) and other contextual signals.

Zero Trust is even more important today since companies of all sizes are faced with an accelerating digital transformation and an increasingly distributed workforce. Moving away from the castle and moat model, to the Internet becoming your corporate network, requires security checks for every user accessing every resource. As a result, all companies, especially those whose use of Microsoft’s broad cloud portfolio is increasing, are adopting a Zero Trust architecture as an essential part of their cloud journey.

Cloudflare’s Zero Trust platform provides a modern approach to authentication for internal and SaaS applications. Most companies likely have a mix of corporate applications - some that are SaaS and some that are hosted on-premise or on Azure. Cloudflare’s Zero Trust Network Access (ZTNA) product as part of our Zero Trust platform makes these applications feel like SaaS applications, allowing employees to access them with a simple and consistent flow. Cloudflare Access acts as a unified reverse proxy to enforce access control by making sure every request is authenticated, authorized, and encrypted.

We have thousands of customers using Azure AD and Cloudflare Access as part of their Zero Trust architecture. Our partnership with Microsoft  announced last year strengthened security without compromising performance for our joint customers. Cloudflare’s Zero Trust platform integrates with Azure AD, providing a seamless application access experience for your organization's hybrid workforce.

As a recap, the integrations we launched solved two key problems:

1. For on-premise legacy applications, Cloudflare’s participation as Azure AD secure hybrid access partner enabled customers to centrally manage access to their legacy on-premise applications using SSO authentication without incremental development. Joint customers now easily use Cloudflare Access as an additional layer of security with built-in performance in front of their legacy applications.

2. For apps that run on Microsoft Azure, joint customers can integrate Azure AD with Cloudflare Zero Trust and build rules based on user identity, group membership and Azure AD Conditional Access policies. Users will authenticate with their Azure AD credentials and connect to Cloudflare Access with just a few simple steps using Cloudflare’s app connector, Cloudflare Tunnel, that can expose applications running on Azure. See guide to install and configure Cloudflare Tunnel.

Recognizing Cloudflare's innovative approach to Zero Trust and Security solutions, Microsoft awarded us the Security Software Innovator award at the 2022 Microsoft Security Excellence Awards, a prestigious classification in the Microsoft partner community.

But we aren’t done innovating. We listened to our customers’ feedback and to address their pain points are announcing several new integrations.

The four new integrations we are announcing today are:

1. Per-application conditional access: Azure AD customers can use their existing Conditional Access policies in Cloudflare Zero Trust.

Azure AD allows administrators to create and enforce policies on both applications and users using Conditional Access. It provides a wide range of parameters that can be used to control user access to applications (e.g. user risk level, sign-in risk level, device platform, location, client apps, etc.). Cloudflare Access now supports Azure AD Conditional Access policies per application. This allows security teams to define their security conditions in Azure AD and enforce them in Cloudflare Access.

For example, customers might have tighter levels of control for an internal payroll application and hence will have specific conditional access policies on Azure AD. However, for a general info type application such as an internal wiki, customers might enforce not as stringent rules on Azure AD conditional access policies. In this case both app groups and relevant Azure AD conditional access policies can be directly plugged into Cloudflare Zero Trust seamlessly without any code changes.

**2. **SCIM****: Autonomously synchronize Azure AD groups between Cloudflare Zero Trust and Azure AD, saving hundreds of hours in the CIO org.

Cloudflare Access policies can use Azure AD to verify a user's identity and provide information about that user (e.g., first/last name, email, group membership, etc.). These user attributes are not always constant, and can change over time. When a user still retains access to certain sensitive resources when they shouldn’t, it can have serious consequences.

Often when user attributes change, an administrator needs to review and update all access policies that may include the user in question. This makes for a tedious process and an error-prone outcome.

The SCIM (System for Cross-domain Identity Management) specification ensures that user identities across entities using it are always up-to-date. We are excited to announce that joint customers of Azure AD and Cloudflare Access can now enable SCIM user and group provisioning and deprovisioning. It will accomplish the following:

• The IdP policy group selectors are now pre-populated with Azure AD groups and will remain in sync. Any changes made to the policy group will instantly reflect in Access without any overhead for administrators.

• When a user is deprovisioned on Azure AD, all the user's access is revoked across Cloudflare Access and Gateway. This ensures that change is made in near real time thereby reducing security risks.

**3. **Risky user isolation****: Helps joint customers add an extra layer of security by isolating high risk users (based on AD signals) such as contractors to browser isolated sessions via Cloudflare’s RBI product.

Azure AD classifies users into low, medium and high risk users based on many data points it analyzes. Users may move from one risk group to another based on their activities. Users can be deemed risky based on many factors such as the nature of their employment i.e. contractors, risky sign-in behavior, credential leaks, etc. While these users are high-risk, there is a low-risk way to provide access to resources/apps while the user is assessed further.

We now support integrating Azure AD groups with Cloudflare Browser Isolation. When a user is classified as high-risk on Azure AD, we use this signal to automatically isolate their traffic with our Azure AD integration. This means a high-risk user can access resources through a secure and isolated browser. If the user were to move from high-risk to low-risk, the user would no longer be subjected to the isolation policy applied to high-risk users.

4. Secure joint Government Cloud customers: Helps Government Cloud customers achieve better security with centralized identity & access management via Azure AD, and an additional layer of security by connecting them to the Cloudflare global network, not having to open them up to the whole Internet.

Via Secure Hybrid Access (SHA) program, Government Cloud (‘GCC’) customers will soon be able to integrate Azure AD with Cloudflare Zero Trust and build rules based on user identity, group membership and Azure AD conditional access policies. Users will authenticate with their Azure AD credentials and connect to Cloudflare Access with just a few simple steps using Cloudflare Tunnel that can expose applications running on Microsoft Azure.

“Digital transformation has created a new security paradigm resulting in organizations accelerating their adoption of Zero Trust. The Cloudflare Zero Trust and Azure Active Directory joint solution has been a growth enabler for Swiss Re by easing Zero Trust deployments across our workforce allowing us to focus on our core business. Together, the joint solution enables us to go beyond SSO to empower our adaptive workforce with frictionless, secure access to applications from anywhere. The joint solution also delivers us a holistic Zero Trust solution that encompasses people, devices, and networks.”– Botond Szakács, Director, Swiss Re

“A cloud-native Zero Trust security model has become an absolute necessity as enterprises continue to adopt a cloud-first strategy. Cloudflare has developed robust product integrations with Microsoft to help security and IT leaders prevent attacks proactively, dynamically control policy and risk, and increase automation in alignment with zero trust best practices.”– Joy Chik, President, Identity & Network Access, Microsoft

Interested in learning more about how our Zero Trust products integrate with Azure Active Directory? Take a look at this extensive reference architecture that can help you get started on your Zero Trust journey and then add the specific use cases above as required. Also, check out this joint webinar with Microsoft that highlights our joint Zero Trust solution and how you can get started.

We are just getting started. We want to continue innovating and make the Cloudflare Zero Trust and Microsoft Security joint solution to solve your problems. Please give us feedback on what else you would like us to build as you continue using this joint solution.

Today, we are very excited to announce our new integration with Microsoft Endpoint Manager (Intune). This integration combines the power of Cloudflare’s expansive network and Zero Trust suite, with Endpoint Manager. Via our existing Intune integration, joint customers can check if a device management profile such as Intune is running on the device or not and grant access accordingly.

With this expanded integration, joint customers can identify, investigate, and remediate threats faster. The integration also includes the latest information from Microsoft Graph API which provides many added, real-time device posture assessments and enables organizations to verify users' device posture before granting access to internal or external applications.

"In today’s work-from-anywhere business culture, the risk of compromise has substantially increased as employees and their devices are continuously surrounded by a hostile threat environment outside the traditional castle-and-moat model. By expanding our integration with Cloudflare, we are making it easier for joint customers to strengthen their Zero Trust security posture across all endpoints and their entire corporate network."– Dave Randall, Sr Program Manager, Microsoft Endpoint Manager

Before we get deep into how the integration works, let’s first recap Cloudflare’s Zero Trust Services.

Cloudflare Access determines if a user should be allowed access to an application or not. It uses our global network to check every request or connection for identity, device posture, location, multifactor method, and many more attributes to do so. Access also logs every request and connection — providing administrators with high-visibility. The upshot of all of this: it enables customers to deprecate their legacy VPNs.

Cloudflare Gateway protects users as they connect to the rest of the Internet. Instead of backhauling traffic to a centralized location, users connect to a nearby Cloudflare data center where we apply one or more layers of security, filtering, and logging, before accelerating their traffic to its final destination.

Cloudflare’s customers can now build Access and Gateway policies based on the device being managed by Endpoint Manager (Intune) with a compliance policy defined. In conjunction with our Zero Trust client, we are able to leverage the enhanced telemetry that Endpoint Manager (Intune) provides surrounding a user’s device.

Microsoft’s Graph API delivers continuous real-time security posture assessments such as Compliance State across all endpoints in an organization regardless of the location, network or user. Those key additional device posture data enable enforcement of conditional policies based on device health and compliance checks to mitigate risks. These policies are evaluated each time a connection request is made, making the conditional access adaptive to the evolving condition of the device.

With this integration, organizations can build on top of their existing Cloudflare Access and Gateway policies ensuring that a ‘Compliance State’ has been met before a user is granted access. Because these policies work across our entire Zero Trust platform, organizations can use these to build powerful rules invoking Browser Isolation, tenant control, antivirus or any part of their Cloudflare deployment.

Customers using our Zero Trust suite can add Microsoft Intune as a device posture provider in the Cloudflare Zero Trust dashboard under Settings → Devices → Device Posture Providers. The details required from the Microsoft Endpoint Manager admin center to set up policies on Cloudflare dashboard include: ClientID, Client Secret, and Customer ID.

After creating the Microsoft Endpoint Manager Posture Provider, customers can create specific device posture checks requiring users’ devices to meet certain criteria such as device ‘Compliance State’.

These rules can now be used to create conditional Access and Gateway policies to allow or deny access to applications, networks, or sites. Administrators can choose to block or isolate users or user groups with malicious or insecure devices.

In the coming months, we will be further strengthening our integrations with the Microsoft Graph API by allowing customers to correlate many other fields in the Graph API to enhance our joint customers’ security policies.

If you’re using Cloudflare Zero Trust products today and are interested in using this integration with Microsoft Intune, please visit our documentation to learn about how you can enable it. If you want to learn more or have additional questions, please fill out the form or get in touch with your Cloudflare CSM or AE, and we'll be happy to help you.

Recently, Microsoft announced the winners for the 2022 Microsoft Security Excellence Awards, a prestigious classification in the Microsoft partner community. We are honored to announce that Cloudflare has won the Security Software Innovator award. This award recognized Cloudflare's innovative approach to Zero Trust and Security solutions. Our transformative technology in collaboration with Microsoft provides world-class joint solutions for our mutual customers.

The third annual Microsoft Security awards celebrated finalists in 10 categories spanning security, compliance, and identity. Microsoft unveiled the winners of the Microsoft Security Partner Awards, voted on by a group of industry veterans, on June 6, 2022.

Through this award, Microsoft recognizes Cloudflare’s approach to constantly deliver the most innovative solutions for joint customers. Together with Microsoft, we have supported thousands of customers including many of the largest Fortune 500 companies on their Zero Trust journey, enabling customers to simply and easily support their security needs with faster performance.

Cloudflare has built deep integrations with Microsoft to help organizations take the next step in their Zero Trust journey. These integrations empower organizations to make customer implementations operationally efficient while delivering a seamless user experience. Currently, all our mutual customers benefit from several integrations across Microsoft 365 and Azure to secure web applications and safeguard employees with identity and device protections. Working with Microsoft has been critical in helping our customers on their Zero Trust journey. It is a complex undertaking that Cloudflare has been simplifying through our extremely easy to adopt product portfolio such as Cloudflare One via a  single pane of glass.

We want to thank Microsoft for its continued collaboration with Cloudflare. We are committed to serving our joint customers as we expand our integrations across Microsoft’s suite of products and continuously innovate against the latest threats.

“Partners are critical to solving customers’ constantly evolving security challenges and threat landscape. The close collaboration and deep integrations between Cloudflare and Microsoft ensure our joint customers are equipped with innovative technologies that are seamlessly integrated to address their security challenges. We are pleased to recognize Cloudflare with the Security Software Innovator Award at this year's Microsoft Security Excellence Awards.”– Ann Johnson, Corporate Vice President of Security, Compliance, Identity, and Management, Business Development at Microsoft.

Perimeter based security models are breaking under pressure

The rapid transition to remote work and the rise of SaaS applications has disrupted how businesses need to think about protecting their networks. Organizations historically protected their sensitive applications and networks by building a “castle-and-moat”, piecing together disparate point solutions for each defensive layer.

Comprehensive solutions require a layered defensive architecture for Internet security (DNS and HTTPS filtering), endpoint and data protection (Remote Browser Isolation and Data-Loss Prevention) as well as SaaS app security (CASB) and connecting users both in-and-away from the office via private network connections. This model is difficult to implement and manage, and doesn’t scale in the modern workplace with users and applications residing everywhere that is connected to the Internet.

Why Zero Trust is a must-have:

1. Apps can now live anywhere on-prem, cloud or SaaS

2. Employees can access those resources from anywhere

3. Attacks are getting more sophisticated constantly

4. Internet is the new ‘Office’ away from ‘Castle-Moat’ model

Current world of how applications are deployed and accessed

Cloudflare One protects any application or network for users everywhere by running our full suite of product across our global network present in more than 270 cities around the world:

• Protect any self-hosted or SaaS application with Access.

• Inspect and protect Internet access with Gateway.

• Isolate sensitive applications and high-risk browsing with Browser Isolation.

• Protection from data-loss with CASB and DLP controls.

Finally, any device, office or network can be protected by Cloudflare One by connecting to our closest point of presence via our Roaming Agent (WARP) or via tunneled or direct connectivity.

Our current integrations with Microsoft within the context of a request flow

This is the first year that Microsoft has a  Software Security Innovator award category, and we’re extremely proud to have won. Cloudflare is committed to strive and deliver next generation innovative Zero Trust solutions to our customers. If you are interested in our Cloudflare One suite, please reach out. Also, if you are interested in partnering with our Zero Trust solutions, fill out the form here.

Deploying an application to the cloud requires a robust network, ample storage, and compute power. But that only covers the architecture, it doesn’t include all the other tools and services necessary to build, deploy, and support your applications. It’s easy to get bogged down in researching whether everything you want to use works together, and that takes time away from actually developing and building.

Cloudflare is focused on building a modern cloud with a developer-first experience. That developer experience includes creating an ecosystem filled with the tools and services developers need.

Deciding to build an application starts with the planning and design elements. What programming languages, frameworks, or runtimes will be used? The answer to that question can influence the architecture and hosting decisions. Choosing a runtime may lock you into using specific vendors. Our goal is to provide flexibility and eliminate the friction involved when wanting to migrate on to (or off) Cloudflare.

You can’t forget the tools necessary for configuration and interoperability. This can include areas such as authentication and infrastructure management. You have multiple pieces of your application that need to talk with one another to ensure a secure and seamless experience. We support and build standards for compatibility and to simplify the configuration and interoperability when deploying your applications.

During the build process and after your application has been deployed, it is essential to test and verify that things are working properly. This is where logging, analytics, and alerting come in. Having the ability to quickly be notified when something isn’t working properly is part of the equation. The other part is reducing the context switching among various debugging tools and applications to quickly resolve the issue.

And finally, improving your application through iterating and extending functionality. This can come from adding plug-ins from third party providers, running A/B tests to determine whether a feature is moving in the right direction, or not.

At Cloudflare our mission is to help build a better Internet, all of these elements are necessary to achieve this. Our focus is on providing the network, storage, and compute power to deliver apps. We look to partners to help in the other areas and provide an ecosystem. Our intention with this strategy is to get out of the way, so developers can build faster and safer.

To achieve our mission we look for partners with similar philosophies. We’ve previously announced integrations around

• Content platforms to help with the planning, creating and building of applications.

• Database connectivity

• And observability to assist with debugging and troubleshooting

Everything we do is built on the foundation of the open Internet and open standards. We want to provide choice to developers to pick from a variety of tools and may the best solution win. We believe developers should have the freedom and flexibility to choose the best tool ‘for their work’ not the best for within a ‘locked-in-platform’.

We understand that partnering is key to achieving the above goal. Today, we’re excited to announce a growing ecosystem of partners who are tapping into our world’s best network. Our developer platform products and partner products fit seamlessly into existing stacks and improve developer productivity through simple development and deploy workflows.

Along with having the best network in the world, we also have really good compute and database products. However, for developers to plan, build and continuously iterate, they need a host of other services to work together seamlessly to have a great building experience end to end. This is why we partnered with DevOps platforms and CMSs to make the entire development lifecycle better.

On the DevOps front, developers can create new Pages projects by connecting thier repos stored on GitLab or GitHub and make site changes there via the usual git commands.

“Developers can be more productive when they create, test, secure, and deploy software from a single DevOps application instead of bouncing between multiple different tools. Cloudflare Pages’ integration with GitLab makes it easier for joint users to develop and deploy new code to Cloudflare’s network using the same syntax and git commands they’re already comfortable using.”— Michael LeBeau, Alliance Manager at GitLab

Also, on the CMS front, we have built deploy hooks that are the key to what allows developers to connect and trigger deployments in Cloudflare Pages via updates made in their favorite headless CMSs.

“With this integration, our customers can work more efficiently and cross-functionally with their teams. Marketing teams can update Contentful directly to automatically trigger deployments without relying on their developers to update any of their code base, which results in a better, more productive experience for all teams involved.”— Jeff Blattel, Director of Technical Partnerships at Contentful

“We’re delighted to partner with Cloudflare and excited by this new release from Cloudflare Pages. At Sanity, we care deeply about people working with content on our platform. Cloudflare’s new deploy hooks allow developers to automate builds for static sites based on content changes, which is a huge improvement for content creators. Combining these with structured content and our GROQ-powered Webhooks, our customers can be strategic about when these builds should happen. We’re stoked to be part of this release and can’t wait to see what the community will build with Sanity and Cloudflare!”— Even Westvang, Co-founder, Sanity.io

“At Strapi, we’re excited about this partnership with Cloudflare because it enables developers to abstract away the complexity of deploying changes to production for content teams. By using the Deploy Hook for Strapi, everyone can push new content and updates quickly and autonomously.”— Pierre Burgy, CEO, Strapi.io

Database partners:We’ve consistently heard a common theme: developers want to build more of their applications on Workers. In addition to having our own first-party solutions such as Workers KV, Durable Objects, R2 and D1 we have also partnered with best-of-breed data companies to bring their capabilities to the Workers developer platform and open up a wide variety of use cases on Workers.

“With Cloudflare and Macrometa, developers can now build and deliver powerful and compelling data-driven experiences in a way that the centralized clouds will never be able to. The world deserves a better cloud - an edge cloud.”— Chetan Venkatesh, Co-founder/CEO, Macrometa

“The integration of Cloudflare workers with Fauna allows our joint customers to simply and cost effectively bring both data and logic to the edge. Fauna was developed from the ground up as a globally distributed, serverless database and when combined with Cloudflare Workers serverless compute fabric provides developers with a simple, yet powerful abstraction over the complexities of distributed infrastructure.”— Evan Weaver, Co-founder/CTO, Fauna

Debugging, analytics, troubleshooting partners:Troubleshooting is key in the developer workflow. True observability means having an end-to-end view of an entire system, inclusive of third-party integrations and deployments. By combining real-time data with data from other logging and monitoring endpoints, customers can get robust, instant feedback into how their website and how applications are performing all in one place.

“Teams using Cloudflare Workers with Splunk Observability get full-stack visibility and contextualized insights from metrics, traces and logs across all of their infrastructure, applications and user transactions in real-time and at any scale. With Splunk Observability, IT and DevOps teams now have a seamless and analytics-powered workflow across monitoring, troubleshooting, incident response and optimization. We're excited to partner with Cloudflare to help developers and operations teams slice through the complexity of modern applications and ship code more quickly and reliably.”— Jeff Lo, Director of Product Marketing, Splunk

“Maintaining a strong security posture means ensuring every part of your toolchain is being monitored - from the datacenter/VPC, to your edge network, all the way to your end users. With Datadog’s partnership with Cloudflare, you get edge computing logs alongside the rest of your application stack’s telemetry - giving you an end to end view of your application’s health, performance and security.”— Michael Gerstenhaber, Sr. Director, Datadog

“Reduce downtime and solve customer-impacting issues faster with an integrated observability platform for all of your Cloudflare data, including its Workers serverless platform. By using Cloudflare Workers with Sumo Logic, customers can seamlessly correlate system issues measured by performance monitoring, gain deep visibility provided by logging, and monitor user experience provided by tracing and transaction analytics.”— Abelardo Gonzalez, Director of Product Marketing, Sumo Logic

“Monitoring your Cloudflare Workers serverless environment with New Relic lets you deliver serverless apps with confidence by rapidly identifying when something goes wrong and quickly pinpointing the problem—without wading through millions of invocation logs. Monitor, visualize, troubleshoot, and alert on all your Workers apps in a single experience.”— Raj Ramanujam, Vice President, Alliances and Channels, New Relic

“With Cloudflare Workers and Sentry, software teams immediately have the tools and information to solve issues and learn continuously about their code health instead of worrying about systems and resources. We’re thrilled to partner with Cloudflare on building technologies that make it easy for developers to deploy with confidence.”— Elain Szu, Vice President of Marketing, Sentry

“Honeycomb is excited to partner with Cloudflare as they build an ecosystem of tools that support the full lifecycle of delivering successful apps. Writing and deploying code is only part of the equation. Understanding how that code performs and behaves when it is in the hands of users also determines success. Cloudflare and Honeycomb together are shining the light of observability all the way to the edge, which helps technical teams build and operate better apps.”— Charity Majors, CTO & Co-Founder, Honeycomb

And we are just getting started. Over the last 10 years, Cloudflare has built one of the fastest, most reliable, and most secure networks in the world. We are now enabling developers to build on that network using best in class products seamlessly. Visit our Developer Platform Partner Ecosystem directory for more information on integrations.

If you’re interested in learning more about how to partner with us, reach out to us by filling out this quick form.

As a company, we are constantly asking ourselves what we can do to provide more value to our customers, including integrated solutions with our partners. Joint customers benefit from our integrations below with Azure Active Directory by:

First, centralized identity and access management via Azure Active Directory which provides single sign-on, multifactor authentication, and access via conditional authentication.

Second, policy oriented access to specific applications using Cloudflare Access—a VPN replacement service.

Third, an additional layer of security for internal applications by connecting them to Cloudflare global network and not having to open them up to the whole Internet.

Let’s step back a bit.

Companies of all sizes are faced with an accelerating digital transformation of their IT stack and an increasingly distributed workforce, changing the definition of the security perimeter. We are moving away from the castle and moat model to the whole Internet, requiring security checks for every user accessing every resource. As a result, all companies, especially those whose use of Azure’s broad cloud portfolio is increasing, are adopting Zero Trust architectures as an essential part of their cloud and SaaS journey.

Cloudflare Access provides secure access to Azure hosted applications and on-premise applications. Also, it acts as an on-ramp to the world’s fastest network to Azure and the rest of the Internet. Users connect from their devices or offices via Cloudflare’s network in over 250 cities around the world. You can use Cloudflare Zero Trust on that global network to ensure that every request to every resource is evaluated for security, including user identity. We are excited to bring this secure global network on-ramp to Azure hosted applications and on-premise applications.

Also, performance is one of our key advantages in addition to security. Cloudflare serves over 32 million HTTP requests per second, on average, for the millions of customers who secure their applications on our network. When those applications do not run on our network, we can rely on our own global private backbone and our connectivity with over 10,000 networks globally to connect the user.

We are excited to bring this global security and performance perimeter network as our Cloudflare Zero Trust product for your Azure hosted applications and on-premises applications.

Cloudflare’s Zero Trust solution Cloudflare Access provides a modern approach to authentication for internally managed applications. When corporate applications on Azure or on-premise are protected with Cloudflare Access, they feel like SaaS applications, and employees can log in to them with a simple and consistent flow. Cloudflare Access acts as a unified reverse proxy to enforce access control by making sure every request is authenticated, authorized, and encrypted.

Identity: Cloudflare Access integrates out of the box with all the major identity providers, including Azure Active Directory, allowing use of the policies and users you already created to provide conditional access to your web applications. For example, you can use Cloudflare Access to ensure that only company employees and no contractors can get to your internal kanban board, or you can lock down the SAP finance application hosted on Azure or on-premise.

Devices: You can use TLS with Client Authentication and limit connections only to devices with a unique client certificate. Cloudflare will ensure the connecting device has a valid client certificate signed by the corporate CA, and authenticate user credentials to grant access to an internal application.

Additional security: Want to use Cloudflare Access in front of an internal application but don’t want to open up that application to the whole Internet? For additional security, you can combine Access with Cloudflare Tunnel. Cloudflare Tunnel will connect from your Azure environment directly to Cloudflare’s network, so there is no publicly accessible IP.

For on-premise legacy applications, we are excited to announce that Cloudflare is an Azure Active Directory secure hybrid access partner. Azure AD secure hybrid access enables customers to centrally manage access to their legacy on-premise applications using SSO authentication without incremental development. Starting today, joint customers can easily use Cloudflare Access solution as an additional layer of security with built-in performance, in front of their legacy applications.

Traditionally for on-premise applications, customers have to change their existing code or add additional layers of code to integrate Azure AD or Cloudflare Access–like capabilities. With the help of Azure Active Directory secure hybrid access, customers can integrate these capabilities seamlessly without much code changes. Once integrated, customers can take advantage of the below Azure AD features and more:

1. Multi-factor authentication (MFA)

2. Single sign-on (SSO)

3. Passwordless authentication

4. Unified user access management

5. Azure AD Conditional Access and device trust

Very similarly, the Azure AD and Cloudflare Access combo can also be used to secure your Azure hosted applications. Cloudflare Access enables secure on-ramp to Azure hosted applications or on-premise applications via the below two integrations:

1. Cloudflare Access integration with Azure AD:

Cloudflare Access is a Zero Trust Network Access (ZTNA) solution that allows you to configure precise access policies across their applications. You can integrate Microsoft Azure Active Directory with Cloudflare Zero Trust and build rules based on user identity, group membership and Azure AD Conditional Access policies. Users will authenticate with their Azure AD credentials and connect to Cloudflare Access. Additional policy controls include Device Posture, Network, Location and more. Setup typically takes less than a few hours!

2. Cloudflare Tunnel integration with Azure:

Cloudflare Tunnel can expose applications running on the Microsoft Azure platform. See guide to install and configure Cloudflare Tunnel. Also, a prebuilt Cloudflare Linux image exists on the Azure Marketplace. To simplify the process of connecting Azure applications to Cloudflare’s network, deploy the prebuilt image to an Azure resource group. Cloudflare Tunnel is now available on Microsoft’s Azure marketplace.

“The hybrid work environment has accelerated the cloud transition and increased the need for CIOs everywhere to provide secure and performant access to applications for their employees. This is especially true for self-hosted applications. Cloudflare's global network security perimeter via Cloudflare Access provides this additional layer of security together with Azure Active Directory to enable employees to get work done from anywhere securely and performantly.” said David Gregory, Principal Program Manager- Microsoft Azure Active Directory

Over the last ten years, Cloudflare has built one of the fastest, most reliable, most secure networks in the world. You now have the ability to use that network as a global security and performance perimeter to your Azure hosted applications via the above integrations, and it is easy.

In the coming months, we will be further strengthening our integrations with Microsoft Azure allowing customers to better implement our SASE perimeter.

If you’re using Cloudflare Zero Trust products today and are interested in using this integration with Azure, please visit our above developer documentation to learn about how you can enable it. If you want to learn more or have additional questions, please fill out the form or get in touch with your Cloudflare CSM or AE, and we'll be happy to help you.

We are excited to announce that Cloudflare has joined the Microsoft 365 Networking Partner Program (NPP).  Cloudflare One, which provides an optimized path for traffic from Cloudflare customers to Microsoft 365, recently qualified for the NPP by demonstrating that on-ramps through Cloudflare’s network help optimize user connectivity to Microsoft.

Customers who deploy Cloudflare One give their team members access to the world’s fastest network, on average, as their on-ramp to the rest of the Internet. Users connect from their devices or offices and reach Cloudflare’s network in over 250 cities around the world. Cloudflare’s network accelerates traffic to its final destination through a combination of intelligent routing and software improvements.

We’re also excited that, in many cases, the final destination that a user visits already sits on Cloudflare’s network. Cloudflare serves over 28M HTTP requests per second, on average, for the millions of customers who secure their applications on our network. When those applications do not run on our network, we can rely on our own global private backbone and our connectivity with over 10,000 networks globally to connect the user.

For Microsoft 365 traffic, we focus on breaking out traffic as locally and direct as possible to bring users to the productivity tools they need without slowing them down. Legacy security solutions can introduce additional hops or backhauling that slows down connectivity to tools like Microsoft 365. With Cloudflare One, we provide the flexibility to identify that traffic and give it the most direct path to Microsoft’s own network of service endpoints around the world.

With this setting, trusted traffic to Microsoft uses the most direct path without additional processing. However, the rest of the Internet should not be trusted. Cloudflare’s network also secures the connections, queries, and requests your teams make to protect organizations from attacks and data loss. We can do that without slowing users down because we deliver that security in the data centers at our edge.

SaaS applications delivered over the Internet can make any device with a browser into a workstation. However, that also means that those same devices can connect to the rest of the Internet. Attackers try to lure users into lookalike sites to steal credentials, or they attempt to have users download malware to compromise the device. Either type of attack can put the data stored in SaaS applications at risk.

Cloudflare helps organizations stop those types of attacks through a defense-in-depth strategy. First, Cloudflare starts by delivering a next-generation network firewall in our data centers, filtering traffic for connections to potentially dangerous destinations. Next, Cloudflare runs the world’s fastest DNS resolver and combines it with the data we see about the rest of the Internet to filter queries to phishing domains or sites that host malware.

Finally, Cloudflare’s Secure Web Gateway can inspect HTTP traffic for data loss, viruses, or can choose to isolate the browser for specific sites or entire categories. While Cloudflare’s network secures users from attacks on the rest of the Internet, Cloudflare One ensures that users have a direct, unfettered connection to the Microsoft 365 tools they need.

With traffic secured, Cloudflare can also give administrators visibility into the other applications used in their organization. Without any additional software or features, Cloudflare uses its Zero Trust security suite to analyze and categorize the requests to all applications in a comprehensive Shadow IT report. Administrators can mark applications as approved, unapproved, or unknown and pending investigation so for example Administrators could mark Microsoft 365 traffic as approved -- which is also the default setting in deployments that use the one-click enablement being released today.

In some cases, that visibility leads to surprises. Security and IT teams discover that users are doing work in SaaS platforms that have not been reviewed and approved by the organization. In those cases, teams can use Cloudflare’s Secure Web Gateway to block requests to those destinations or just to prevent certain types of activities like blocking file uploads to tools other than OneDrive. With Shadow IT, we can help teams that use Microsoft 365 ensure that data only stays in Microsoft 365.

Cloudflare has joined the Microsoft 365 Networking Partner Program (NPP). The program is designed to offer customers a set of partners whose deployment practices and guidance are aligned with Microsoft’s networking principles for Microsoft 365 to provide users with the best user experience. Microsoft established the NPP to work with networking companies for optimal connectivity to its service. We are excited to work with a partner whose global network and security principles align with ours.

Starting today, through Cloudflare One, organizations have the ability to ensure as direct a connection as possible for Microsoft 365 traffic. This allows our customers with our WARP client to benefit from a seamless user experience for Microsoft 365, while at the same time securing the rest of their traffic either to SaaS apps, on-prem apps or direct internet traffic through Cloudflare’s global network and security suite of products.

To do this all customers need to do is to enable the Microsoft 365 traffic optimization setting in their Cloudflare One dashboard. Via the setting even if Microsoft 365 connections are routed through the Cloudflare gateways, they are being handled with the least amount of additional overhead for example "Do not inspect" policy is automatically enabled.

It's very easy to enable with just a few clicks:

1. Log into the Cloudflare for Teams dashboard.

2. Go to Settings > Network.

3. For Exclude Office 365 traffic and Bypass Office 365 traffic, click Create entries.

“We’re thrilled to welcome Cloudflare into the Networking Partner Program for Microsoft 365,” said Scott Schnoll, Senior Product Marketing Manager, Microsoft. “Cloudflare is a valued partner that is focused on helping Microsoft 365 customers implement the Microsoft 365 Network Connectivity Principles. Microsoft only recommends Networking Partner Program member solutions for connectivity to Microsoft 365.”

Your organization can start deploying Cloudflare One today alongside your existing Microsoft 365 usage. We’re excited to work with Microsoft to give your team members fast, reliable, and secure connectivity to the tools they need to do their jobs.

Interest in headless CMSes has seen spectacular growth over the past few years with many businesses looking to adopt the tooling. As audiences consume content through new interfaces taking different forms — smartphones, wearables, personal devices — the idea of decoupling content with its backend begins to provide a better experience both for developing teams and end users. Because of this, we believe there are and will be more opportunities in the future to utilize headless CMSes which is why today, we’re thrilled to announce our partnerships with Sanity and Strapi and also share existing integrations with Contentful and WordPress — all your favorite CMS providers.

Headless CMSes are one of the most common API integrations we’ve seen so far among you and your teams — whether it’s for your marketing site, blog or e-commerce site. It provides your teams the ability to input the contents of your site through a user-friendly interface and store them in a database, so that updates can easily be made to your site without touching the code base. As a Jamstack platform, a big part of our roadmap is understanding how we can build our own tools or provide integrations for tools that fit in with your development ecosystem and Pages, which is why in August this year we announced Pages support for Deploy Hooks.

Deploy Hooks are the key to what allows you to connect and trigger deployments in Pages via updates made in your headless CMS. As developers, instead of getting pinged several times a day to make content updates to your site, your marketing team can update the site directly within the headless CMS’s interface by way of a Deploy Hook. This is a URL created on Pages that accepts an HTTP POST request to trigger new deployments outside the realm of your git commands. You can configure settings within your CMS to accept the Deploy Hook so that anytime content is updated within your CMS, a new deployment is started in the Pages dashboard automatically — it couldn’t be any easier!

Within the Pages interface, there are two things you need to do to create your Deploy Hook:

1. Choose your Deploy Hook name: You can name your deploy hook anything you’d like

2. Select the Branch to Build: You can specify which branch will be built and deployed when the URL is requested with the Deploy Hook.

Once you are given your Deploy Hook, you’re all set to set up a webhook within your chosen CMS where you will paste your Deploy Hook.

That’s it! Now leave it to your marketing team to update their rich content and watch the builds trigger automatically to update your site!

Of course, Deploy Hooks is just a starting point of ways we can provide a better dev experience for your team when using the headless CMS of your choice with your Pages site. But our story of integrations does not stop here. Introducing our CMS partners and integrations: Sanity, Strapi, Contentful, and WordPress!

We continue to see the highest usage rates of these four CMSes on Pages among you and your teams, and in the months to come we’ll be working closely with our partners to build even more for you.

We’re delighted to partner with Cloudflare and excited by this new release from Cloudflare Pages. At Sanity, we care deeply about people working with content on our platform. Cloudflare’s new deploy hooks allow developers to automate builds for static sites based on content changes, which is a huge improvement for content creators. Combining these with structured content and our GROQ-powered Webhooks, our customers can be strategic about when these builds should happen. We’re stoked to be part of this release and can’t wait to see what the community will build with Sanity and Cloudflare!- Even Westvang, Co-founder, Sanity.io

Check out Sanity’s video tutorial on how to build your site using Pages and Sanity!

At Strapi, we’re excited about this partnership with Cloudflare because it enables developers to abstract away the complexity of deploying changes to production for content teams. By using the Deploy Hook for Strapi, everyone can push new content and updates quickly and autonomously.- Pierre Burgy, CEO, Strapi.io

With this integration, our customers can work more efficiently and cross-functionally with their teams. Marketing teams can update Contentful directly to automatically trigger deployments without relying on their developers to update any of their code base, which results in a better, more productive experience for all teams involved.- Jeff Blattel, Director of Technical Partnerships at Contentful

For now, to learn more about how you can connect your Pages project to one of our partner CMSes, check out our Deploy Hooks documentation to deploy your first project today!

In the midst of the hottest summer on record, Cloudflare held its first ever Impact Week. We announced a variety of products and initiatives that aim to make the Internet and our planet a better place, with a focus on environmental, social, and governance projects. Today, we’re excited to share an update on Crawler Hints, an initiative announced during Impact Week. Crawler Hints is a service that improves the operating efficiency of approximately 45% of the Internet traffic that comes from web crawlers and bots.

Crawler Hints achieves this efficiency improvement by ensuring that crawlers get information about what they’ve crawled previously and if it makes sense to crawl a website again.

Today we are excited to announce two updates for Crawler Hints:

1. The first: Crawler Hints now supports IndexNow, a new protocol that allows websites to notify search engines whenever content on their website content is created, updated, or deleted. By collaborating with Microsoft and Yandex, Cloudflare can help improve the efficiency of their search engine infrastructure, customer origin servers, and the Internet at large.

2. The second: Crawler Hints is now generally available to all Cloudflare customers for free. Customers can benefit from these more efficient crawls with a single button click. If you want to enable Crawler Hints, you can do so in the Cache Tab of the Dashboard.

Crawlers help make the Internet work. Crawlers are automated services that travel the Internet looking for… well, whatever they are programmed to look for. To power experiences that rely on indexing content from across the web, search engines and similar services operate massive networks of bots that crawl the Internet to identify the content most relevant to a user query. But because content on the web is always changing, and there is no central clearinghouse for when these changes happen on websites, search engine crawlers have a Sisyphean task. They must continuously wander the Internet, making guesses on how frequently they should check a given site for updates to its content.

Companies that run search engines have worked hard to make the process as efficient as possible, pushing the state-of-the-art for crawl cadence and infrastructure efficiency. But there remains one clear area of waste: excessive crawl.

At Cloudflare, we see traffic from all the major search crawlers, and have spent the last year studying how often these bots revisit a page that hasn't changed since they last saw it. Every one of these visits is a waste. And, unfortunately, our observation suggests that 53% of this crawler traffic is wasted.

With Crawler Hints, we expect to make this task a bit more tractable by providing an additional heuristic to the people who run these crawlers. This will allow them to know when content has been changed or added to a site instead of relying on preferences or previous changes that might not reflect the true change cadence for a site. Crawler Hints aims to increase the proportion of relevant crawls and limit crawls that don’t find fresh content, improving customer experience and reducing the need for repeated crawls.

Cloudflare sits in a unique position on the Internet to help give crawlers hints about when they should recrawl a site. Don’t knock on a website’s door every 30 seconds to see if anything is new when Cloudflare can proactively tell your crawler when it’s the best time to index new or changed content. That’s Crawler Hints in a nutshell!

If you want to learn more about Crawler Hints, see the original blog.

IndexNow is a standard that was written by Microsoft and Yandex search engines. The standard aims to provide an efficient manner of signaling to search engines and other crawlers for when they should crawl content. Cloudflare’s Crawler Hints now supports IndexNow.

​​In its simplest form, IndexNow is a simple ping so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results.- www.indexnow.org

By enabling Crawler Hints on your website, with the simple click of a button, Cloudflare will take care of signaling to these search engines when your content has changed via the IndexNow protocol. You don’t need to do anything else!  

What does this mean for search engine operators? With Crawler Hints you’ll receive a near real-time, pushed feed of change events of Cloudflare websites (that have opted in). This, in turn, will dramatically improve not just the quality of your results, but also the energy efficiency of running your bots.

Cloudflare is in a unique position to have a sizable portion of the Internet proxied behind us. As a result, we are able to see trends in the way bots access web resources. That visibility allows us to be proactive about signaling which crawls are required vs. not. We are excited to work with partners to make these insights useful to our customers. Search engines are key constituents in this equation. We are happy to collaborate and share this vision of a more efficient Internet with Microsoft Bing, and Yandex. We have been testing our interaction via IndexNow with Bing and Yandex for months with some early successes.  

This is just the beginning. Crawler Hints is a continuous process that will require working with more and more partners to improve Internet efficiency more generally. While this may take time and participation from other key parts of the industry, we are open to collaborate with any interested participant who relies on crawling to power user experiences.

“The cache data from CDNs is a really valuable signal for content freshness. Cloudflare, as one of the top CDNs, is key in the adoption of IndexNow to become an industry-wide standard with a large portion of the internet actually using it. Cloudflare has built a really easy 1-click button for their users to start using it right away. Cloudflare’s mission of helping build a better Internet resonates well with why I started IndexNow i.e. to build a more efficient and effective Search.”- Fabrice Canel, Principal Program Manager

“Yandex is excited to join IndexNow as part of our long-term focus on sustainability. We have been working with the Cloudflare team in early testing to incorporate their caching signals in our crawling mechanism via the IndexNow API. The results are great so far.”- Maxim Zagrebin, Head of Yandex Search

"DuckDuckGo is supportive of anything that makes search more environmentally friendly and better for end users without harming privacy. We're looking forward to working with Cloudflare on this proposal."- Gabriel Weinberg, CEO and Founder

Crawler Hints doesn’t just benefit search engines. For our customers and origin owners, Crawler Hints will ensure that search engines and other bot-powered experiences will always have the freshest version of your content, translating into happier users and ultimately influencing search rankings. Crawler Hints will also mean less traffic hitting your origin, improving resource consumption. Moreover, your site performance will be improved as well: your human customers will not be competing with bots!

And for Internet users? When you interact with bot-fed experiences — which we all do every day, whether we realize it or not, like search engines or pricing tools — these will now deliver more useful results from crawled data, because Cloudflare has signaled to the owners of the bots the moment they need to update their results.

Crawler Hints is free to use for all Cloudflare customers and promises to revolutionize web efficiency. If you’d like to see how Crawler Hints can benefit how your website is indexed by the worlds biggest search engines, please feel free to opt-into the service:

1. Sign in to your Cloudflare Account.

2. In the dashboard, navigate to the Cache tab.

3. Click on the Configuration section.

4. Locate the Crawler Hints sign up card and enable. It's that easy.

Once you’ve enabled it, we will begin sending hints to search engines about when they should crawl particular parts of your website. Crawler Hints holds tremendous promise to improve the efficiency of the Internet.

We’re thrilled to collaborate with industry leaders Microsoft Bing, and Yandex to bring IndexNow to Crawler Hints, and to bring Crawler Hints to a wide audience in general availability. We look forward to working with additional companies who run crawlers to help make this process more efficient for the whole Internet.

Today, we are excited to announce that Cloudflare customers can choose Microsoft Azure with a lower cost data transfer solution via the Microsoft Routing Preference service. Mutual customers can benefit from lower cost and predictable performance across our interconnected networks. Microsoft Azure has developed a seamless process to allow customers to choose this cost optimized routing solution.  We have customers using this new integration today and are excited to make this generally available to all our customers and prospects.

So how are we able to enable this great solution for our customers? The answer lies in our globally interconnected network.

Cloudflare is one of the most interconnected networks in the world, peering with over 9,500 networks globally, including major ISPs, cloud providers, and enterprises. We currently interconnect with Azure through private or public peering across all major regions — including private interconnections at key locations (see below).

Private Network Interconnects typically occur within the same facility through a fiber optic cable between routers for the two networks; peered connections occur at Internet exchanges offering high performance and availability. We are actively working on expanding on this interconnectivity between Azure and Cloudflare for our customers.

In addition to the private interconnections, we also have five Internet exchanges with private peering, and over 108 public peering links with Azure

Wondering what this really means? Let’s look at an example. Say an Internet visitor is in Sydney and requests content from an origin that's hosted in an Azure location in Chicago. When the visitor makes a request, Cloudflare automatically carries it to the Cloudflare data center in Sydney. The traffic is then routed over Cloudflare’s network all the way to Chicago where the origin is hosted on Azure. The request is then handed over to an Azure data center over our private interconnections.

On the way back (egress path), the request is handed over from Azure network to Cloudflare at the origin in Chicago via our private interconnection (without involving any ISP). Then it’s carried entirely over the Cloudflare network to Sydney and back to the visitor.

Customer choice. That’s an important ingredient to help build a better Internet for our customers — free of vendor lock-in, and with open Internet standards. We’ve worked with the Azure team to enable this interconnectivity, giving the customers the flexibility to choose multiple best-of-breed products without having to worry about high data transfer costs.

What is even more exciting is working with Microsoft, a company that shares our philosophy of promoting customer flexibility and helping customers resist vendor lock-in:

“Microsoft Azure is committed to offering services that make it easy to use offerings from industry leaders like Cloudflare - enabling choice to address customer’s business need."- Jeff Cohen, Partner Group Program Manager for Azure Networking.

Cloudflare customers now have the option to leverage Azure routing preference and as a result use both platforms for their respective features and services offering the most secure and performant solution.

Most importantly customers can avail of this lower cost solution with just three simple steps.

Step 1: Choose Internet routing on your Azure dashboard for origin in Azure storage:

Step 2: Enable Internet routing on your Firewall and virtual network tab:

Step 3: Enter your updated endpoint urls from Azure into your Cloudflare dashboard:

Once enabled, the discounting is automatic and ongoing from the next monthly bill. Further details on the discounted rates can be found in Azure’s Bandwidth pricing.

A number of customers are already enjoying these benefits:

“Enabling cost-optimized egress by Cloudflare and Azure via Routing Preference from the Azure dashboard has been very smooth for us with minimal effort. Cloudflare was proactive in reaching out with its customer-centric approach."- Joakim Jamte, Engineering Manager, Bannerflow

"Before taking advantage of the Routing Preference by Azure via Cloudflare, Egress fees were one of the key reasons that restricted us from having more multi-cloud solutions since it can be high and unpredictable at times as the traffic scales. Enabling Routing Preference on the Azure dashboard was quick and easy. It was a one-and-done effort and we get discounted Egress rates on every Azure bill.”- Darin MacRae, Chief Architect / Cloud Computing, MyRadar.com

"Along with Cloudflare's excellent security features and high performing CDN, the data transfer rates from Azure's Routing Preference enabled by Cloudflare make the offer very compelling. Enabling and receiving the discount was very easy and helped us optimize our investment without any effort.”- Arthur Roodenburg, CIO, Act-3D B.V.

We’re pleased today to offer this benefit to all Cloudflare customers. If you are interested in taking advantage of Routing Preference please reach out.

Cloudflare and Microsoft Azure Active Directory have partnered to provide an integration specifically for web applications using Azure Active Directory B2C. From today, customers using both services can follow the simple integration steps to protect B2C applications with Cloudflare’s Web Application Firewall (WAF) on any custom domain. Microsoft has detailed this integration as well.

The Web Application Firewall (WAF) is a core component of the Cloudflare platform and is designed to keep any web application safe. It blocks more than 70 billion cyber threats per day. That is 810,000 threats blocked every second.

The WAF is available through an intuitive dashboard or a Terraform integration, and it enables users to build powerful rules. Every request to the WAF is inspected against the rule engine and the threat intelligence built from protecting approximately 25 million internet properties. Suspicious requests can be blocked, challenged or logged as per the needs of the user, while legitimate requests are routed to the destination regardless of where the application lives (i.e., on-premise or in the cloud). Analytics and Cloudflare Logs enable users to view actionable metrics.

The Cloudflare WAF is an intelligent, integrated, and scalable solution to protect business-critical web applications from malicious attacks, with no changes to customers' existing infrastructure.

Azure AD B2C is a customer identity management service that enables custom control of how your customers sign up, sign in, and manage their profiles when using iOS, Android, .NET, single-page (SPA), and other applications and web experiences. It uses standards-based authentication protocols including OpenID Connect, OAuth 2.0, and SAML. You can customize the entire user experience with your brand so that it blends seamlessly with your web and mobile applications. It integrates with most modern applications and commercial off-the-shelf software, providing business-to-customer identity as a service. Customers of businesses of all sizes use their preferred social, enterprise, or local account identities to get single sign-on access to their applications and APIs. It takes care of the scaling and safety of the authentication platform, monitoring and automatically handling threats like denial-of-service, password spray, or brute force attacks.

When setting up Azure AD B2C, many customers prefer to customize their authentication endpoint by hosting the solution under their own domain — for example, under store.example.com — rather than using a Microsoft owned domain. With the new partnership and integration, customers can now place the custom domain behind Cloudflare’s Web Application Firewall while also using Azure AD B2C, further protecting the identity service from sophisticated attacks.

This defense-in-depth approach allows customers to leverage both Cloudflare WAF capabilities along with Azure AD B2C native Identity Protection features to defend against cyberattacks.

Instructions on how to set up the integration are provided on the Azure website and all it requires is a Cloudflare account.

Azure customers need support for a strong set of security and performance tools once they implement Azure AD B2C in their environment. Integrating Cloudflare Web Application Firewall with Azure AD B2C can provide customers the ability to write custom security rules (including rate limiting rules), DDoS mitigation, and deploy advanced bot management features. The Cloudflare WAF works by proxying and inspecting traffic towards your application and analyzing the payloads to ensure only non-malicious content reaches your origin servers. By incorporating the Cloudflare integration into Azure AD B2C, customers can ensure that their application is protected against sophisticated attack vectors including zero-day vulnerabilities, malicious automated botnets, and other generic attacks such as those listed in the OWASP Top 10.

This integration is a great match for any B2C businesses that are looking to enable their customers to authenticate themselves in the easiest and most secure way possible.

Please give it a try and let us know how we can improve it. Reach out to us for other use cases for your applications on Azure. Register here for expressing your interest/feedback on Azure integration and for upcoming webinars on this topic.

Cloudflare Workers is the easiest way for developers to deploy their application’s code with performance, scale and security baked in. No configuration necessary. Worker code scales to serve billions of requests close to your users across Cloudflare’s 200+ data centers.

But that’s not the only interesting problem we need to solve. Every application has two parts: code and state.

State isn’t always the easiest to work in a massive distributed system. When an application runs in 200+ data centers simultaneously, there’s an inherent tradeoff between distributing the data for better performance, availability, scale, and guaranteeing that all data centers see the same data at a given point in time.

Our goal is to make state at the edge seamless. We started that journey with Workers KV, which provides low-latency access to globally distributed data. We’re since added Durable Objects, with strong consistency and the ability to design coordination patterns on top of Workers. We’re continuing to invest in and build out these products.

However, some use cases aren’t easily implemented with Workers KV or Durable Objects. Think querying complex datasets, or communicating with an existing system-of-record. Even if we built this functionality ourselves, there will always be customers who want to use a different tool for data storage and access.

Today, we’re making our data strategy clear so that developers can build on Workers with confidence. First, we’re announcing a partnership with two distributed data companies, Macrometa and Fauna, to help developers choose an edge-first database when they build a new application on Workers. Next, we’ve introduced tutorials and guides for databases that already support HTTP connections to make connecting to existing databases even easier.

With this, the Workers runtime is the easiest way to handle both parts of your application that matter: code and state.

Cloudflare is excited to announce a partnership for global data with Macrometa. Macrometa provides a serverless data platform for developers to build rich, geo-distributed, stateful applications on the edge with Cloudflare Workers. The Macrometa Global Data Network (GDN) is an ultra-fast, globally replicated noSQL database with integrated features for search, pub/sub, streams, and stream processing.

Macrometa complements Cloudflare Workers by enabling stateful functions to run, store, query, mutate, and serve data globally with local read/write latencies. Tables replicate across Macrometa’s global data centers with configurable consistency and without additional code, cost, or complexity. Data calls between Workers and Macrometa are automatically routed and served from the closest Macrometa region relative to the Cloudflare Worker invocation. We have seen that end-to-end latency for a database request from a client to the edge and back is <75ms for 99th percentile.

Macrometa combines typically disparate data services and APIs (each with their own data model and copy of data) into a single, integrated, and intuitive data API (with a single, unified common data set and model). At a high level, one can think of Macrometa as combining a noSQL database together with pub/sub, streaming data and event processing along with search into a seamless, globally replicated, low latency data platform.

The following multimodal interfaces are available to the developer via a SQL-like query language:

• Key/value database with high write performance, low latencies, and Time to Live (TTL) support

• JSON document database with a SQL-like query language

• Graph database for storing and retrieving relationships between entities

• DynamoDB API compatible database

• Low latency edge cache for database and API results with global cache invalidation and Time to Live (TTL)

• Search

• Pub/Sub & queues with Kafka, Pulsar and MQTT message compatibility

• Streaming data and complex event processing - evaluate events, aggregate, enrich, filter, summarize, pattern match on stream data in real time.

Apps written using Cloudflare Workers consume Macrometa via a JavaScript SDK or REST API. Developers have a choice of using one of two SDKs tightly integrated with Cloudflare Workers:

• A Cloudflare Worker specific JavaScript SDK which provides the full set of capabilities mentioned above

• The Dynamo client SDK that enables developers to use Macrometa as a drop-in replacement AWS DynamoDB (or as an edge cache).

With Cloudflare and Macrometa, developers can now build and deliver powerful and compelling data-driven experiences in a way that the centralized clouds will never be able to. The world deserves a better cloud - an edge cloud.- Chetan Venkatesh, Co-founder/CEO, Macrometa

If you are interested in learning more about using Macrometa and Cloudflare Workers together, please register here.

To showcase the power and performance of Cloudflare and Macrometa together, we re-built AWS’s demo application — a complete E-Commerce backend that serves an online bookstore. eCommerce backends for large global sites are complex and require many different types of databases. These complex architectures are necessary to store, perform multi-table or multi-data store queries, and serve product catalogs, pricing, search, recommendations, order management, cart operations, fulfillment, etc.

The bookstore is a serverless app built as a Cloudflare Worker to serve a React single page application and uses Macrometa for storing and serving product/catalog data, orders, cart operations, etc. It uses streams and complex event processing to provide a real-time leaderboard of the top-selling books and uses the Macrometa search service to deliver near-instant search results. You can test drive the end-user experience here. Our tests show orders of magnitude latency improvement for a user vs. the same use case built on a legacy cloud provider.

To check out the source code, head over to GitHub.

Cloudflare is excited to announce a new partnership for transactional workloads with Fauna, a data API for modern applications. Together, Cloudflare and Fauna enable developers to build more powerful stateful applications that scale dynamically and maintain strong consistency without any additional configuration.

Fauna is a flexible, developer-friendly, global transactional database delivered as a secure, global, cloud API — no database operations required. With Fauna, developers can simplify code and ship faster by replacing their database infrastructure with a data API that combines the flexibility of relational and document-centric querying with the power of custom business logic, and the ease of GraphQL at the edge.

Fauna is a simple yet scalable backend for applications built on Cloudflare’s services such as Workers and Pages. By leveraging a web-native (HTTP-based) invocation model with support for modern security protocols, Fauna eliminates the connection limits introduced by traditional databases and can be integrated directly with serverless functions and applications running at the edge.

Fauna complements Cloudflare KV and Durable Objects by providing a global, queryable, strongly consistent persistence layer for the mission-critical data required to build modern web and mobile applications. It’s underlying globally distributed storage engine is fast, consistent, and reliable. Developers can rely on Fauna’s unique global transactional capabilities to ensure that mission-critical business data is always strongly consistent with minimal latency.

Fauna, a serverless offering, is easy to get started with for free, and lets developers experience freedom from database operations at any scale. Fauna is available through GraphQL or drivers that work with Workers in several popular programming languages.

If you’re interested in learning how to use Fauna with Cloudflare Workers, please visit this step-by-step tutorial for a hands-on experience! If you are interested in learning more about using Cloudflare Workers and Fauna together, please register here.

The integration of Cloudflare workers with Fauna allows our joint customers to simply and cost effectively bring both data and logic to the edge. Fauna was developed from the ground up as a globally distributed, serverless database and when combined with Cloudflare Workers serverless compute fabric provides developers with a simple, yet powerful abstraction over the complexities of distributed infrastructure.- Evan Weaver, Co-founder/CTO, Fauna

Let’s take a look at a current customer of Fauna and Cloudflare who benefits from Fauna’s ability to offer globally-distributed transactional data access at low latency.

MeetKai is an AI-powered voice assistant that makes a consumer’s life easier through conversation, personalization & curation. Deployed by enterprises with Monthly Active Users in the hundred million plus range, MeetKai uses Fauna and Cloudflare together to deliver a responsive, global experience.

Cloudflare Workers act as the proxy to all downstream requests. Doing so enables MeetKai to perform much smarter load balancing at the edge, using the Worker rather than some other competing solutions. When combined with Fauna, this serverless-first approach delivers scale and performance at predictable costs, while reducing time to market. By pushing data to the edge with Fauna, MeetKai are able to inject personalization information before a request ever hits the backend! Learn more about how MeetKai uses Cloudflare Workers and Fauna in this technical case study.

Combining Cloudflare Workers with Fauna allows us to bring both data and logic to the edge, enabling us to deliver a differentiated search engine while still meeting high operations requirements in latency and uptime -- all at a fraction of the cost and complexity of non-serverless solutions.- James Kaplan, Founder/CEO MeetKai

In addition to our partners, Cloudflare Workers can work with any database that accepts connections over HTTP.

In particular, we’ve documented connections to both DynamoDB and AWS Aurora, supporting either Postgres or MySQL. These connections let you access already existing databases directly from Cloudflare’s edge, making it easy to build new applications on top of existing data.

Connect a Worker to DynamoDB

Connect a Worker to AWS Aurora

In addition, we’ll be adding more tutorials to document how to connect to other databases that work with Workers, like Google’s Cloud Firestore.

The Workers platform has built a new paradigm for developers for serverless. Across our own Durable Objects and Workers KV, our partner databases, and connections to existing providers, we’re bringing that same paradigm shift to state at the edge. As we continue to build out our database partnerships and data platform, we want to hear from you! Send your ideas and feedback to the Workers Product team to let us know what databases and features you’d most like to see us build.