Automated security level: Cloudflare’s Security Level setting has been improved and no longer requires manual configuration. By integrating botnet data along with other request rate signals, all customers are protected from confirmed known malicious botnet traffic without any action required.

Cipher suite selection: You now have greater control over encryption settings via the Cloudflare dashboard, including specific cipher suite selection based on our client or compliance requirements.

Improved URL scanner: New features include bulk scanning, similarity search, location picker and more.

These updates are designed to give you more power and flexibility when managing online security, from proactive threat detection to granular control over encryption settings.

Cloudflare’s Security Level feature was designed to protect customer websites from malicious activity.

Available to all Cloudflare customers, including the free tier, it has always had very simple logic: if a connecting client IP address has shown malicious behavior across our network, issue a managed challenge. The system tracks malicious behavior by assigning a threat score to each IP address. The more bad behavior is observed, the higher the score. Cloudflare customers could configure the threshold that would trigger the challenge.

We are now announcing an update to how Security Level works, by combining the IP address threat signal with threshold and botnet data. The resulting detection improvements have allowed us to automate the configuration, no longer requiring customers to set a threshold.

The Security Level setting is now Always protected in the dashboard, and ip_threat_score fields in WAF Custom Rules will no longer be populated. No change is required by Cloudflare customers. The “I am under attack” option remains unchanged.

Although we always favor simplicity, privacy-related services, including our own WARP, have seen growing use. Meanwhile, carrier-grade network address translation (CGNATs) and outbound forward proxies have been widely used for many years.

These services often result in multiple users sharing the same IP address, which can lead to legitimate users being challenged unfairly since individual addresses don’t strictly correlate with unique client behavior. Moreover, threat actors have become increasingly adept at anonymizing and dynamically changing their IP addresses using tools like VPNs, proxies, and botnets, further diminishing the reliability of IP addresses as a standalone indicator of malicious activity. Recognising these limitations, it was time for us to revisit Security Level’s logic to reduce the number of false positives being observed.

In February 2024, we introduced a new security system that automatically combines the real-time DDoS score with a traffic threshold and a botnet tracking system. The real-time DDoS score is part of our autonomous DDoS detection system, which analyzes traffic patterns to identify potential threats. This system superseded and replaced the existing Security Level logic, and is deployed on all customer traffic, including free plans. After thorough monitoring and analysis over the past year, we have confirmed that these behavior-based mitigation systems provide more accurate results. Notably, we've observed a significant reduction in false positives, demonstrating the limitations of the previous IP address-only logic.

Our new logic combines IP address signals with behavioral and threshold indicators to improve the accuracy of botnet detection. While IP addresses alone can be unreliable due to potential false positives, we enhance their utility by integrating them with additional signals. We monitor surges in traffic from known "bad" IP addresses and further refine this data by examining specific properties such as path, accept, and host headers.

We also introduced a new botnet tracking system that continuously detects and tracks botnet activity across the Cloudflare network. From our unique vantage point as a reverse proxy for nearly 20% of all websites, we maintain a dynamic database of IP addresses associated with botnet activity. This database is continuously updated, enabling us to automatically respond to emerging threats without manual intervention. This effect is visible in the Cloudflare Radar chart below, as we saw sharp growth in DDoS mitigations in February 2024 as the botnet tracking system was implemented.

Customers now get better protection while having to manage fewer configurations, and they can rest assured that their online presence remains fully protected. These security measures are integrated and enabled by default across all of our plans, ensuring protection without the need for manual configuration or rule management. This improvement is particularly beneficial for users accessing sites through proxy services or CGNATs, as these setups can sometimes trigger unnecessary security checks, potentially disrupting access to websites.

Our team is looking at defining the next generation of threat scoring mechanisms. This initiative aims to provide our customers with more relevant and effective controls and tools to combat today's and tomorrow's potential security threats.

Effective March 17, 2025, we are removing the option to configure manual rules using the threat score parameter in the Cloudflare dashboard. The "I'm Under Attack" mode remains available, allowing users to issue managed challenges to all traffic when needed.

By the end of Q1 2026, we anticipate disabling all rules that rely on IP threat score. This means that using the threat score parameter in the Rulesets API and via Terraform won’t be available after the end of the transition period. However, we encourage customers to be proactive and edit or remove the rules containing the threat score parameter starting today.

Building upon our core security features, we're also giving you more control over your encryption: cipher suite selection is now available in the Cloudflare dashboard! 

When a client initiates a visit to a Cloudflare-protected website, a TLS handshake occurs, where clients present a list of supported cipher suites — cryptographic algorithms crucial for secure connections. While newer algorithms enhance security, balancing this with broad compatibility is key, as some customers prioritise reach by supporting older devices, even with less secure ciphers. To accommodate varied client needs, Cloudflare's default settings emphasise wide compatibility, allowing customers to tailor cipher suite selection based on their priorities: strong security, compliance (PCI DSS, FIPS 140-2), or legacy device support.

Previously, customizing cipher suites required multiple API calls, proving cumbersome for many users. Now, Cloudflare introduces Cipher Suite Selection to the dashboard. This feature introduces user-friendly selection flows like security recommendations, compliance presets, and custom selections.  

Cipher suites are collections of cryptographic algorithms used for key exchange, authentication, encryption, and message integrity, essential for a TLS handshake. During the handshake’s initiation, the client sends a "client hello" message containing a list of supported cipher suites. The server responds with a "server hello" message, choosing a cipher suite from the client's list based on security and compatibility. This chosen cipher suite forms the basis of TLS termination and plays a crucial role in establishing a secure HTTPS connection. Here’s a quick overview of each component:

• Key exchange algorithm: Secures the exchange of encryption keys between parties.

• Authentication algorithm: Verifies the identities of the communicating parties.

• Encryption algorithm: Ensures the confidentiality of the data.

• Message integrity algorithm: Confirms that the data remains unaltered during transmission.

Perfect forward secrecy is an important feature of modern cipher suites. It ensures that each session's encryption keys are generated independently, which means that even if a server’s private key is compromised in the future, past communications remain secure.

You can find cipher suite configuration under Edge Certificates in your zone’s SSL/TLS dashboard. There, you will be able to view your allow-listed set of cipher suites.

Additionally, you will be able to choose from three different user flows, depending on your specific use case, to seamlessly select your appropriate list. Those three user flows are: security recommendation selection, compliance selection, or custom selection. The goal of the user flows is to outfit customers with cipher suites that match their goals and priorities, whether those are maximum compatibility or best possible security.

1. Security recommendations 

To streamline the process, we have turned our cipher suites recommendations into selectable options. This is in an effort to expose our customers to cipher suites in a tangible way and enable them to choose between different security configurations and compatibility. Here is what they mean:

• Modern: Provides the highest level of security and performance with support for Perfect Forward Secrecy and Authenticated Encryption (AEAD). Ideal for customers who prioritize top-notch security and performance, such as financial institutions, healthcare providers, or government agencies. This selection requires TLS 1.3 to be enabled and the minimum TLS version set to 1.2.

• Compatible: Balances security and compatibility by offering forward-secret cipher suites that are broadly compatible with older systems. Suitable for most customers who need a good balance between security and reach. This selection also requires TLS 1.3 to be enabled and the minimum TLS version set to 1.2.

• Legacy: Optimizes for the widest reach, supporting a wide range of legacy devices and systems. Best for customers who do not handle sensitive data and need to accommodate a variety of visitors. This option is ideal for blogs or organizations that rely on older systems.

2. Compliance selection

Additionally, we have also turned our compliance recommendations into selectable options to make it easier for our customers to meet their PCI DSS or FIPS-140-2 requirements.

• PCI DSS Compliance: Ensures that your cipher suite selection aligns with PCI DSS standards for protecting cardholder data. This option will enforce a requirement to set a minimum TLS version of 1.2, and TLS 1.3 to be enabled, to maintain compliance.

  • Since the list of supported cipher suites require TLS 1.3 to be enabled and a minimum TLS version of 1.2 in order to be compliant, we will disable compliance selection until the zone settings are updated to meet those requirements. This effort is to ensure that our customers are truly compliant and have the proper zone settings to be so. 

• FIPS 140-2 Compliance: Tailored for customers needing to meet federal security standards for cryptographic modules. Ensures that your encryption practices comply with FIPS 140-2 requirements.

3. Custom selection 

For customers needing precise control, the custom selection flow allows individual cipher suite selection, excluding TLS 1.3 suites which are automatically enabled with TLS 1.3. To prevent disruptions, guardrails ensure compatibility by validating that the minimum TLS version aligns with the selected cipher suites and that the SSL/TLS certificate is compatible (e.g., RSA certificates require RSA cipher suites).

The API will still be available to our customers. This aims to support an existing framework, especially to customers who are already API reliant. Additionally, Cloudflare preserves the specified cipher suites in the order they are set via the API and that control of ordering will remain unique to our API offering. 

With your Advanced Certificate Manager or Cloudflare for SaaS subscription, head to Edge Certificates in your zone’s SSL dashboard and give it a try today!

Cloudflare's URL Scanner is a tool designed to detect and analyze potential security threats like phishing and malware by scanning and evaluating websites, providing detailed insights into their safety and technology usage. We've leveraged our own URL Scanner to enhance our internal Trust & Safety efforts, automating the detection and mitigation of some forms of abuse on our platform. This has not only strengthened our own security posture, but has also directly influenced the development of the new features we're announcing today. 

Phishing attacks are on the rise across the Internet, and we saw a major opportunity to be "customer zero" for our URL Scanner to address abuse on our own network. By working closely with our Trust & Safety team to understand how the URL Scanner could better identify potential phishing attempts, we've improved the speed and accuracy of our response to abuse reports, making the Internet safer for everyone. Today, we're excited to share the new API version and the latest updates to URL Scanner, which include the ability to scan from specific geographic locations, bulk scanning, search by Indicators of Compromise (IOCs), improved UI and information display, comprehensive IOC listings, advanced sorting options, and more. These features are the result of our own experiences in leveraging URL Scanner to safeguard our platform and our customers, and we're confident that they will prove useful to our security analysts and threat intelligence users.

Cloudflare Enterprise customers can now conduct routine scans of their web assets to identify emerging vulnerabilities, ensuring that potential threats are addressed proactively, by using the Bulk Scanning API endpoint. Another use case for the bulk scanning functionality is developers leveraging bulk scanning to verify that all URLs your team is accessing are secure and free from potential exploits before launching new websites or updates.

Scanning of multiple URLs addresses the specific needs of our users engaged in threat hunting. Many of them maintain extensive lists of URLs that require swift investigation to identify potential threats. Currently, they face the task of submitting these URLs one by one, which not only slows down their workflow but also increases the manual effort involved in their security processes. With the introduction of bulk submission capabilities, users can now submit up to 100 URLs at a time for scanning. 

Let’s look at a regular workflow:

In this workflow, when the user submits a new scan, we create a Durable Object with the same ID as the scan, save the scan options, like the URL to scan, to the Durable Objects’s storage and schedule an alarm for a few seconds later. This allows us to respond immediately to the user, signalling a successful submission. A few seconds later the alarm triggers, and we start the scan itself. 

However, with bulk scanning, the process is slightly different:

In this case, there are no Durable Objects involved just yet; the system simply sends each URL in the bulk scan submission as a new message to the queue.

Notice that in both of these cases the scan is triggered asynchronously. In the first case, it starts when the Durable Objects alarm fires and, in the second case, when messages in the queue are consumed. While the durable object alarm will always fire in a few seconds, messages in the queue have no predetermined processing time, they may be processed seconds to minutes later, depending on how many messages are already in the queue and how fast the system processes them.

When users bulk scan, having the scan done at some point in time is more important than having it done now. When using the regular scan workflow, users are limited in the number of scans per minute they can submit. However, when using bulk scan this is not a concern, and users can simply send all URLs they want to process in a single HTTP request. This comes with the tradeoff that scans may take longer to complete, which is a perfect fit for Cloudflare Queues. Having the ability to configure retries, max batch size, max batch timeouts, and max concurrency is something we’ve found very useful. As the scans are completed asynchronously, users can request the resulting scan reports via the API.

The Related Scans feature allows API, Cloudflare dashboard and Radar users alike to view related scans directly within the URL Scanner Report. This helps users analyze and understand the context of a scanned URL by providing insights into similar URLs based on various attributes. Filter and search through URL Scanner reports to retrieve information on related scans, including those with identical favicons, similar HTML structures, and matching IP addresses.

The Related Scans tab presents a table with key headers corresponding to four distinct filters. Each entry includes the scanned URL and a direct link to view the detailed scan report, allowing for quick access to further information. 

We've introduced the ability to search by indicators of compromise (IOCs), such as IP addresses and hashes, directly within the user interface. Additionally, we've added advanced filtering options by various criteria, including screenshots, hashes, favicons, and HTML body content. This allows for more efficient organization and prioritization of URLs based on specific needs. While attackers often make minor modifications to the HTML structure of phishing pages to evade detection, our advanced filtering options enable users to search for URLs with similar HTML content. This means that even if the visual appearance of a phishing page changes slightly, we can still identify connections to known phishing campaigns by comparing the underlying HTML structure. This proactive approach helps users identify and block these threats effectively.

Another use case for the advanced filtering options is the search by hash; a user who has identified a malicious JavaScript file through a previous investigation can now search using the file's hash. By clicking on an HTTP transaction, you'll find a direct link to the relevant hash, immediately allowing you to pivot your investigation. The real benefit comes from identifying other potentially malicious sites that have that same hash. This means that if you know a given script is bad, you can quickly uncover other compromised websites delivering the same malware.

The user interface has also undergone significant improvements to enhance the overall experience. Other key updates include:

• Page title and favicon surfaced, providing immediate visual context

• Detailed summaries are now available

• Redirect chains allow users to understand the navigation path of a URL

• The ability to scan files from URLs that trigger an automatic file download

With the latest updates to our URL Scanner, users can now download both the HAR (HTTP Archive) file and the JSON report from their scans. The HAR file provides a detailed record of all interactions between the web browser and the scanned website, capturing crucial data such as request and response headers, timings, and status codes. This format is widely recognized in the industry and can be easily analyzed using various tools, making it invaluable for developers and security analysts alike.

For instance, a threat intelligence analyst investigating a suspicious URL can download the HAR file to examine the network requests made during the scan. By analyzing this data, they can identify potential malicious behavior, such as unexpected redirects and correlate these findings with other threat intelligence sources. Meanwhile, the JSON report offers a structured overview of the scan results, including security verdicts and associated IOCs, which can be integrated into broader security workflows or automated systems.

Finally, we’re announcing a new version of our API, allowing users to transition effortlessly to our service without needing to overhaul their existing workflows. Moving forward, any future features will be integrated into this updated API version, ensuring that users have access to the latest advancements in our URL scanning technology.

We understand that many organizations rely on automation and integrations with our previous API version. Therefore, we want to reassure our customers that there will be no immediate deprecation of the old API. Users can continue to use the existing API without disruption, giving them the flexibility to migrate at their own pace. We invite you to try the new API today and explore these new features to help with your web security efforts.

In summary, these updates to Security Level, cipher suite selection, and URL Scanner help us provide comprehensive, accessible, and proactive security solutions. Whether you're looking for automated protection, granular control over your encryption, or advanced threat detection capabilities, these new features are designed to empower you to build a safer and more secure online presence. We encourage you to explore these features in your Cloudflare dashboard and discover how they can benefit your specific needs.

We’ll continue to share roundup blog posts as we build and innovate. Follow along on the Cloudflare Blog for the latest news and updates. 

Today, we’re excited to talk about URL Scanner, a tool that helps everyone from security teams to everyday users to detect and safeguard against malicious websites by scanning and analyzing them. URL Scanner has executed almost a million scans since its launch last March on Cloudflare Radar, driving us to continuously innovate and enhance its capabilities. Since that time, we have introduced unlisted scans, detailed malicious verdicts, enriched search functionality, and now, integration with Security Center and an official API, all built upon the robust foundation of Cloudflare Workers, Durable Objects, and the Browser Rendering API.

Security Center is the single place in the Cloudflare Dashboard to map your attack surface, identify potential security risks, and mitigate risks with a few clicks. Its users can now access the URL scanner directly from the Investigate Portal, enhancing their cybersecurity workflow. These scans will be unlisted by default, ensuring privacy while facilitating a deep dive into website security. Users will be able to see their historic scans and access the related reports when they need to, and they will benefit from automatic screenshots for multiple screen sizes, enriching the context of each scan.

Customers with Cloudflare dashboard access will enjoy higher API limits and faster response times, crucial for agile security operations. Integration with internal workflows becomes seamless, allowing for sophisticated network and user protection strategies.

Security Center in the Cloudflare Dashboard

The URL Scanner API is a powerful asset for developers, enabling custom scans to detect phishing or malware risks, analyze website technologies, and much more. With new features like custom HTTP headers and multi-device screenshots, developers gain a comprehensive toolkit for thorough website assessment.

Using the API, here’s the simplest way to submit a scan request:

curl --request POST \
	--url https://api.cloudflare.com/client/v4/accounts/<accountId>/urlscanner/scan \
	--header 'Content-Type: application/json' \
--header "Authorization: Bearer <API_TOKEN>" \
	--data '{
		"url": "https://www.cloudflare.com",
	}'

New features include the option to set custom HTTP headers, like User-Agent and Authorization, request multiple target device screenshots, like mobile and desktop, as well as set the visibility level to “unlisted”. This essentially marks the scan as private and was often requested by developers who wanted to keep their investigations confidential. Public scans, on the other hand, can be found by anyone through search and are useful to share results with the wider community. You can find more details in our developer documentation.

Scan results for www.cloudflare.com on Cloudflare Radar

Once a scan concludes, fetch the final report and the full network log. Recently added features include the `verdict` property, indicating the site’s malicious status, and the `securityViolations` section detailing CSP or SRI policy breaches — as a developer, you can also scan your own website and see our recommendations. Expect improvements on verdict accuracy over time, as this is an area we’re focusing on.

Developers can now search scans by hostname, a specific URL or even any URL the page connected to during the scan. This allows, for example, to search for websites that use a JavaScript library named jquery.min.js (‘?path=jquery.min.js’). Future plans include additional features like searching by IP address, ASN, and malicious website categorisation.

The URL Scanner can be used for a diverse range of applications. These include capturing a website's evolving state over time (such as tracking changes to the front page of an online newspaper), analyzing technologies employed by a website, preemptively assessing potential risks (as when scrutinizing shortened URLs), and supporting the investigation of persistent cybersecurity threats (such as identifying affected websites hosting a malicious JavaScript file).

In recounting the process of developing the URL Scanner, we aim to showcase the potential and versatility of Cloudflare Workers as a platform. This story is more than a technical journey, but a testament to the capabilities inherent in our platform's suite of APIs. By dogfooding our own technology, we not only demonstrate confidence in its robustness but also encourage developers to harness the same capabilities for building sophisticated applications. The URL Scanner exemplifies how Cloudflare Workers, Durable Objects, and the Browser Rendering API seamlessly integrate.

High level overview of the Cloudflare URL Scanner technology stack

As seen above, Cloudflare’s runtime infrastructure is the foundation the system runs on. Cloudflare Workers serves the public API, Durable Objects handles orchestration, R2 acts as the primary storage solution, and Queues efficiently handles batch operations, all at the edge. However, what truly enables the URL Scanner’s capabilities is the Browser Rendering API. It’s what initially allowed us to release in such a short time frame, since we didn’t have to build and manage an entire fleet of Chrome browsers from scratch. We simply request a browser, and then using the well known Puppeteer library, instruct it to fetch the webpage and process it in the way we want. This API is at the heart of the entire system.

The entire process of scanning a website, can be split into 4 phases:

1. Queue a scan

2. Browse to the website and compile initial report

3. Post-process: compile additional information and build final report

4. Store final report, ready for serving and searching

In short, we create a Durable Object, the Scanner, unique to each scan, which is responsible for orchestrating the scan from start to finish. Since we want to respond immediately to the user, we save the scan to the Durable Object’s transactional Key-Value storage, and schedule an alarm so we can perform the scan asynchronously a second later.  We then respond to the user, informing them that the scan request was accepted.

When the Scanner’s alarm triggers, we enter the second phase:

There are 3 components at work in this phase, the Scanner, the Browser Pool and the Browser Controller, all Durable Objects.

In the initial release, for each new scan we would launch a brand-new browser. However, This operation would take time and was inefficient, so after review, we decided to reuse browsers across multiple scans. This is why we introduced both the Browser Pool and the Browser Controller components. The Browser Pool keeps track of what browsers we have open, when they last pinged the browser pool (so it knows they’re alive), and whether they’re free to accept a new scan. The Browser Controller is responsible for keeping the browser instance alive, once it’s launched, and orchestrating (ahem, puppeteering) the entire browsing session. Here’s a simplified version of our Browser Controller code:

export class BrowserController implements DurableObject {
	//[..]
	private async handleNewScan(url: string) {
		if (!this.browser) {
			// Launch browser: 1st request to durable object
			this.browser = await puppeteer.launch(this.env.BROWSER)
			await this.state.storage.setAlarm(Date.now() + 5 * 1000)
		}
		// Open new page and navigate to url
		const page = await this.browser.newPage()
		await page.goto(url, { waitUntil: 'networkidle2', timeout: 5000, })

		// Capture DOM
		const dom = await page.content()

		// Clean up
		await page.close()

		return {
			dom: dom,
		}
	}

	async alarm() {
		if (!this.browser) {
			return
		}
		await this.browser.version() // stop websocket connection to Chrome from going idle
		
		// ping browser pool, let it know we're alive
		
		// Keep durable object alive
		await this.state.storage.setAlarm(Date.now() + 5 * 1000)
	}
}

Launching a browser (Step 6) and maintaining a connection to it is abstracted away from us thanks to the Browser Rendering API. This API is responsible for all the infrastructure required to maintain a fleet of Chrome browsers, and led to a much quicker development and release of the URL Scanner. It also allowed us to use a well-known library, Puppeteer, to communicate with Google Chrome via the DevTools protocol.

The initial report is made up of the network log of all requests, captured in HAR (HTTP Archive) format. HAR files, essentially JSON files, provide a detailed record of all interactions between a web browser and a website. As an established standard in the industry, HAR files can be easily shared and analyzed using specialized tools. In addition to this network log, we augment our dataset with an array of other metadata, including base64-encoded screenshots which provide a snapshot of the website at the moment of the scan.

Having this data, we transition to phase 3, where the Scanner Durable Object initiates a series of interactions with a few other Cloudflare APIs in order to collect additional information, like running a phishing scanner over the web page's Document Object Model (DOM), fetching DNS records, and extracting information about categories and Radar rank associated with the main hostname.

This process ensures that the final report is enriched with insights coming from different sources, making the URL Scanner more efficient in assessing websites. Once all the necessary information is collected, we compile the final report and store it as a JSON file within R2, Cloudflare’s object storage solution. To empower users with efficient scan searches, we use Postgres.

While the initial approach involved sending each completed scan promptly to the core API for immediate storage in Postgres, we realized that, as the rate of scans grew, a more efficient strategy would be to batch those operations, and for that, we use Worker Queues:

This allows us to better manage the write load on Postgres. We wanted scans available as soon as possible to those who requested them, but it’s ok if they’re only available in search results at a slightly later point in time (seconds to minutes, depending on load).

In short, Durable Objects together with the Browser Rendering API power the entire scanning process. Once that’s finished, the Cloudflare Worker serving the API will simply fetch it from R2 by ID. All together, Workers, Durable Objects, and R2 scale seamlessly and will allow us to grow as demand evolves.

While we've extensively covered the URL scanning workflow, we've yet to delve into the construction of the API worker itself. Developed with Typescript, it uses itty-router-openapi, a Javascript router with Open API 3 schema generation and validation, originally built for Radar, but that’s been improving ever since with contributions from the community. Here’s a quick example of how to set up an endpoint, with input validation built in:

import { DateOnly, OpenAPIRoute, Path, Str, OpenAPIRouter } from '@cloudflare/itty-router-openapi'

import { z } from 'zod'
import { OpenAPIRoute, OpenAPIRouter, Uuid } from '@cloudflare/itty-router-openapi'

export class ScanMetadataCreate extends OpenAPIRoute {
  static schema = {
    tags: ['Scans'],
    summary: 'Create Scan metadata',
    requestBody: {
      scan_id: Uuid,
      url: z.string().url(),
      destination_ip: z.string().ip(),
      timestamp: z.string().datetime(),
      console_logs: [z.string()],
    },
  }

  async handle(
    request: Request,
    env: any,
    context: any,
    data: any,
  ) {
    // Retrieve validated scan
    const newScanMetadata = data.body

    // Insert the scan

    // Return scan as json
    return newScanMetadata
  }
}


const router = OpenAPIRouter()
router.post('/scan/metadata/', ScanMetadataCreate)

// 404 for everything else
router.all('*', () => new Response('Not Found.', { status: 404 }))

export default {
  fetch: router.handle,
}

In the example above, the ScanMetadataCreate endpoint will make sure to validate the incoming POST data to match the defined schema before calling the ‘async handle(request,env,context,data)’ function. This way you can be sure that if your code is called, the data argument will always be validated and formatted.

You can learn more about the project on its GitHub page.

Looking ahead, we're committed to further elevating the URL Scanner's capabilities. Key upcoming features include geographic scans, where users can customize the location that the scan is done from, providing critical insights into regional security threats and content compliance; expanded scan details, including more comprehensive headers and security details; and continuous performance improvements and optimisations, so we can deliver faster scan results.

The evolution of the URL Scanner is a reflection of our commitment to Internet safety and innovation. Whether you're a developer, a security professional, or simply invested in the safety of the digital landscape, the URL Scanner API offers a comprehensive suite of tools to enhance your efforts. Explore the new features today, and join us in shaping a safer Internet for everyone.

Remember, while Security Center's new capabilities offer advanced tools for URL Scanning for Cloudflare’s existing customers, the URL Scanner remains accessible for basic scans to the public on Cloudflare Radar, ensuring our technology benefits a broad audience.

If you’re considering a new career direction, check out our open positions. We’re looking for individuals who want to help make the Internet better; learn more about our mission here.

Radar 2.0 was built on the learnings of Radar 1.0 and was launched last month during Cloudflare's Birthday Week as a complete product revamp. We wanted to make it easier for our users to find insights and navigate our data, and overall provide a better and faster user experience.

We're building a Supercloud. Cloudflare's products now include hundreds of features in networking, security, access controls, computing, storage, and more.

This blog will explain how we built the new Radar from an engineering perspective. We wanted to do this to demonstrate that anyone could build a somewhat complex website that involves demanding requirements and multiple architectural layers, do it on top of our stack, and how easy it can be.

Hopefully, this will inspire other developers to switch from traditional software architectures and build their applications using modern, more efficient technologies.

The following diagram is a birds-eye view of the Radar 2.0 architecture. As you can see, it's divided into three main layers:

• The Core layer is where we keep our data lake, data exploration tools, and backend API.

• The Cloudflare network layer is where we host and run Radar and serve the public APIs.

• The Client layer is essentially everything else that runs in your browser. We call it the Radar Web app.

As you can see, there are Cloudflare products everywhere. They provide the foundational resources to host and securely run our code at scale, but also other building blocks necessary to run the application end to end.

By having these features readily available and tightly integrated into our ecosystem and tools, at the distance of a click and a few lines of code, engineering teams don't have to reinvent the wheel constantly and can use their time on what is essential: their app logic.

Let's dig in.

Radar 2.0 is deployed using Cloudflare Pages, our developer-focused website hosting platform. In the early days, you could only host static assets on Pages, which was helpful for many use cases, including integrating with static site generators like Hugo, Jekyll, or Gatsby. Still, it wouldn't solve situations where your application needs some sort of server-side computing or advanced logic using a single deployment.

Luckily Pages recently added support to run custom Workers scripts. With Functions, you can now run server-side code and enable any kind of dynamic functionality you'd typically implement using a separate Worker.

Cloudflare Pages Functions also allow you to use Durable Objects, KV, R2, or D1, just like a regular Worker would. We provide excellent documentation on how to do this and more in our Developer Documentation. Furthermore, the team wrote a blog on how to build a full-stack application that describes all the steps in detail.

Radar 2.0 needs server-side functions for two reasons:

• To render Radar and run the server side of Remix.

• To implement and serve our frontend API.

We use Remix with Cloudflare Pages on Radar 2.0.

Remix follows a server/client model and works under the premise that you can't control the user's network, so web apps must reduce the amount of Javascript, CSS, and JSON they send through the wire. To do this, they move some of the logic to the server.

In this case, the client browser will get pre-rendered DOM components and the result of pre-fetched API calls with just the right amount of JSON, Javascript, and CSS code, rightfully adjusted to the UI needs. Here’s the technical explanation with more detail.

Typically, Remix would need a Node.js server to do all of this, but guess what: It can also run on Cloudflare Workers and Pages.

Here’s the code to get the Remix server running on Workers, using Cloudflare Pages:

import { createPagesFunctionHandler } from "@remix-run/cloudflare-pages";
import * as build from "@remix-run/dev/server-build";

const handleRequest = createPagesFunctionHandler({
  build: {
    ...build,
    publicPath: "/build/",
    assetsBuildDirectory: "public/build",
  },
  mode: process.env.NODE_ENV,
  getLoadContext: (context) => ({
    ...context.env,
    CF: (context.request as any).cf as IncomingRequestCfProperties | undefined,
  }),
});

const handler: ExportedHandler<Env> = {
  fetch: async (req, env, ctx) => {
    const r = new Request(req);
    return handleRequest({
      env,
      params: {},
      request: r,
      waitUntil: ctx.waitUntil,
      next: () => {
        throw new Error("next() called in Worker");
      },
      functionPath: "",
      data: undefined,
    });
  },
};

In Remix, routes handle changes when a user interacts with the app and changes it (clicking on a menu option, for example). A Remix route can have a loader, an action and a default export. The loader handles API calls for fetching data (GET method). The action handles submissions to the server (POST, PUT, PATCH, DELETE methods) and returns the response. The default export handles the UI code in React that’s returned for that route. A route without a default export returns only data.

Because Remix runs both on the server and the client, it can get smart and know what can be pre-fetched and computed server-side and what must go through the network connection, optimizing everything for performance and responsiveness.

Here’s an example of a Radar route, simplified for readability, for the Outage Center page.

import type { MetaFunction } from "@remix-run/cloudflare";
import { useLoaderData } from "@remix-run/react";
import { type LoaderArgs } from "@remix-run/server-runtime";

export async function loader(args: LoaderArgs) {
  const ssr = await initialFetch(SSR_CHARTS, args);
  return { ssr, };
}

export default function Outages() {
  const { ssr } = useLoaderData<typeof loader>();

  return (
    <Page
      filters={["timerange"]}
      title={
        <>
          <Svg use="icon-outages" />
          {t("nav.main.outage-center")}
        </>
      }
    >
      <Grid columns={[1, 1, 1, 1]}>
        <Card.Article colspan={[1, 1, 1, 1]} rowspan={[1, 1, 1, 1]}>
          <Card.Section>
            <Components.InternetOutagesChoropleth ssr={ssr} />
          </Card.Section>
          <Divider />
          <Card.Section>
            <Components.InternetOutagesTable ssr={ssr} />
          </Card.Section>
        </Card.Article>
      </Grid>
    </Page>
  );
}

And here’s what it produces:

Remix and SSR can also help you with your Lighthouse scores and SEO. It can drastically improve metrics like Cumulative Layout Shift, First Contentful Paint and Largest Contentful Paint by reducing the number of fetches and information traveling from the server to the browser and pre-rendering the DOM.

Another project porting their app to Remix is Cloudflare TV. This is how their metrics looked before and after the changes.

Radar’s Desktop Lighthouse score is now nearly 100% on Performance, Accessibility, Best Practices, and SEO.

Another Cloudflare product that we use extensively on Radar 2.0 is Speed. In particular, we want to mention the Early Hints feature. Early Hints is a new web standard that defines a new HTTP 103 header the server can use to inform the browser which assets will likely be needed to render the web page while it's still being requested, resulting in dramatic load times improvements.

You can use Cloudflare Pages with Early Hints.

Radar has two APIs. The backend which has direct access to our data sources, and the frontend, which is available on the Internet.

The backend API was written using Python, Pandas and FastAPI and is protected by Cloudflare Access, JWT tokens and an authenticated origin pull (AOP) configuration. Using Python allows anyone on the team, engineers or data scientists, to collaborate easily and contribute to improving and expanding the API, which is great. Our data science team uses JupyterHub and Jupyter Notebooks as part of their data exploration workflows, which makes prototyping and reusing code, algorithms and models particularly easy and fast.

It then talks to the upstream frontend API via a Strawberry based GraphQL server. Using GraphQL makes it easy to create complex queries, giving internal users and analysts the flexibility they need when building reports from our vast collection of data.

We built Radar's frontend API on top of Cloudflare Workers. This worker has two main functions:

• It fetches data from the backend API using GraphQL, and then transforms it.

• It provides a public REST API that anyone can use, including Radar.

Using a worker in front of our core API allows us to easily add and separate microservices, and also adds notable features like:

• Cloudflare's Cache API allows finer control over what to cache and for how long and supports POST requests and customizable cache control headers, which we use.

• Stale responses using R2. When the backend API cannot serve a request for some reason, and there’s a stale response cached, it’ll be served directly from R2, giving end users a better experience.

• CSV and JSON output formats. The CSV format is convenient and makes it easier for data scientists, analysts, and others to use the API and consume our API data directly from other tools.

One last feature on the frontend API is OpenAPI 3 support. We automatically generate an OpenAPI schema and validate user input with it. This is done through a custom library that we built on top of itty-router, which we also use. Today we’re open sourcing this work.

itty-router-openapi provides an easy and compact OpenAPI 3 schema generator and validator for Cloudflare Workers. Check our GitHub repository for more information and details on how to use it.

Today we’re also launching our developer’s documentation pages for the Radar API where you can find more information about our data license, basic concepts, how to get started and the available API methods. Cloudflare Radar's API is free, allowing academics, data sleuths and other web enthusiasts to investigate Internet usage across the globe, based on data from our global network.

To facilitate using our API, we also put together a Colab Notebook template that you can play with, copy and expand to your use case.

The Radar App is the code that runs in your browser. We've talked about Remix, but what else do we use?

Radar relies on a lot of data visualizations. Things like charts and maps are essential to us. We decided to build our reusable library of visualization components on top of two other frameworks: visx, a "collection of expressive, low-level visualization primitives for React," D3, a powerful JavaScript library for manipulating the DOM based on data, and MapLibre, an open-source map visualization stack.

Here’s one of our visualization components in action. We call it the “PewPew map”.

And here’s the Remix React code for it, whenever we need to use it in a page:

<Card.Section
    title={t("card.attacks.title")}
    description={t("card.attacks.description")}
  >
    <Flex gap={spacing.medium} align="center" justify="flex-end">
      <SegmentedControl
        label="Sort order:"
        name="attacksDirection"
        value={attacksDirection}
        options={[
          { label: t("common.source"), value: "ORIGIN" },
          { label: t("common.target"), value: "TARGET" },
        ]}
      onChange={({ target }: any) => setAttacksDirection(target.value)}
      />
    </Flex>

    <Components.AttacksCombinedChart
      ssr={ssr}
      height={400}
      direction={attacksDirection}
    />
  </Card.Section>

Another change we made to Radar was switching our images and graphical assets to Scalable Vector Graphics. SVGs are great because they're essentially a declarative graphics language. They're XML text files with vectorial information. And so, they can be easily manipulated, transformed, stored, or indexed, and of course, they can be rendered at any size, producing beautiful, crisp results on any device and resolution.

SVGs are also extremely small and efficient in size compared to bitmap formats and support internationalization, making them easier to translate to other languages (localization), thus providing better accessibility.

Here’s an example of a Radar Bubble Chart, inspected, where you can see the SVG code and the strings embedded.

React Cosmos is a "sandbox for developing and testing UI components in isolation." We wanted to use Cosmos with Radar 2.0 because it's the perfect project for it:

1. It has a lot of visual components; some are complex and have many configuration options and features.

2. The components are highly reusable across multiple pages in different contexts with different data.

3. We have a multidisciplinary team; everyone can send a pull request and add or change code in the frontend.

Cosmos acts as a component library where you can see our palette of ready-to-use visualizations and widgets, from simple buttons to complex charts, and you play with their options in real-time and see what happens. Anyone can do it, not only designers or engineers but also other project stakeholders. This effectively improves team communications and makes contributing and iterating quickly.

Here’s a screenshot of our Cosmos in action:

Continuous integration is important for any team doing modern software. Cloudflare Pages provides multiple options to work with CI tools using direct uploads, out of the box. The team has put up documentation and examples on how to do that with GitHub Actions, CircleCI, and Travis, but you can use others.

In our case, we use BitBucket and TeamCity internally to build and deploy our releases. Our workflow automatically builds, tests, and deploys Radar 2.0 within minutes on an approved PR and follow-up merge.

Unit tests are done with Vitest and E2E tests with Playwright. Visual Regression testing is planned and Playwright can also help with that.

Furthermore, we have multiple environments to stage and test our releases before they go live to production. Our CI/CD setup makes it easy to switch from one environment to the other or quickly roll back any undesired deployment.

Again Cloudflare Pages makes it easy to do this using Preview deployments, aliases, or Branch build controls. The same is true for regular Workers using Environments.

Radar 1.0 wasn't particularly fast doing CI/CD, we confess. We had a few episodes when a quick fix could take some good 30 minutes from committing the code to deployment, and we felt frustrated about it.

So we invested a lot in ensuring that the new CI would be fast, efficient, and furious.

One cool thing we ended up doing was fast preview links on any commit pushed to the code repository. Using a combination of intelligent caching during builds and doing asynchronous tests when the commit is outside the normal release branches, we were able to shorten the deployment time to seconds.

This is the notification we get in our chat when anyone pushes code to any branch:

Anyone can follow a thread for a specific branch in the chat and get notified on new changes when they happen.

Blazing-fast builds, preview links and notifications are game-changers. An engineer can go from an idea or a quick fix to showing the result on a link to a product manager or another team member. Anyone can quickly click the link to see the changes on a fully working end-to-end version of Radar.

Cloudflare is committed to web accessibility. Recently we announced how we upgraded Cloudflare’s Dashboard to adhere to industry accessibility standards, but this premise is valid for all our properties. The same is true for localization. In 2020, we internationalized our Dashboard and added support for new languages and locales.

Accessibility and localization go hand in hand and are both important, but they are also different. The Web Content Accessibility Guidelines define many best practices around accessibility, including using color and contrast, tags, SVGs, shortcuts, gestures, and many others. The A11Y project page is an excellent resource for learning more.

Localization, though, also known as L10n, is more of a technical requirement when you start a new project. It's about making sure you choose the right set of libraries and frameworks that will make it easier to add new translations without engineering dependencies or code rewrites.

We wanted Radar to perform well on both fronts. Our design system takes Cloudflare's design and brand guidelines seriously and adds as many A11Y good practices as possible, and the app is fully aware of localization strings across its pages and UI components.

Adding a new language is as easy as translating a single JSON file. Here's a snippet of the en-US.json file with the default American English strings:

{
  "abbr.asn": "Autonomous System Number",
  "actions.chart.download.csv": "Download chart data in CSV",
  "actions.chart.download.png": "Download chart in PNG Format",
  "actions.chart.download.svg": "Download chart in SVG Format",
  "actions.chart.download": "Download chart",
  "actions.chart.maximize": "Maximize chart",
  "actions.chart.minimize": "Minimize chart",
  "actions.chart.share": "Share chart",
  "actions.download.csv": "Download CSV",
  "actions.download.png": "Download PNG",
  "actions.download.svg": "Download SVG",
  "actions.share": "Share",
  "alert.beta.link": "Radar Classic",
  "alert.beta.message": "Radar 2.0 is currently in Beta. You can still use {link} during the transition period.",
  "card.about.cloudflare.p1": "Cloudflare, Inc. ({website} / {twitter}) is on a mission to help build a better Internet. Cloudflare's suite of products protects and accelerates any Internet application online without adding hardware, installing software, or changing a line of code. Internet properties powered by Cloudflare have all web traffic routed through its intelligent global network, which gets smarter with every request. As a result, they see significant improvement in performance and a decrease in spam and other attacks. Cloudflare was named to Entrepreneur Magazine's Top Company Cultures 2018 list and ranked among the World's Most Innovative Companies by Fast Company in 2019.",
  "card.about.cloudflare.p2": "Headquartered in San Francisco, CA, Cloudflare has offices in Austin, TX, Champaign, IL, New York, NY, San Jose, CA, Seattle, WA, Washington, D.C., Toronto, Dubai, Lisbon, London, Munich, Paris, Beijing, Singapore, Sydney, and Tokyo.",
  "card.about.cloudflare.title": "About Cloudflare",
...

You can expect us to release Radar in other languages soon.

Radar Reports are documents that use data exploration and storytelling to analyze a particular theme in-depth. Some reports tend to get updates from time to time. Examples of Radar Reports are our quarterly DDoS Attack Trends, or the IPv6 adoption.

The source of these Reports is Jupyter Notebooks. Our Data Science team works on some use-case or themes with other stakeholders using our internal Jupyter Hub tool. After all the iteration and exploration are done, and the work is signed off, a notebook is produced.

A Jupyter Notebook is a JSON document containing text, source code, rich media such as images or charts, and other metadata. It is the de facto standard for presenting data science projects, and every data scientist uses it.

With Radar 1.0, converting a Jupyter Notebook to a Radar page was a lengthy and manual process implicating many engineering and design resources and causing much frustration to everyone involved. Even updating an already-published notebook would frequently cause trouble for us.

Radar 2.0 changed all of this. We now have a fully automated process that takes a Jupyter Notebook and, as long as it's designed using a list of simple rules and internal guidelines, converts it automatically, hosts the resulting HTML and assets in an R2 bucket, and publishes it on the Reports page.

The conversion to HTML takes into account our design system and UI components, and the result is a beautiful document, usually long-form, perfectly matching Radar's look and feel.

We will eventually open-source this tool so that anyone can use it.

We gave examples of using Cloudflare's products and features to build your next-gen app without worrying too much about things that aren't core to your business or logic. A few are missing, though.

Once the app is up and running, you must protect it from bad traffic and malicious actors. Cloudflare offers you DDoS, WAF, and Bot Management protection out of the box at a click's distance.

For example, here are some of our security rules. This is traffic we don't have to worry about in our app because Cloudflare detects it and acts on it according to our rules.

Another thing we don't need to worry about is redirects from the old site to the new one. Cloudflare has a feature called Bulk Redirects, where you can easily create redirect lists directly on the dashboard.

It's also important to mention that every time we talk about what you can do using our Dashboard, we're, in fact, also saying you can do precisely the same using Cloudflare's APIs. Our Dashboard is built entirely on top of them. And if you're the infrastructure as code kind of person, we have you covered, too; you can use the Cloudflare Terraform provider.

Deploying and managing Workers, R2 buckets, or Pages sites is obviously scriptable too. Wrangler is the command-line tool to do this and more, and it goes the extra mile to allow you to run your full app locally, emulating our stack, on your computer, before deploying.

We hope you enjoyed this Radar team write-up and were inspired to build your next app on top of our Supercloud. We will continue improving and innovating on Radar 2.0 with new features, share our findings and open-sourcing our tools with you.

In the meantime, we opened a Radar room on our Developers Discord Server. Feel free to join it and ask us questions; the team is eager to receive feedback and discuss web technology with you.

You can also follow us on Twitter for more Radar updates.

An AS, or Autonomous System, is a group of routable IP prefixes belonging to a single entity, and is one of the key building blocks of the Internet. Internet providers, public clouds, governments, and other organizations have one or more ASes that they use to connect their users or systems to the rest of the Internet by advertising how to reach them.

Per AS traffic statistics and trends help when we need insight into unusual events, like Internet outages, infrastructure anomalies, targeted attacks, or any other changes from service providers.

Today, we are opening more of our data and launching the Cloudflare Radar pages for Autonomous Systems. When navigating to a country or region page on Cloudflare Radar you will see a list of five selected ASes for that country or region. But you shouldn’t feel limited to those, as you can deep dive into any AS by plugging its ASN (Autonomous System Number) into the Radar URL (for example: https://radar.cloudflare.com/asn/812). We have excluded some statistical trends from ASes with small amounts of traffic as that data would be difficult to interpret.

The AS page is similar to the country page on Cloudflare Radar. You can find traffic levels, protocol use, and security details such as application and network-level DDoS attack information. Additionally, we show a geographical distribution map of the traffic and the volume of BGP announcements we see for the list of prefixes associated with the specific AS.

A sudden increase in BGP announcements often suggests disruptive changes to the Internet in the region or institution associated with the AS. Spikes in BGP announcements were visible when the submarine cable was cut in Tonga in 2022, on the Facebook outage in October 2021, and when governments limited the Internet access in their countries (as seen in Sudan and Syria in 2021).

At Cloudflare, we are committed to keep increasing transparency on the inner workings of the Internet, so that we can all do our part in keeping the Internet more open and secure for everyone. Keep an eye on Cloudflare Radar for more insights like these.

The years come and go, Internet traffic continues to grow (at least so far and with some ‘help’ from the pandemic), and Internet applications, be they websites, IoT devices or mobile apps, continue to evolve throughout the year, depending on if they attract human beings.

We’ll have a more broad Internet traffic-related Year in Review 2021 in the next few days (you can check the 2020 one here), but for now, let’s focus on the most popular domains this year according to our data on Cloudflare Radar and those domains’ changes in our popularity ranking. With Alexa.com going away, if you need a domain ranking, you can get it from Cloudflare.

We’ll focus on space (NASA and SpaceX flew higher), e-commerce (Amazon and Taobao rule), and social media (TikTok ‘danced’ to take the crown from Facebook). We’ll also take a little ‘bite’ on video streaming wars. Netflix is a Squid Game of its own and January 2021 was at the highest in our ranking — probably lockdown and pandemic-related.

Chat domains (WhatsApp, what else) will also be present and, of course, the less established metaverse domains of sorts (Roblox took the lead from Fortnite late in the game). Come with us, let’s travel through 2021.

The following will show the way Cloudflare saw Internet traffic focusing on specific domains (some of which have many websites aggregated into them) and their highs and lows in our global popularity ranking.

Let’s start with our Top Domains Ranking and 2021 brought us a very interesting duel for the Number 1 spot in our global ranking. Google.com (which includes Maps, Translate, Photos, Flights, Books, and News, among others) ended 2020 as the undefeated leader in our ranking — from September to December of last year it was always on top. Back then TikTok.com was only ranked #7 or #8.

1 TikTok.com2 Google.com3 Facebook.com4 Microsoft.com5 Apple.com6 Amazon.com7 Netflix.com8 YouTube.com9 Twitter.com10 WhatsApp.com

1 Google.com2 Facebook.com3 Microsoft.com4 Apple.com5 Netflix.com*6 Amazon.com7 TikTok.com8 YouTube.com9 Instagram.com *10 Twitter.com

Amazon was #5 in November, but Netflix surpassed in December 2020 (on some days it was higher than Apple, in #4); Instagram and Twitter were constantly changing positions throughout November and December.

2021 told a different story. It was on February 17, 2021, that TikTok got the top spot for a day. Back in March, TikTok got a few more days and also in May, but it was after August 10, 2021, that TikTok took the lead on most days. There were some days when Google was #1, but October and November were mostly TikTok’s days, including on Thanksgiving (November 25) and Black Friday (November 26).

There are other trends we can see comparing both years — for 2020 we only show data of the end of the year, after September (Cloudflare Radar was launched that month). For example, Facebook.com was steadily number #2 across 2020, but with TikTok.com going up Facebook is now a solid #3, followed by Microsoft.com (Office365 and Teams numbers are included there) and by Apple.com (App Store and Apple TV+ numbers are included), the same trend as in 2020.

Amazon.com is the juggernaut that follows, but it is interesting to see that since January 2021 the e-commerce website (we will talk more about that category in a few paragraphs) jumped in front of Apple.com. But Apple got back in front, after September, with some exceptions like November 28, 2021, the day before Cyber Monday — and also December 1 and 6.

Netflix had a great 2020 Christmas but also January 2021, especially at the weekend

Another trend is that Netflix surpassed Amazon in December 2020, especially around Christmas week. On some days around 2020 Christmas, Netflix was even higher than Apple, in #4, that is the case with December 23, 25, and from December 29 to January 2, 2021.

In our global popularity ranking we also saw another trend: YouTube, usually ranked #6 or #7, got to the top spot of our list on February 2, 2021 — and only on that day.

This fitness instructor video that happened while the Myanmar coup d’état was happening went viral on February 2, 2021, leading to the creation of thousands of memes

Why? One can only guess, but back then, although it was the week of the Super Bowl (some commercials, like the one from Doritos with Matthew McConaughey, were out on that day), there was another big newsworthy event: the Myanmar coup d'état on February 1, 2021. How can a coup in a Southeast Asian country have an impact on YouTube? A video of a fitness instructor who unwittingly filmed as the takeover unfolds behind her took the Internet by storm and became viral as the memes started to pour in.

That February day was also the one where Donald Trump announced his new legal team for the impeachment trial after the previous one quit, and Jeff Bezos announced he would step down as Amazon’s CEO. That was also the week prior to a record in YouTube’s history. On February 11, 2021, the video "Baby Shark Dance" from Korean education brand Pinkfong was the new most-viewed YouTube video of all time, surpassing the former record holder "Despacito" by Luis Fonsi.

Google Trends also shows that the week of February 2 was the one in 2021 that “YouTube” was more searched on Google.

In what was the second year of the pandemic, social media domains continued high on our ranking. The nine main social media applications were all in our top 100 list of most popular global domains — the only one out is Quora.com (during 2021 it was between #687 and #242).

We can see that TikTok (who also surpassed Google, as we explained before in the global #1 spot) took Facebook from its crown of the most popular social media website-domain in our ranking. So, that should mean that TikTok got more Internet traffic from our standpoint (our ranking is derived from our public DNS resolver 1.1.1.1 and so it's not related to the number of unique users or visitors it gets per month) — Facebook is, by far, the platform with more users worldwide).

1 TikTok.com2 Facebook.com3 YouTube.com4 Twitter.com5 Instagram.com6 Snapchat.com7 Reddit.com8 Pinterest.com9 LinkedIn.com10 Quora.com

1 Facebook.com2 TikTok.com3 YouTube.com4 Instagram.com5 Twitter.com6 Snapchat.com7 Reddit.com8 Pinterest.com9 LinkedIn.com10 Quora.com

The Facebook outage — that we explained from our standpoint extensively — on October 4, 2021, also had an impact on Facebook’s position in our ranking, leading to Facebook.com losing its #3 position (it was #4) for seven days in a row in that week. This number of days in #4 was something that never happened before (since September 2020) to the social media giant.

Looking to the top 10 list, it's also clear that, just looking to social media domains, YouTube comes third and Twitter got a bump up and beat Instagram in 2021, getting the #5 place (barely, in what was a very close race). Back in late 2020 Twitter was behind Instagram in our ranking.

LinkedIn is the ninth most popular social media domain in our ranking and is still in our top 100 and throughout 2021 it got higher in our list, especially in February and March. The social media for professionals then started to drop in June and July (in the Northern Hemisphere’s summer), starting in late August to climb again and by November it reached the #52 place, the highest of the year in our global ranking — in January it was ~#78. In a year when terms like The Great Resignation and the reset of people and organizations’ mindsets were talked about, it makes sense to see this social media platform growing.

The so-called video streaming wars got another important round in 2021 with new players appearing and old ones having amazing numbers — not only in subscribers, revenue, and content budgets but also in... Internet traffic. In our ranking, Netflix is still the undefeated hero.

We added YouTube.com (its most important service is free) to the list to compare with the big numbers from Netflix, and still, the Squid Game phenomenon platform won our ranking for most of the year. Amazon Prime is not included because the streaming service mainly uses Amazon.com (ranked #5 or #6 most of the year) as a domain.

The days of the year when Netflix was more popular? January was a great month with Netflix reaching the #4 spot in our global ranking in the first two days of the year (and also all the weekends of January, Fridays included), going through February in the #5 place. For the rest of 2021, the platform was mostly #7. Yes, on the weekends Netflix seems to have a better performance in our ranking.

Roku.com seems to be the next video streaming platform after those two traffic giants, getting around the #80 position in our ranking through 2021. In late 2020 Hulu.com was the next one, but HBOMax.com surpassed Hulu in July 2021 and entered our top 100 list. In 2021, Disneyplus.com also rose in our ranking and surpassed the app-based TV service Sling.com later in the year. Our top 10 chart also includes Iq.com (iQiyi), the Chinese online video platform.

1 Netflix.com2 YouTube.com3 Roku.com4 HBOMax.com5 Hulu.com6 Peacocktv.com7 Disneyplus.com8 ParamountPlus.com9 Sling.com10 Iq.com

1 Netflix.com2 YouTube.com3 Roku.com4 Hulu.com5 HBOMax.com6 Peacocktv.com7 Sling.com8 Disneyplus.com9 Iq.com10 Wetv.vip

In the chart Netflix.com, more dominant in most days, is in pink and YouTube.com is in yellow

Netflix and YouTube, but also Roku, HBOMax, Hulu.com and Peacocktv are a lot higher in our list than the others — Disney+ is getting there

Since the pandemic started e-commerce has continued to strive and grow at an even faster pace than before. The top four e-commerce domains (Amazon, Taobao, eBay and Walmart) in our global ranking are all in the top 100 and that happens steadily throughout the year.

The fifth in the e-commerce list, the Chinese giant Jd.com had a few periods that it also entered the top 100 mainly in May and especially June — on the day of the 618 shopping event, on June 18, 2021, it reached #68 on our list, beating Walmart.com and almost catching Ebay.com.

In the following list it is easy to see that Jd.com surpassed Shopify.com in 2021, occupying the #5 place, and also Bestbuy.com and Target.com rose from one year to another.

1 Amazon.com2 Taobao.com3 Ebay.com4 Walmart.com5 Jd.com6 Shopify.com7 Bestbuy.com8 Target.com9 Rakuten.co.jp10 Homedepot.com

1 Amazon.com2 Taobao.com3 Ebay.com4 Walmart.com5 Shopify.com6 Jd.com7 Olx.com.br8 Rakuten.co.jp9 Target.com10 Bestbuy.com*

Shein.com went ahead of Bestbuy.com and Target.com from December 19 to 24, 2020

Here are other trends:

• Amazon.com is a domain, as we already explained, with more than e-commerce services (that’s why globally it ranks between #4 and #6). In 2021, it had some good days in January and in late April 2021, reaching #4, but by the end of the year it got its best days in our ranking, especially on the day before Cyber Monday, November 28, and on December 1 and 6 — it reached #5.

• Taobao.com had its best day of the year in our global ranking on August 20 — #15 — and by the popular Chinese shopping day, Singles' Day, November 11, it was #17.

• Ebay.com had a solid year and a good late August (#29 on August 31) and grew more after Cyber Monday, peaking on December 1, reaching #27.

• Shopify had a great August (reaching #100 on August 18), the same with Etsy.com that peaked at #128 on August 21. Walmart had a great June (#66) and also end of November (it reached #70).

• Ikea.com had a big increase in importance throughout the year and got very near to Homedepot.com’s position in September (peaked in the #695 position in our global ranking), staying up through November.

• Best Buy peaked on October 6 and had a high growth throughout November, also matching Shopify in December.

When we look to Shein.com we see that it peaked last Christmas and is on the rise since November 2021

• Shein.com, the global Chinese online fast-fashion retailer, went high in our ranking for the Christmas of 2020 — it went ahead of Bestbuy.com and Target.com from December 19 to 24, 2020, reaching the #253 position. In March, it had another peak, and it got the best position in 2021 in our ranking after Cyber Monday — it reached #301 on December 1, 2021.

This year was also a big year for space travel with several achievements. Spacecraft from three Mars exploration programs from the United Arab Emirates, China, and the United States arrived at Mars in February — NASA’s Perseverance rover landed on February 18, 2021, and after that the Ingenuity drone made history, being the first powered aircraft flight on another planet in human history. And there is also another big space event just around the corner — the James Webb Telescope launch.

Virgin Galactic (July 11), Blue Origin (July 20) and SpaceX (September 16 — but with several other events before that regarding satellites and reuse of space capsules) also stormed the Internet with space tourism achievements with different scopes. Only SpaceX offered an orbital ride.

In terms of domains, NASA.gov was way ahead of the others, but Elon Musk’s SpaceX.com was definitely second in our global ranking, followed by Blueorigin.com. Virgingalactic.com only appears once in our top 100k ranking on July 17 and 18 (a few days after Richard Branson’s spaceflight).

Since last year NASA is high on our global ranking, in the top 1,000 domains of our list, but after the rover Perseverance landed on Mars on February 18 NASA.gov entered our top 700 ranking — the highest day of that month was February 25, when it reached #657. In the summer it went down in our ranking, but it picked up in late September and on October 13, 2021, reached the highest position of the year (#637). That was the day the press conference about NASA's Lucy mission, the agency’s first to Jupiter’s Trojan asteroids, took place (the launch was on October 16).

SpaceX.com had a great start of February, it entered our top 8,000, a month with a launch of 60 new Starlink internet satellites into orbit amidst a missed rocket landing and a fresh $850 million of new investment. And then it was after September 16, 2021, with the first orbital launch of an all-private crew, Inspiration4, that it flew again in our ranking.

For Blue Origin, after a strong start of the year — it reached our #32,000 on January 10 (a few days before New Shepard 4’s first test flight) — it went up between July 20- 27 after its first crewed flight, with Jeff Bezos onboard. It also went up in our ranking a few days after October 13, 2021 (the day William Shatner flew aboard a Blue Origin suborbital capsule).

There aren’t as many messaging or chat platforms as there are popular social media sites, video streaming, or e-commerce platforms. So, this ranking is slim, and even slimmer because Messenger (uses Facebook.com) or iMessage (uses Apple.com) aren’t included. Snapchat is both a social media platform and a messaging app — the same with Instagram — and we added them in the social media ranking.

If they were here they would be higher than Telegram but behind WhatsApp — Instagram actually started 2021 (it got to #8) in front of WhatsApp until February and went as low as #13 and Snapchat went between #29 to #16.

1 WhatsApp.com2 Telegram.org3 WeChat.com4 Signal.org

1 WhatsApp.com2 Telegram.org3 Signal.org4 WeChat.com

From our standpoint, WhatsApp is the undisputed leader of the messaging apps ranging from as low as #13 in our global ranking to as high as #8. Its best parts of the year were late March, late April, late October and then late November going through December 2021 as #8 in our ranking.

The only that is closer is Telegram.org, ranging from #170 to #88 (peaked in October 2021) throughout the year.

All the others are far away in our ranking, but 2021 brought three trends we should highlight:

• Signal.org had an incredible month of January — on January 3 it was in #1815 in our ranking and by January 20 it rose to #766, a climb in more than 1,000 positions in just 17 days. Why? WhatsApp's new privacy policy was in the headlines in the second week of January.

• WeChat.com also had an amazing jump in our ranking, but more in February and by April it surpassed Signal.org — it went from #3142 at the start of February to #979 by April 25 and by October both of the messaging apps were almost tied at ~#370 and had a significantly higher place in our ranking than in late 2020. Telegram.org on the other hand started the year on a high note, #107, and rose to #102 after the first week of January, but lost positions in our ranking after that. In August started to grow more to reach the #88 spot we already mentioned — by then, WhatsApp was #10 in our ranking.

“You can't just materialize anywhere in the Metaverse, like Captain Kirk beaming down from on high. This would be confusing and irritating to the people around you. It would break the metaphor. Materializing out of nowhere (or vanishing back into Reality) is considered to be a private function best done in the confines of your own House.“― Neal Stephenson, Snow Crash (1992)

Back in November, we heard in the halls of Web Summit — the 42,000 in-person tech global event in Lisbon — that in a way the metaverse is already here (Roblox’s Global Head of Music had some thoughts on virtual concerts). But we’re still far from the promise of almost living in the virtual world that books like Neal Stephenson’s Snow Crash or Ernest Cline’s Ready Player One showed us.

Oculus shipped a lot of headsets and there are immersive experiences out there that are Metaverse-like (a step further than the now-usual-for-most spending all day working, learning, communicating through a screen) and we focused on that ones, like Fortnite, Roblox, Second Life (the oldest, from 2003), Minecraft and Oculus. But Oculus.com doesn’t have enough direct traffic (playing games using Oculus headset could direct the traffic elsewhere) to be in our top 100k domains ranking, and the same happens with Minecraft.

1 Roblox.com2 Epicgames.com (Fortnite)3 Secondlife.com

Oculus.com and Minecraft.net are not in our 100,000 ranking

1 Epicgames.com (Fortnite)2 Roblox.com3 Secondlife.com

Oculus.com and Minecraft.net are not in our 100,000 ranking

The (short) list from 2020 and 2021 shows us that Roblox.com surpassed Epicgames.com (the home of the popular Fortnite) for the first time in July reaching back then #27 in our list. But it was after late September that it was consistently in front of the rival game platform, ending the year on a good note reaching #20 in our ranking.

Epicgames.com (Fortnite) started the year a lot better, reaching #14 on January 5, 2021, but it started to lose importance in February and that deepened after May, but mostly in July and August. It never truly recovered and ended the year between #26 and #47, depending on the day.

The Internet is not a quiet place, the same way humans on Earth (especially during a pandemic) aren’t quiet or passive but active and reactive. Although on the top of our domain ranking there don’t seem to be drastic ups and downs throughout the year (TikTok, and YouTube, were the exceptions), we saw how an event like the Myanmar coup and the subsequent viral video may have brought YouTube to #1 on our ranking. We also saw how e-commerce was affected throughout the year, how space-related websites had a big (online) year with important events, and how Netflix rose around Christmas time.

And remember: you can keep an eye on Cloudflare Radar to monitor how we see Internet traffic globally and in every country.

(Update 12/22/2021: We have corrected the data about Telegram.org — the domain that was used initially wasn’t the right one.)