How We Extended CloudFlare's Performance and Security Into Mainland China

CloudFlare launched five years ago. Within a year of our launch, the biggest surprise was the strong global demand for our service. From nearly the beginning, China was the second largest source of traffic by country to our network, behind only the United States.

In retrospect, that shouldn't have been a surprise. In 2010, the year we launched, 34% of China's population, or 450 million people, were online. Today, nearly half the country is online. To put it another way, with 700 million people online, China represents a quarter of all Internet users. If your mission is to help build a better Internet, like CloudFlare's is, then China is a country you cannot ignore.

Consequently, starting in 2011, we began to investigate how CloudFlare could bring our service to the Chinese Internet. Four years later, we're excited to announce the extension of CloudFlare's performance and security platform across mainland China. This is the story of how we did it.

There are three major challenges to extending a service like CloudFlare's across mainland China: technical, economic, and regulatory.

Technical

From a technical perspective, the Chinese Internet, despite its many similarities, is different than the rest of the world. Unlike much of the rest of the world where network routing is open, in China core Internet access is largely controlled by two ISPs: China Telecom and China Unicom. These ISPs control IP address allocation and routing inside of the country. Even the Chinese Internet giants rarely own their own IP address allocations, or use BGP to control routing across the Chinese Internet. This makes BGP Anycast and many of the other routing techniques we use across CloudFlare's network impossible inside of China.

At the same time, there are also frequent bottlenecks both within and between the domestic ISPs. For instance, China Telecom operates many distinct networks across several provinces, many of which operate independently of one another. The interconnection points between these networks, and the ISPs themselves, are also candidates for congestion, with too much traffic paired with too little capacity. The connectivity between different ISPs in different provinces can become so fraught that it is sometimes more efficient to route traffic outside the country, across a third party network, and then back in.

Economic

The technical challenges of the Chinese Internet drive up the costs of doing business as a service provider. Because of local market dynamics among ISPs, the cost of bandwidth, and particularly Anycast bandwidth, is among the highest in the world. Moreover, in order to get adequate performance and route around congestion, you need a large number of geographically distributed data centers across the country. Not an easy feat for a new, non-Chinese entrant to the market.

Despite what some non-Chinese cloud providers suggest, to provide a quality service in China is not as simple as putting up a single location in Beijing. In fact, when we tested the performance of other non-Chinese cloud providers who claimed to have established a presence in mainland China, we were surprised to find that Chinese traffic to their networks was often routed outside of the country through the West Coast of the United States before being delivered back into the country (what is known as networking tromboning). This inherently adds hundreds of milliseconds per request and, ironically, often makes it more likely the content will never be delivered.

The high cost of bandwidth, and the requirement to have a large number of data center locations in order to adequately service the country, makes providing service inside China extremely costly. These costs are further compounded by the difficulty of importing equipment into the country from abroad.

Regulatory

As is the case in many countries, Chinese law prohibits the announcement of certain types of content inside of the country. Although such policies vary greatly between countries, in order to maintain local operations, it is necessary to comply with all local laws and regulations in each country in which we operate. In the case of China, any organization that wishes to operate a website inside of the country needs an Internet content provider (ICP) license from the Chinese Ministry of Industry and Information Technology (MIIT). We investigated whether it would be possible for CloudFlare to obtain an ICP license to cover all our customers, but determined that licenses needed to be issued on a per-site basis. This introduced an enormous amount of regulatory complexity.

Another technical and policy challenge involves the determination of what content can, and cannot be served from within China. At CloudFlare, we fundamentally believe that we should not act as an Internet censor. While we strictly adhere to local rules and regulations, we are careful to do so in a manner that preserves a free and open Internet. Although we may not be able to announce certain content from within China, or any other country in which certain content may be prohibited, we continue to serve it across the Internet from the rest of the network.

After a survey of our customer base, we determined that more than 99% of our customers’ websites are locally available in China today. This provides a tremendous opportunity to increase the performance and security for millions of websites to 700 million Internet users in China. In the meantime, those of our customers that do not qualify for a permit would continue to be served across our network outside of China with a level of performance and security that is neither any better nor worse.

These challenges made it clear that we would be unable to enter the Chinese market on our own. Instead, like others in our space, we started looking for a local partner to resell access to China on top of our own service. The problem with this approach is that, in addition to adding expense, it also adds significant complexity. The features of any local provider's platform were different from our own, meaning that our customers couldn't rely on a unified platform to provide global performance and security, and the customer experience was poor.

Even though we didn’t have a presence in mainland China, we were surprised that Chinese companies continued to sign up for CloudFlare's service. When we surveyed them, there were two primary reasons: 1) we were better at mitigating DDoS attacks (a huge problem for businesses in China) than any rival service; and 2) they had an audience outside of China, and wanted access to our global network even if it meant that their performance suffered at home.

By the summer of 2013, CloudFlare's market share inside China began to get the attention of several Internet companies in the region. Despite the fact that several services with similar feature sets to CloudFlare had started to spring up in China, CloudFlare quickly became the market share leader. Beginning that summer, we began to meet with potential partners to discuss whether there was a way to work together.

Traditionally, when tech companies enter China they do so with a partner and form what is known as a joint venture. We studied the various JVs that other tech companies had formed and came away with the conclusion that they were largely unsuccessful. The repeated mistake appeared to be that non-Chinese tech companies applied too heavy a hand, assuming that what had worked outside of China would work inside of the country.

We concluded that if a collaboration was going to work, we needed to start with the premise that it was a true partnership with CloudFlare providing technology and access to our global network, and the partner contributing local Chinese knowledge and operations. This meant that selecting the right partner was critical.

Among the proposals that we received, Baidu's stood out. Baidu is China’s leading search engine. As we got to know the Baidu team, it was clear that their mission and ethos aligned closely with our own. Moreover, as one of the Internet giants of China, they had the expertise and resources to help us overcome the aforementioned technical, economic, and regulatory challenges.

Today, we're proud to announce our partnership with Baidu as well as the launch of 17 data centers across mainland China—in Qingdao, Fuzhou, Hengyang, Dongguan, Shenyang, Luoyang, Hangzhou, Jiaxing, Tianjin, Guangzhou, Chengdu, Langfang, Xian, Nanning, Zhengzhou, Shijiazhuang, and Foshan. In the months ahead, we will continue to expand our footprint in the country, and expect that by the end of 2016 there will be more locations in mainland China than exist across all the rest of CloudFlare's network today.

Globally, CloudFlare’s network now extends to 62 data center locations:

Baidu's regulatory expertise also helped to solve what previously seemed like an insurmountable problem. They developed a process whereby ICP license applications could be automatically submitted on behalf of CloudFlare customers. This removes the burden of individual customers having to navigate local licensing requirements.

In addition to making China available to CloudFlare's customers, we also worked with Baidu to launch their own service: Yunjiasu (百度云加速), which roughly translates to “fast cloud.” Chinese customers of the Yunjiasu network receive the same performance and security benefits as CloudFlare, including access to CloudFlare’s global network. Yunjiasu has grown rapidly since it its launch in December 2014. Already, the service is used by hundreds of thousands of customers, and serves more than 57 billion page views per month. Between CloudFlare and Yunjiasu, we power more than 60% of all websites using a performance and security service in China today.

The performance benefits of our China expansion are staggering. We are now able to reduce the time to serve a request from outside of China by over 200ms. Across the span of a single day, the time savings for all the requests served inside China across the CloudFlare and Yunjiasu services collectively saves more than 240 years of time that Chinese Internet users would otherwise have to wait for websites to load. Moreover, website availability in China for sites served on the China network has nearly doubled. These benefits will only increase as we begin to serve more customers across the China network.

To give you a sense, one of the first customers to be served across the China network was TechCrunch. CloudFlare has a close relationship with the TechCrunch team, having launched at their Disrupt conference in 2010, and we were happy to learn that TechCrunch’s local China edition is just as widely followed as it is in the US. Before TechCrunch.cn went live on CloudFlare's China network, page loads in mainland China averaged 17 seconds. Now they average 2.5 seconds.

Similar improvements were registered for site availability. Before enabling the China network, TechCrunch.cn was only available about 50% of the time in mainland China. Now the site averages nearly 100% availability.

The benefit of a network inside mainland China goes beyond just performance. Given its large Internet population, China, like other countries, has a number of active botnets. These botnets can be used to launch large-scale distributed denial of service (DDoS) attacks. Some of the largest attacks we see come from botnets with a large number of nodes inside China. With a network inside China, CloudFlare is now better able to sinkhole attacks before they leave the country. This means that attack traffic originating inside China is less likely to cause disruptions for customers outside of the region.

As we’ve extended our network into China, we’ve also taken numerous steps to ensure the security and integrity of our customers’ data. CloudFlare operates all services outside of China, and Baidu all services inside of China. No CloudFlare customer traffic will pass through the China network unless a customer explicitly opts-in to the service. A customer’s traffic and log data from outside of China is never sent into China. And, for customers that opt-in to serving content inside China, customer identifiable information such as email addresses, password hashes, and billing information is never stored in the China network or shared with our partner.

The security and privacy of other potentially sensitive information is also strictly maintained. For instance, CloudFlare's Keyless SSL technology allows us to serve encrypted traffic for customers who opt-in to the China network without having to store private SSL keys within the China network. This allows any customer to receive the benefits of CloudFlare’s full suite of services, even if they elect to have their keys stored outside of China.

The same is true for Yunjiasu customers. While they get the benefits of CloudFlare's global network, we’ve worked with Baidu to ensure that personal information is kept with Baidu and never shared with CloudFlare.

As part of this partnership, CloudFlare was never asked nor did we ever volunteer to provide any data about any of our users to China, the United States, or any other governmental authority. Had that been a requirement of entering the region, we would have passed on the opportunity.

Existing and new CloudFlare customers can request to be served in China by filling out an information request at:

https://www.cloudflare.com/china

Initially, the China network will be limited to Enterprise customers. Over time, as we are better able to operationalize the onboarding of customers, we hope to extend the benefits to all plan levels.

This is an announcement that has been four years in the making. We’re excited to have built the only truly global performance and security platform. And, while China is the largest country in the world that--until today--didn’t have any CloudFlare data centers, there’s another one that’s almost as big that’s still missing some. Stay tuned as that’s soon about to change.