(A photo I took on a night out with the team while visiting the Cloudflare Austin office)

Dogfooding is when an organization uses its own products. In this case, we dogfed our newest cloud service, Magic Transit, which both protects and accelerates our customers’ entire network infrastructure—not just their web properties or TCP/UDP applications. With Magic Transit, Cloudflare announces your IP prefixes via BGP, attracts (routes) your traffic to our global network edge, blocks bad packets, and delivers good packets to your data centers via Anycast GRE.

We decided to use Austin’s network because we wanted to test the new service on a live network with real traffic from real people and apps. With the target identified, we began onboarding the Austin office in an always-on routing topology.

In an always-on routing mode, Cloudflare data centers constantly advertise Austin’s prefix, resulting in faster, almost immediate mitigation. As opposed to traditional on-demand scrubbing center solutions with limited networks, Cloudflare operates within 100 milliseconds of 99% of the Internet-connected population in the developed world. For our customers, this means that always-on DDoS mitigation doesn’t sacrifice performance due to suboptimal routing. On the contrary, Magic Transit can actually improve your performance due to our network’s reach.

Now that we’ve completed onboarding Austin to Magic Transit, all we needed was a motivated attacker to launch a DDoS attack. Luckily, we found more than a few willing volunteers on our Site Reliability Engineering (SRE) team to execute the attack. While the teams were still assembling in multiple locations around the world, our SRE volunteer started firing packets at our target from an undisclosed location.

Without Magic Transit, the Austin office would’ve been hit directly with the packet flood. Two things could have happened in this case (not mutually exclusive):

1. Austin’s on-premise equipment (routers, firewalls, servers, etc.) would have been overwhelmed and failed

2. Austin’s service providers would have dropped packets that exceeded its bandwidth allowance

Both cases would result in a very bad day for everyone.

Instead, when our SRE attacker launched the flood the packets were automatically routed via BGP to Cloudflare’s network. The packets reached the closest data center via Anycast and encountered multiple defenses in the form of XDP, eBPF and iptables. Those defenses are populated with pre-configured static firewall rules as well as dynamic rules generated by our DDoS mitigation systems.

Static rules can vary from straightforward IP blocking and rate-limiting to more sophisticated expressions that match against specific packet attributes. Dynamic rules, on the other hand, are generated automatically in real-time. To play fair with our attacker, we didn’t pre-configure any special rules against the attack. We wanted to give our attacker a fair opportunity to take Austin down. Although due to our multi-layered protection approach, the odds were never actually in their favor.

Source: https://imgflip.com

As part of our multi-layered protection approach, Dynamic Rules are generated on-the-fly by analyzing the packets that route through our network. While the packets are being routed, flow data is asynchronously sampled, collected, and analyzed by two main detection systems. The first is called Gatebot and runs across the entire Cloudflare network; the second is our newly deployed DoSD (denial of service daemon) which operates locally within each data center. DoSD is an exciting improvement that we’ve just recently rolled out and we look forward to writing more about its technical details here soon. DoSD samples at a much faster rate (1/100 packets) versus Gatebot which samples at a lower rate (~1/8000 packets), allowing it to detect even more attacks and block them faster.

The asynchronous attack detection lifecycle is represented as the dotted lines in the diagram below. Attacks are detected out of path to assure that we don’t add any latency, and mitigation rules are pushed in line and removed as needed.

Multiple packet attributes and correlations are taken into consideration during analysis and detection. Gatebot and DoSD search for both new network anomalies and already known attacks. Once an attack is detected, rules are automatically generated, propagated, and applied in the optimal location within 10 seconds or less. Just to give you an idea of the scale, we’re talking about hundreds of thousands of dynamic rules that are applied and removed every second across the entire Cloudflare network.

One of the beauties of Gatebot and DoSD is that they don’t require a traffic learning period. Once a customer is onboarded, they’re protected immediately. They don’t need to sample traffic for weeks before kicking in. While we can always apply specific firewall rules if requested by the customer, no manual configuration is required by the customer or our teams. It just works.

Let’s look at what happened in Austin when one of our SREs tried to DDoS Austin and failed. During one of the first attempts, before DoSD had rolled out globally, a degradation in audio and video quality was noticed for Austin employees on video calls for a few seconds before Gatebot kicked in. However, as soon as Gatebot kicked in, the quality was immediately restored. If we hadn’t had Magic Transit in-line, the degradation of service would’ve worsened until the point of full denial of service. Austin would have been offline and our Austin colleagues wouldn’t have had a very productive day.

On a subsequent attack attempt which took place after DoSD was deployed, our SRE launched a SYN flood on Austin. The attack targeted multiple IP addresses in Austin’s prefix and peaked just above 250,000 packets per second. DoSD detected the attack and blocked it in approximately 3 seconds. DoSD’s quick response resulted in no degradation of service for the Austin team.

Green line = Attack traffic to Cloudflare edge, Yellow line = clean traffic from Cloudflare to origin over GRE

Dogfooding Magic Transit served as a valuable experiment for us with lots of lessons learned both from the engineering and procedural aspects. From the engineering aspect, we fine-tuned our mitigations and optimized routings. From the procedural aspects, we drilled members of multiple teams including the Security Operations Center and Solution Engineering teams to help refine our run-books. By doing so, we reduced the onboarding duration to hours instead of days in order to assure a quick and smooth onboarding experience for our customers.

Request a demo and learn how you can protect and accelerate your network with Cloudflare.

Last Friday, at the end of Cloudflare’s 9th birthday week, we announced Workers Sites.

Now, using the Wrangler CLI, you can deploy entire websites directly to the Cloudflare Network using Cloudflare Workers and Workers KV. If you can statically generate the assets for your site, think create-react-app, Jekyll, or even the WP2Static plugin, you can deploy it to our global network, which spans 194 cities in more than 90 countries.

If you’d like to learn more about how it was built, you can read more about this in the technical blog post. Additionally, I wanted to give you an opportunity to meet with some of the developers who contributed to this product and hear directly from them about their process, potential use cases, and what it took to build.

Check out these events. If you’re based in Austin or San Francisco (more cities coming soon!), join us on-site. If you’re based somewhere else, you can watch the recording of the events afterwards.

Growing Dev Platforms at Scale & Deploying Static Websites

Serverless platforms like Cloudflare Workers provide benefits like scalability, high performance, and lower costs. However, when talking to developers, one of the most common reactions is, "this sounds interesting, but what do I build with it?"

In this talk, we’ll cover how at Cloudflare we’ve been able to answer this question at scale with Workers Sites. We’ll go over why this product exists and how the implementation leads to some unintended discoveries.

Speaker Bio:Victoria Bernard is a full-stack, product-minded engineer focused on Cloudflare Workers Developer Experience. An engineer who started a career working at large firms in hardware sales and moved throughout Cloudflare from support to product and to development. Passionate about building products that make developer lives easier and more productive.

When building a platform for developers, you can’t anticipate every use case. So, how do you build new functionality into a platform in a sustainable way, and inspire others to do the same?

Let’s talk about how we took a globally distributed serverless platform (Cloudflare Workers) and key-value store (Workers KV) intended to store short-lived data and turned them into a way to easily deploy static websites. It wasn’t a straightforward journey, but join as we overcome roadblocks and learn a few lessons along the way.

Speaker Bio:Ashley Lewis headed the development of the features that became Workers Sites. She's process and collaboration oriented and focused on user experience first at every level of the stack. Ashley proudly tops the leaderboard for most LOC deleted.

Agenda:

• 6:00pm - Doors open

• 6:30pm - Talk 1: Inspiring with Content: How to Grow Developer Platforms at Scale

• 7:00pm - Talk 2:  Extending a Serverless Platform: How to Fake a File System…and Get Away With It

• 7:30pm - Networking over food and drinks

• 8:00pm - Event conclusion

• DATE/TIME - October 3, 6:00pm-8:00pm

• LOCATION: Cloudflare Austin

Register Here »

• DATE/TIME - October 14, 6:00pm-8:00pm

• LOCATION - Cloudflare San Francisco

Register Here »

While you’re at it, check out our monthly developer newsletter: The Serverlist

Have you built something interesting with Workers? Let us know @CloudflareDev!

Want some ideas on what to start with? Check out the idea suggestion list on our Cloudflare Community page. It's a great idea to review our Apps documentation available here.

Want to request a Cloudflare Apps workshop in your city? Please drop a line to community@cloudflare.com

Share your works in progress and compare notes with other developers on the community forum.

JS Conf EU 2016. Photo by Holger Blank.

If you’re at JS Conf EU (May 6-7) or CSS Conf EU (May 5):

• Be sure to make it to John Fawcett’s session, “Invisible Code: Building JavaScript Libraries For Non-Technical People.”

• Made a Cloudflare App? Drop by the Cloudflare Hacker Lounge to show me and I have a special gift for you.

• Started working on a Cloudflare App and have questions? Drop by the Cloudflare Hacker Lounge and get them answered.

Just happen to be in Berlin? Tweet @qiqing and @IcyApril to come hang out with us in person.

Cloudflare Apps Preview. Visualize your app here.

Join the core developers of Cloudflare Apps for the inaugural Cloudflare meetup in Austin. It will feature an introduction by Zack Bloom (tech lead) to the new Cloudflare Apps including details of how apps get created, moderated, installed, and served to millions of users all over the world.

Bring your laptops!

What: Zack Bloom: “How to make a Cloudflare App: Developer Preview”Where: Capital Factory, 701 Brazos St Ste 1601, Austin, TXWhen: Thursday, May 11, 2017; 6:00 PM to 9:00 PM

RSVP on meetup.com >>

Core devs will be on-hand to help answer questions and serve as teaching assistants. T-shirts and dinner for every attendee. Special goodies for everyone who makes and tests an app.

Eric Teffen Ellis (core dev) will be live-coding a Cloudflare App, followed by Q&A. Bring your laptops! New coders and new friends welcome. Say hi, make noise, and ask questions.

What: Eric Teffen Ellis: “Building apps for the web with Cloudflare.”Where: Vacasa, 926 NW 13th Ave #200, Portland, ORWhen: Thursday, May 11, 2017; 6:00 PM to 9:00 PM

RSVP for PDX Node on meetup.com >>

T-shirts and pizza for everyone.

Want to host a Cloudflare event? Want to request a speaker to teach Cloudflare Apps to your group? Drop a line to community@cloudflare.com.

The CloudFlare team will be at HostingCon 2013 in Austin next week. This is our third year at the show and we have a lot of things in store for partners.

Here's a sneak peek:

• Complimentary limousine transfers from Austin-Bergstrom International Airport to the Hilton Austin hotel on Sunday, June 16th. Reserve your spot today!

• New CloudFlare tshirts

• Live music to supercharge your day during breakfast each morning

• Charging stations at our booth (#523) to keep your devices supercharged

• Bigger and better Nerf Railguns. There is limited quantity, so be sure to visit us at booth #523 to get your Railgun

CloudFlare Railguns ready for HostingCon 2013

We are looking forward to connecting with our current partners and meeting new partners at the show. If you are already a CloudFlare Certified Partner, be sure to stop by and introduce yourself. If you are not a partner yet, stop by to learn more about how CloudFlare can reduce your server load, improve the performance of your network, block spammers, botnets and other web threats, and provide DDOS protection. More details about the CloudFlare Certified Partner program here.

Here's where the CloudFlare team will be all week:

Limo transfers from Austin-Bergstrom International Airport to the Hilton Austin Hotel.

Registration is still open, reserve your spot now!

• 7:45am-8:45am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Alternator Jones

• 5:00pm onwards: Come find the CloudFlare team at the welcome reception!

• 7:45am-8:45am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Jackie Venson

• 12:00pm-4:00pm: CloudFlare is in Exhibit Hall 4 at booth #523

• 4:00pm-6:30pm: Visit our booth during the exhibit hall happy hour for a beverage and to supercharge your mobile phone!

• 8:00am-10:00am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Sean Evan

• 9:00am-9:45am: Our co-founder and CEO Matthew Prince will be speaking on the IPv6 panel discussion "Now is the Time for IPv6" in room #18D

• 9:00-9:45am: Maria Karaivanova and John Roberts from CloudFlare will be co-hosting a talk on partnerships, "Strategies for Successful Partnerships" in room #16

• 12:00-4:00pm: CloudFlare is in Exhibit Hall 4 at Booth #523

Connect with us on Twitter during the event to find out where we are and what's coming up next:#hostingcon, @hostingcon, @CloudFlare

See you in Austin!

Each year, the SXSW Music, Film and Interactive Conferences (sxsw.com) solicit session ideas from the community for the upcoming event using Panel Picker. After submissions are collected, PanelPicker launches a voting interface that allows the world to pick what sessions they'd like to see at SXSW. As soon as the polls open, there is a flood of social media-fueled traffic and all sorts of other challenges for a small IT team to manage. Earlier this week the PanelPicker for the 2012 SXSW Festival began taking votes and traffic to it was intense. That's where CloudFlare came in.

While the site continues to hum along powering millions of page views, CloudFlare has helped SXSW cut bandwidth use by almost half and substantially decrease the load on their servers. SXSW manages their own system security, so they turned CloudFlare's security settings to a low and just used us for the performance benefits.

You can see the effects of enabling CloudFlare on the SXSW servers for yourself:

The SXSW conferences are being held in Austin, TX from March 9 - 18. Members of the CloudFlare team are represented on four proposed panels, so we recommend you test to make sure the PanelPicker is running fast and smooth yourself by heading over there and voting up the following proposed panels:

• Scaling to Infinity: Dealing with Rocket Growth

• Anything You Can Do, I Can Do Backwards in Heels

• JavaScript Performance MythBusters™

• Surviving Lulz: Behind the Scenes of LulzSec

Finally, let us know if you're planning on attending SXSW 2012. We plan to be there in force and would love to celebrate in Austin with everyone at SXSW from the CloudFlare community!