If you want to know the best mince pie to buy in 2018 right now, skip straight to the bottom of this post where we reveal the winner. If you want to understand more about what makes a mince pie great and how we can learn this at Cloudflare’s scale - read on.

With a very short amount of time to get this research out to a discerning and demanding public, we engaged the entire Cloudflare London team to help. Team members diligently went out and purchased mince pies from all over the South East of England for everyone to taste.

On Monday the team assembled for a “Mince Pie Jam”, where we would taste & consistently review each pie:

As we can see from even the most cursory Internet search, there are a lot of questions surrounding mince pies. After some confusion relating to our 2017 edition of the IMDB we should clearly state: modern mince pies do not contain meat. A Mince Pie is “a sweet pie of British origin, filled with a mixture of dried fruits and spices called mincemeat, that is traditionally served during the Christmas season in the English world.” - Wikipedia for Mince Pie

We reviewed the mince pies in terms of its characteristics as well as overall. Each mince pie was given a score of 1 to 10 for each of the following:

• Innovation

• Booziness

• Pastry to Filling Ratio

• Pastry

• Mince

• Overall rating

The top 3 categories there are somewhat objective measurements compared to the more subjective ones below. During analysis, this created some exciting opportunities.

With this combination of 3 subjective & 3 objective metrics and a total of 214 submitted reviews by our team, we had a lot of data. To break this data down Nela quickly settled on the Pearson Correlation Coefficient (PCC).

Wikipedia describes the PCC as follows:

The correlation coefficient ranges from −1 to 1. A value of 1 implies that a linear equation describes the relationship between X and Y perfectly, with all data points lying on a line for which Y increases as X increases. A value of −1 implies that all data points lie on a line for which Y decreases as X increases. A value of 0 implies that there is no linear correlation between the variables.

By setting our X Value as the Overall Rating, we were able to plug in each of the other metrics from our reviews as the Y value, to understand how important each of these characteristics are in producing a high quality result.

2018 Internet Mince pie Database evaluation criteria coefficient. Explore the full dataset

From this data we can conclude that satisfaction with a pie’s mince filling is the most likely criteria for overall pie satisfaction with r=.841, while innovation creates a negative correlation implying that traditional mince pies are favoured, and straying from this will only lead to dissatisfaction. Alcohol content had a minor positive correlation while pastry, pastry to filling ratio, and aesthetic had an average positive correlation.

Along with scores, our reviewers also left tasting notes which gave an insight into how they arrived at their scores. Taking the reviews which scored a mince pie from 1-5 we’re able to see the terms most associated with poorer performing pies:

Pies scoring between 6 and 10 points had the following terms used the most in tasting notes:

Before we get to the final overall rankings, there are a few categories we wanted to mention.

Unsurprisingly, some of the most expensive mince pies come from London’s fanciest Restaurants & Bakeries. The 3 most expensive mince pies did not rank highly enough to feature elsewhere - which goes to show you that money doesn’t necessarily buy you the best mince pie.

A Dickensian-styled thing of beauty, I expected a pie-seller to appear muttering "gawd blimey guv'nor thems good pies". Good bite, alcohol was raw and overly pungent, tasty but I needed a quiet lie down afterwards.

Deeelicious and moist. A little citrus and Christmas on the finish.

In a word, dense. Dense filling, dense pastry. A bit too heavy all round.

Below are the 3 lowest scoring mince pies in 2018. We publish this not to humiliate but in the hope that these bakers will learn & improve for 2019.

The 10 on "experimental" here means ... experiment gone badly wrong. If ratings allowed for complex numbers, this one would score an overall '10i'.

A bit dry and what on earth is all that air doing there?! Get out and put mince in!

BANG average from Aldi. I don't know where Holly Lane is but if I were to guess it would be firmly in the middle of somewhere. Slightly chalky, pasty pastry and sweet uninteresting mince.

Nela’s impressive Pearson Correlation work showed that there is actually a minor negative correlation with mince pies that are highly innovative. During tasting we observed that innovative pies drew people in but they most definitely divided opinion. Here are the 3 most innovative mince pies (as ranked by their innovation score).

Bad.

Interesting but this is not a mince pie.

Absolutely amazing taste however questionable as a mince pie. 😱😱😱😱

Here are 2018’s top mince pies as stringently tested by the Cloudflare mince pie QA engineers:

2018 Internet Mince pie Database Rankings. Explore the full dataset.

I tried it twice. Will probably try to grab another one. Even with 14 others in-between the first and the second sampling, my preference hasn't changed. To me this is the star of the show. Yum!

Tasty filling, with a solid pastry. A great all rounder, if a little sweet with extra filling peel and topped sugar on the pastry.

Iceland are a dark horse here - the mince here elevates the fairly pedestrian pastry - which is pale and a little bit chalky in texture. But that mince - full of complexity - there's a hint of banana in here despite there being no actual bananas. HOW DID THEY DO THIS? DELICIOUS.

Thanks to everyone at Cloudflare who took part in this extensive research. If you’re inspired by some of the questions answered in this blog post, and you like spiced fruit and intense pastry debates then Cloudflare is hiring in London and all over the world.

Happy Holidays!

CC-BY-SA 2.0 image by Phil! Gold

Since joining Cloudflare I’ve always known that as we grew, incredible things would be possible. It’s been a long held ambition to work in an organisation with the scale to answer a very controversial and difficult question. To do so would require a collection of individuals with a depth of experience, passion, dedication & above all collaborative spirit.

As Cloudflare’s London office has grown in the last 4 years I believe 2017 is the year we reach the tipping point where this is possible. A paradigm-shift in the type of challenges Cloudflare is able to tackle. We could finally sample every commercially available mince pie in existence before the 1st of December. In doing so, we would know conclusively which mince pie we should all be buying over Christmas to share with our friends & families.

For the uninitiated, a Mince Pie is “a sweet pie of British origin, filled with a mixture of dried fruits and spices called mincemeat, that is traditionally served during the Christmas season in the English world.” - Wikipedia for Mince Pie

The original Mince Pie was typically filled with a mixture of minced meat, suet and a variety of fruits and spices like cinnamon, cloves and nutmeg. Today, many mince pies are vegetarian-friendly, containing no meat or suet. They are churned out by both large commercial operations and 200 year-old, family-run bakeries alike to feed hungry Brits at Christmas. Some factories peak at at more than 27pps (pies per second).

Early on we settled on 4 key metrics to score each pie on, on a scale from 1-10. When reviewing anything with a scientific approach, consistency is key. Much like a well made pastry case.

Very quickly we realised that we needed some hard rules on what counted as a pie. For example, we had some "Frangipane Mince Pies" from Marks & Spencer which caused a lot of controversy– these do not have a top, but instead cover the mince with a baked frangipane.

Although these rule-breaking pies were not included in our leaderboard, they're definitely worth a try... one reviewer described the filling as “inoffensive” and another left these comments;

"ZERO air gap! How do you solve the problem of an air gap in your pie? Fill it with some delicious tasty frangipane, that's how. The crunchy almonds on the top really cut through the softness with some texture, too. Most excellent."

Tom Arnfeld, Systems Engineer

Pastry / Filling RatioThe ratio is really key to a good mince pie, but it is also possible for other aspects of the pie to be bad while the ratio itself is excellent. To be clear (there was confusion and debate internally) a score of 5/10 ratio would mean the ratio was average in quality. It does not directly measure the ratio itself. A 10/10 would have the perfect ratio of pastry to mince. Air gap was also a consideration, and the detailed comments each reviewer made on each pie often explain this.

PastryThe right pastry needs to be not too thick, crispy but still have chew and be moist but not soggy. It should hold the filling without it spilling out.

FillingThe filling itself is probably the thing most pies were scored harshly on. A good filling has a variety of fruits and textures and possibly even some other flavours such as brandy.

OverallWe left it to each reviewer to add an overall score judging the entire pie.

30 types of mince pie, 68 reviewers, 18 hungry Cloudflare staff.

After collecting all of the reviews together, here’s our top 5 pies.

£0.75p per pie

Look, I'm going to get straight to it: if you are a mince pie traditionalist, these are not the mince pies for you. HOWEVER, for me they were a revelation. They push boundaries, break all the rules but somehow retain the essence of a great mince pie.

Pastry: TWO DIFFERENT TYPES for the base and the topping. Some people have an issue with the pastry 'rubble' on top. I found it DELIGHTFUL. 9/10

Filling: Not only was there excellent standard mincemeat, but also there was a thin layer of lemon jelly in there as well. SPLENDID. Strong nutmeg and clove themes throughout.

Extra marks for the surreal box art. ("Ceci n'est pas une pie")

Sam Howson, Support Engineer

£1.35p per pie

Pastry / Filling Ratio: Absolutely no visible air gap. TFL could learn from this. Actually fulfilling the promise of deep filled for the first time ever. 10/10

Pastry: Really great. Well constructed, cooked & even all the way around. We've said this a lot but probably "needs more butter" 8/10

Filling: Ironically the reason I like this is also the reason why it's not getting a 10. The citrus - it's so good to have some citrus in there - sorely lacking in lots of other fillings. But it's just too much - it's really the only discernible flavour. Some booze wouldn't go amiss here. 8/10

Overall: This competition might be Dunn Dunn Dunn. Spectacular. 8.5

Simon Moore, Lead Customer Support Engineer

£1.83p per pie

These were by far the most controversial, from our reviewers' point of view.

Pastry / Filling Ratio: About right... a well filled pie with little air gap and a pastry that wasn't so thick that it would diminish the filling. 9/10

Pastry: The weakest part, butter and soft, lovely all over... except that the base was also like this and needed to be a bit crisper and less soft. 7/10

Filling: Subtle flavours, good spices, a hint of Christmas warmth. Could be improved with a little more aftertaste like the K&C ones had, but this is a good filling and with lots of filling, very tasty. 8/10

Overall: A pretty good pie, one of the best yet but the pastry not being crisp and solid underneath makes it fall apart in your hands a little. 8/10

David Kitchen, Engineering Manager

For contrast, one anomalous reviewer wrote;

Pastry / Filling Ratio: EXCELLENT. The pie is deep, and with little air gap. Other reviews mention an air gap however, so QC is clearly not a priority in spite of its cost. 9/10

Pastry: Just fine. Kind of dry, not very buttery. Sugar was nice on top, but really nothing to write home about. Structurally this pie was terrible as the lid lifted clean off when I tried to get it out of its tin. 4/10

Filling: The worst part of this pie, by a massive margin. The mince was almost a pureé, with no discernable fruit textures. Very lightly spiced, again with no flavour components of it being distinguishable. Lack of texture here made it feel like a mince pie for the elderly or small children as after the pastry had disintegrated, there was nothing to chew. 3/10

Overall: Decadently priced farce of a pie filled with a disappointment flavoured mincemeat. Might as well have been packaged in a Londis box. As my girlfriend said "That sounds like it needs covering in cream and then putting in the bin". I wholeheartedly agree. 3/10

Bhavin Tailor, Support Engineer

(price unknown)

Pastry / Filling Ratio: Filled to the brim. Excellent! 9/10

Pastry: Buttery, right thickness, just lovely. Although a bit too much on the sweet side. 7/10

Filling: Great! Nice flavours, still some texture, nothing overwhelms the other flavours. Just wonderful. 8/10

Overall: Great pastry, could eat it all day. 8/10

Tom Strickx, Network Automation Engineer

£0.33p per pie

Pastry / Filling Ratio: Massive air gap. Probably has more air than mince, which is very disappointing. 5/10

Pastry: Buttery, but a bit too thick. Feels a bit heavy. 7/10

Filling: Very noticeable brandy smell, luckily not as present in the taste. Subtle brandy flavour, but doesn't overwhelm the actual mince. A+ 9/10

Overall: Great flavour, love the touch of brandy, unfortunately a bit let down by the filling ratio, and the heaviness of the crust. 7/10

Tom Strickx, Network Automation Engineer

While on our quest to try every pie on the market, we encountered some great ones that are worth a mention. Of aldi mince pies we've tried a single Marks & Spencer pie asda best taste of all, but one had the lidlest per pie cost.

£0.25p per pie

Pastry / Filling Ratio: There's no easy way of saying this, there's more pastry here than there are Greggs branches in Coventry. The only way I would score this lower was if there was no mince at all and I was just eating a solid puck of pastry. 1/10

Pastry: Overcooked, brittle and really just miserable. I'm giving it a point only because it exists. 1/10

Filling: Sweet & Bland. 1/10

Overall: Horrible. 1/10

Simon Moore, Lead Customer Support Engineer

Bigger than the standard supermarket ones, which is definitely nice.

Pastry / Filling Ratio: I think kids these day call it "dat gap". Unfortunately in this case, it's not a good thing.

Pastry: Buttery, crumbly, good thickness, bit bland. 6/10

Filling: Bit bland as well, no specific highlights or notes of flavour. 6/10

Overall: Pretty bland, but not too shabby.

Tom Strickx, Network Automation Engineer

£0.44p per pie

Pastry/Filling Ratio: 9/10 To solve the problem of the 'air gap', Costco decided to top the mince pie with sponge cake. This technically makes the pastry/filling ratio near-perfect since the tiny amount of pastry on the outside matches the tiny amount of filling. Well played, Costco.

Pastry: 3/10 The pastry itself is acceptable but minimal - the majority of the cake is sponge. Yes, I said cake - this is no pie.

Filling: 1/10 They decided to fill it one currant high. And not even like a currant standing proud like the California Raisins, this is a teensy portion. I'm pretty sure the icing on top is thicker than the filling.

Overall: 2/10 This is not a mince pie, it's clearly a sugar cake that has some regulation-mandated minimum amount of mincemeat content to call it a mince pie. Making the 'pies' huge doesn't compensate for anything. Poor showing, Costco.

Chris Branch, Systems Engineer

£0.25p per pie (from Tesco)

Pastry / Filling Ratio: 5/10

Pastry: More salty than buttery. 4/10

Filling: Drabness cloaked in excessive sweetness. 4/10

Mr Kipling purports his products to be "exceedingly good" in his television advertisements, but this pie did not lend support to that claim.

David Wragg, Systems Engineer

£0.38p per pie

Pastry / Filling Ratio: This pie has more air than Michael Jordan 4/10

Pastry: Good but far too thick on the lid. 6/10

Filling: I can detect the booze but it's just not really adding anything. There's nothing of distinction here. 5/10

Overall: 4/10 The ratio and the lid spoil what would otherwise be a serviceable mince pie.

Simon Moore, Lead Customer Support Engineer

With so many mince pies moving through the office on a daily basis, one of our resident staff bakers decided to bake some of his own to add into the mix. Jimmy Crutchfield (Systems Reliability Engineer) brought in 12 lovingly made pies for us all to try...

Pastry / Filling Ratio: Nice and deep, but a little too moist. Surprising given apparently the mincemeat was shop bought – you'd think it would have the right consistency.

Pastry: The pastry was pretty well cooked, and not too dry. I think it could do with a bit more butter though. 8/10

Filling: The added apple bits introduced some delightful new texture. 6/10

Overall: Little in the way of decoration on the top, though bonus points for the lovingly home-made look. Pretty excited about the next version. 9/10

Tom Arnfeld, Systems Engineer

A couple of weeks later, Jimmy had his hand at making a second batch, too!

Pastry / Filling Ratio: Small air gap. 8/10

Pastry: Buttery & crumbly, very good. 9/10

Filling: Just a little too tart. 8/10

Overall: I'd be happy if i'd paid for a box of them. 9/10

Michael Daly, Systems Reliability Engineering Manager

There are too many pies and reviews to mention in full detail, but here’s a full list of the other pies we haven’t mentioned, sorted by their rating.

• Sainsbury’s Bakery (fresh)

• Carluccio’s

• Coco di Mama Mini Mince Pies

• Riverford Farm Shop Classic

• Tesco Standard

• Marks & Spencer Lattice-top

• Marks & Spencer All Butter

• Tesco Finest (with Cognac)

• Aldi Sloe Gin Mince Tarts

• Waitrose All Butter

• Gail’s

• LIDL Favorina

• Co-op Irresistible

• Sainsbury’s Deep Filled

• LIDL Brandy Butter

• Aldi Almond Mince Tarts

With so many reviews from so many staff, we’d like to thank everyone that took part in our quest! Alex Palaistras, Bhavin Tailor, Chris Branch, David Kitchen, David Wragg, Etienne Labaume, John Graham-Cumming, Jimmy Crutchfield, Lorenz Bauer, Matthew Bullock, Michael Daly, Sam Howson, Scott Pearson, Simon Moore, Sophie Bush, Tim Ruffles, Tom Arnfeld, Tom Strickx.

If you want to join a passionate, dedicated, talented and mince pie-filled team - we’re hiring!

CC-BY 2.0 image by Steve Jurvetson

Today Cloudflare’s DNS service answers around 1 million queries per second – not including attack traffic – via a global anycast network. Naturally as a growing startup, the technology we used to handle tens or hundreds of thousands of zones a few years ago became outdated over time, and couldn't keep up with the millions we have today. Last year we decided to replace two core elements of our DNS infrastructure: the part of our DNS server that answers authoritative queries and the data pipeline which takes changes made by our customers to DNS records and distributes them to our edge machines across the globe.

The rough architecture of the system can be seen above. We store customer DNS records and other origin server information in a central database, convert the raw data into a format usable by our edge in the middle, and then distribute it to our >100 data centers (we call them PoPs - Points of Presence) using a KV (key/value) store.

The queries are served by a custom DNS server, rrDNS, that we’ve been using and developing for several years. In the early days of Cloudflare, our DNS service was built on top of PowerDNS, but that was phased out and replaced by rrDNS in 2013.

The Cloudflare DNS team owns two elements of the data flow: the data pipeline itself and rrDNS. The first goal was to replace the data pipeline with something entirely new as the current software was starting to show its age; as any >5 year old infrastructure would. The existing data pipeline was originally built for use with PowerDNS, and slowly evolved over time. It contained many warts and obscure features because it was built to translate our DNS records into the PowerDNS format.

In the old system, the data model was fairly simple. We’d store the DNS records roughly in the same structure that they are represented in our UI or API: one entry per resource record (RR). This meant that the data pipeline only had to perform fairly rudimentary encoding tasks when generating the zone data to be distributed to the edge.

Zone metadata and RRs were encoded using a mix of JSON and Protocol Buffers, though we weren’t making particularly good use of the schematized nature of the protocols so the schemas were very bloated and the resulting data ended up being larger than necessary. Not to mention that as the number of total RRs in our database headed north of 100 million, these small differences in encoding made a significant difference in aggregate.

It’s worth remembering here that DNS doesn’t really operate on a per-RR basis when responding to queries. You query for a name and a type (e.g example.com and AAAA) and you’ll be given an RRSet which is a collection of RRs. The old data format had RRSets broken out into multiple RR entries (one key per record) which typically meant multiple roundtrips to our KV store to answer a single query. We wanted to change this and group data by RRSet so that a single request could be made to the KV store to retrieve all the data needed to answer a query. Because Cloudflare optimizes heavily for DNS performance, multiple KV lookups were limiting our ability to make rrDNS go as fast as possible.

In a similar vein, for lookups like A/AAAA/CNAME we decided to group the values into a single “address” key instead of one key per RRset. This further avoids having to perform extra lookups in the most common cases. Squishing keys together also helps reduce memory usage of the cache we use in front of the KV store, since we’re storing more information against a single cache key.

After settling on this new data model, we needed to figure out how to serialize the data and pass it to the edge. As mentioned, we were previously using a mix of JSON and Protocol Buffers, and we decided to replace this with a purely MessagePack-based implementation.

MessagePack is a binary serialization format that is typed, but does not have a strict schema built into the format. In this regard, it can be considered a little like JSON. For both the reader and the writer, extra fields can be present or absent and it’s up to your application code to compensate.

In contrast, Protocol Buffers (or other formats like Cap’n Proto) require a schema for data structures defined in a language agnostic format, and then generate code for the specific implementation. Since DNS already has a large structured schema, we didn’t want to have to duplicate all of this schema in another language and then maintain it. In the old implementation with Protocol Buffers, we’d not properly defined schemas for all DNS types – to avoid this maintenance overhead – which resulted in a very confusing data model for rrDNS.

When looking for new formats we wanted something that would be fast, easy to use and that could integrate easily into the code base and libraries we were already using. rrDNS makes heavy use of the miekg/dns Go library which uses a large collection of structs to represent each RR type, for example:

type SRV struct {
	Hdr      RR_Header
	Priority uint16
	Weight   uint16
	Port     uint16
	Target   string `dns:"domain-name"`
}

When decoding the data written by our pipeline in rrDNS we need to convert the RRs into these structs. As it turns out, the tinylib/msgp library we had been investigating has a rather nice set of code generation tools. This would allow us to auto-generate efficient Go code from the struct definitions without having to maintain another schema definition in another format.

This meant we could work with the miekg RR structs (of which we are already familiar with from rrDNS) in the data pipeline, serialize them straight into binary data, and then deserialize them again at the edge straight into a struct we could use. We didn't need to worry about mapping from one set of structures to another using this technique, which simplified things greatly.

MessagePack also performs incredibly well compared to other formats on the market. Here’s an excerpt from a Go serialization benchmarking test; we can see that on top of the other reasons MessagePack benefits our stack, it outperforms pretty much every other viable cross-platform option.

One unexpected surprise after switching to this new model was that we actually reduced the space required to store the data at the edge by around 9x, which was a significantly higher saving compared to our initial estimates. It just goes to show how much impact a bloated data model can have on a system.

Another very important feature of Cloudflare’s DNS is our ability to propagate zone changes around the globe in a matter of seconds, not minutes or hours. Our existing pipeline was struggling to keep up with the growing number of zones, and with changes to at least 5 zones each second, even at the quietest of times we needed something new.

For a while now we’ve had this monitoring, and we are able to visualize propagation times across the globe. The graph below is taken from our end-to-end monitoring: it makes changes to DNS via our API and watches for the change from various probes around the world. Each dot on the graph represents a particular probe talking to one of our PoPs, and the delay is tracked as the time it took for a change made via our API to be visible externally.

Due to various layers of caches – both inside and outside of our control – we see some banding on 10s intervals under 1 minute, and it fluctuates all the time. For monitoring and alerting of this nature, the granularity we have here is sufficient but it’s something we’d definitely like to improve. In normal operation, new DNS data is actually available to 99% of our global PoPs in under 5s.

In this time frame we can see there were a couple of incidents where delays of a few minutes were visible for a small number of PoPs due to network connectivity, but generally all probes reported stable propagation times.

In contrast, here’s a graph of the old data pipeline for the same period. We can see how the graph represents the growing delay in visible changes for all PoPs at any given time.

With a new data model designed and ready to go, one that better matched our query patterns, we set out implementing a new service to pick up changes to our zones in the central data store, do any needed processing and send the resulting output to our KV store.

The new service (written in our favourite language Go) has been running in production since July 2016, and we’ve now migrated over 99% of Cloudflare customer zones over to it. If we exclude incidents where issues with congestion across the internet affect connectivity to or from a particular location, the new pipeline itself has experienced zero delays thus far.

rrDNS is a modular application, which allows us to write different “filters” that can hand off processing of different types of queries to different code. The Authoritative filter is responsible for taking an incoming DNS query, looking up the zone the query name belongs to, and performing all relevant logic to find the RRSet to send back to the client.

Since we’ve completely revised the underlying DNS data model at our edge, we needed to make significant changes to the “Authoritative Filter” in rrDNS. This too is an old area of the code base that hasn’t significantly changed in a number of years. As with any ageing code base, this brings a number of challenges, so we opted to re-write the filter completely. This allowed us to redesign it from the ground up on our new data model, keeping a keen eye on performance, and to better suit the scale and shape of our DNS traffic today. Starting fresh also made it much easier to build in good development practices, such as high test coverage and better documentation.

We’ve been running the v2 version of the authoritative filter in production alongside the existing code since the later months of 2016, and it has already played a key role in the DNS aspects of our new load balancing product.

The results with the new filter have been great: we’re able to respond to DNS queries on average 3x faster than before, which is excellent news for our customers and improves our ability to mitigate large DNS attacks. We can see here that as the percentage of zones migrated increased, we saw a significant improvement in our average response time.

The most time consuming part of the project was migrating customers from the old system to something entirely new, without impacting customers or anybody noticing what we were doing. Achieving this involved a significant effort from variety of people in our customer facing, support and operations teams. Cloudflare has many offices in different time zones – London, San Francisco, Singapore and Austin – so keeping everyone in sync was key to our success.

Already, as a part of the release process for rrDNS we automatically sample and replay production queries against existing and upcoming code to detect unexpected differences, so naturally we decided to extend this idea for our migration. For any zone to pass the migration test, we compared the possible answers for the entire zone from the old system and the new system. Just one failure would result in the tool skipping the zone.

This allowed us to iteratively test the migration of zones and fix issues as they arose, keeping releases simple and regular. We chose not to do a single – and very scary – switch away from the old system, but run them both in parallel and slowly move zones over keeping them both in sync. Meaning we quickly could migrate zones back in case something unexpected happened.

Once we got going we were safely migrating zones at several hundred thousand per day, and we kept a close eye on how far we were from our initial goal of 99%. The last mile is still in progress, as there is often an element of customer engagement for some complex configurations that need attention.

Replacing a piece of infrastructure this core to Cloudflare took significant effort from a large variety of teams. So what did we gain?

• Average of 3x performance boost in code handling DNS queries

• Faster and more consistent updates to DNS data around the globe

• A much more robust system for SREs to operate and engineers to maintain

• Consolidated feature-set based on today’s requirements, and better documentation of edge case behaviours

• More test coverage, better metrics and higher confidence in our code, making it safer to make changes and develop our DNS products

Now that we’re now able to process our customers DNS more quickly, we’ll soon be rolling out support for a few new RR types and some other exciting new things in the coming months.

Does solving these kinds of technical and operational challenges excite you? Cloudflare is always hiring for talented specialists and generalists within our Engineering, Technical Operations and other teams.