About a year ago, we realized that CloudFlare's current DNS infrastructure had some challenges. We were using PowerDNS, an open source DNS server that is popular with hosting providers. We'd chosen PowerDNS four years ago for several reasons: 1) it was reasonably fast as an authoritative DNS server; 2) it seamlessly allowed us to add new records without rebooting; and 3) it allowed us to write custom backends that tied in to the rest of our system and allowed us to quickly update DNS records.
While PowerDNS got us a long way, it started to run into issues as we scaled and dealt with an increasing number of large denial of service attacks. There were three main challenges. First, PowerDNS when under large in-bound DDoS loads would consume excessive resources and occasionally fall over. Second, and more insidious, because the version of PowerDNS we were using did not have abuse detection and rate limiting, we were increasingly seeing attempts at using CloudFlare authoritative DNS network to launch reflection attacks against others. And, third, it was very difficult to extend PowerDNS and add any sort of application logic which kept us from doing interesting things... like adding Easter Eggs.
Enter RRDNS
To their credit, the PowerDNS community responded to the first two problems with some efforts at rate limiting and other abuse detection. However, because of CloudFlare's unique needs, we realized the only real solution to what we needed was a custom build authoritative name server. About nine months ago, we began the creation of what we affectionately refer to as RRDNS.
Today, every DNS query to CloudFlare runs through RRDNS and we have eliminated PowerDNS from our stack. We're planning a more significant blog post on the technical details of RRDNS and how we build its attack detection and prevention. This, however, is a quick story of how the ease of adding application logic to RRDNS allowed us to create a couple fun Easter Eggs.
Easter Eggs
RRDNS is fully extensible in a modular framework. This allows us to easily add application logic to our name servers, allowing them to respond more dynamically based on the requests they receive. Over time, using this extensibility we plan to add features like DNSSEC. In the meantime, Ian, one of our engineers, wrote a quick extension to demonstrate the functionality.
DNS has many common records such as A, AAAA, CNAME, TXT, MX, etc. One of the largely deprecated records is the CH (Chaos) protocol. Ian decided to allow you to query for CloudFlare's current job listings via a DNS request. He wrote 15 lines of code as an extension to RRDNS. He pulled the listings dynamically from our RSS feed of job listings and returned them if you ran a particular query for the CH record for jobs.cloudflare directly against one of our DNS servers. Here's the output:
$ dig ch jobs.cloudflare @emma.ns.cloudflare.com TXT ;; Truncated, retrying in TCP mode.; <<>> DiG 9.8.3-P1 <<>> ch jobs.cloudflare @emma.ns.cloudflare.com;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28339;; flags: qr aa rd; QUERY: 1, ANSWER: 21, AUTHORITY: 0, ADDITIONAL: 0;; WARNING: recursion requested but not available;; QUESTION SECTION:;jobs.cloudflare. > CH > A;; ANSWER SECTION:jobs.cloudflare. > 86400 > CH > TXT > "Content Marketer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Customer Support Leader, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "General Marketer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Marketing Analytics Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Talent, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Build Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "DDoS Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Database Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Front End Developer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Integration Developer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "JavaScript Performance Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "PHP Developer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Product Manager, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "SDK Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Systems Engineer, London, United Kingdom"jobs.cloudflare. > 86400 > CH > TXT > "Systems Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Network Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Technical Operations Engineer, San Francisco, CA"jobs.cloudflare. > 86400 > CH > TXT > "Technical Operations Engineer, London, United Kingdom"jobs.cloudflare. > 86400 > CH > TXT > "Technical Support Engineer, London, United Kingdom"jobs.cloudflare. > 86400 > CH > TXT > "Technical Support Engineer, San Francisco, CA";; Query time: 16 msec;; SERVER: 2400:cb00:2049:1::adf5:3a70#53(2400:cb00:2049:1::adf5:3a70);; WHEN: Mon Aug 26 19:42:26 2013;; MSG SIZE rcvd: 1122
The power of this isn't a new way for you to tell what job listings we have. Instead it is that we can easily embed dynamic application logic into our DNS system. For an even more useless, albeit fun Easter Egg, try querying for the CH record for whois.cloudflare against one of our name servers. You may need to expand the size of your terminal window for full effect.
All Fun & Games Until Someone Gets Hurt
One of my first questions when I saw these Easter Eggs was: can someone now use these to in order to use CloudFlare's network to amplify a DNS reflection attack.
Thankfully, Ian had already thought of this and built in a safeguard. DNS reflection attacks rely on the attacker spoofing the source IP of the victim when querying a DNS server. Since most DNS requests are sent over UDP, which has no handshake, it is possible to cause DNS servers to respond to the spoofed source.
If you look at the response to the query above, the first line of the response reads:
;; Truncated, retrying in TCP mode.
Because RRDNS is highly flexible, Ian was able to force it when getting one of the Easter Egg-generating queries to respond back with a 0-byte UDP response and the DNS message truncated flag. This causes the client making the DNS request to retry via TCP. Since TCP has a handshake as part of its protocol, it prevents the source IP from being forged. Since the UDP response is smaller than the original query, and since the TCP response will only be sent to a source IP that has been verified with a full handshake, the risk of the Easter Eggs being used to amplify a DNS reflection attack is eliminated.
It doesn't take very much thought to see how we could use this extensibility to better defend against some of the DNS attacks we see, both those directed at us and also those using us to target others. Over the next few weeks we'll dive into more of the technical details behind RRDNS. Until then, enjoy the Easter Eggs.