A week ago we published a story about new amplification attacks using memcached protocol on UDP port 11211. A few things happened since then:
Github announced it was a target of 1.3Tbps memcached attack.
OVH and Arbor reported similar large attacks with the peak reported at 1.7Tbps.
Let's take a deep breath and discuss why such large DDoS attacks are even possible on the modern internet.
Large attacks use IP spoofing
CC BY-SA 2.0 image by DaPuglet
All the gigantic headline-grabbing attacks are what we call "L3" (Layer 3 OSI[1]). This kind of attack has a common trait - the malicious software sends as many packets as possible onto the network. For greater speed these packets are hand crafted by attackers - they are not generated using high-level, well-behaved libraries. Packets are mashed together as a series of bytes and fired onto the network to inflict the greatest damage.
L3 attacks can be divided into two categories, depending on where the attacker directs their traffic:
Direct: where the traffic is sent directly against a victim IP. A SYN flood is a common attack of this type.
Amplification: the traffic is sent to vulnerable UDP servers. They in turn reflect and amplify it by sending large responses to the unbeknown victim.
Both of these types of attack require the attacker to do IP spoofing[2].
IP Spoofing happens when the attacker sends IP packets with a fake source IP address. It's like forging a return address on a letter and pretending to be someone else.
Spoofing source IP addresses is not technically challenging. Every machine connected to the internet can transmit any bytes of their choosing - including setting arbitrary values in the source IP address field. It's just that such packets can do a great deal of damage when they are permitted onto the wide internet.
Direct attacks
In direct attacks the traffic is sent directly towards the victim. There is a bright side of this - the target can investigate the source IPs! Since Cloudflare is often a target of such attacks, occasionally we look at the data and have some fun.
To visualize the source IPs of the attacks, it's best to use Hilbert Curves. Randall Munroe popularized this technique in this famous XKCD. The general idea is great - draw all 2**32 IPs on a two dimensional map by employing fancy math.
Here are a couple of direct attacks doing IP spoofing that we saw in past.
In the first one, the attacker was spoofing indiscriminately all over the IP space, including multicast IP ranges, reserved blocks, and military prefixes.
What's even more funny, the attacker even spoofed the 127.0.0.0/8 range. Indeed, we can confirm the dreadful source of attacks - 127.0.0.1 - hits on regular basis.
Here is another example. During this one some IP ranges - reserved multicast and 127.0.0.0/8 - were not used:
Were attackers smart enough to skip the reserved source IPs? Unlikely. A plausible interpretation is that the traffic flows through a router that filters these obviously wrong sources.
We've also seen attackers that prefer left:
Or right side of the internet:
We've seen many more exciting spoofed IP source patterns. The take away is: during direct attacks, you can't trust the source IPs! Attackers can set them to values of their choosing. Using source IPs in mitigations might not work, drawing them on geographical map makes no sense.
Amplification attacks
In amplification attacks the traffic sent by the attacker has different characteristics .
The source IP is spoofed and set to the victim's IP address.
Traffic is sent vulnerable amplification servers.
The victim can only see the IPs of the vulnerable servers. Most common visualizations show them on a globe somehow. For example in the recent memcached-based attacks here is the map of abused servers:
Why is IP Spoofing a thing?
The attacks discussed so far rely on IP spoofing. But why is IP spoofing even possible?
It's a side effect of the design of the internet.
There are three properties of the internet that make it impossible to verify if the packet is legitimate on the receiving side.
First there's "multihoming" - a feature allowing the internet to be inter-connected. With multihoming networks can have more than a single internet provider. On the receiving side we don't know how many providers the sender has, and there is no way to ensure that a specific path is legitimate.
Next, there's the internet's dynamic nature. The paths can change over time, which makes it fragile to deploy static filtering rules. Even if we knew that certain IP ranges can reach us only from one path, that can change over time.
Then there's the routing asymmetry. The naive rule "a best path from me to you must be the same as from you to me" breaks at Tier 2 and Tier 1 layers. In the real world, the path in either direction differs, so we can't use best paths as hints for filtering rules.
Effective filtering preventing IP spoofing can only be done on the edge of the network - at the last-mile ISPs. Even doing this is hard. If I haven't convinced you yet, here's a more concrete example illustrating the problem.
FooCorp example
Let me introduce an example company, FooCorp which owns the IP ranges: 192.0.2.0/24 and 2001:db8::/48.
This company resides in a shiny office building in Snowville, Utah, and is connected to the internet by a single cable, operated by a carrier named UtahCom.
Say we, at Cloudflare, are very worried about these specific IP ranges, and want to ensure that nobody other than FooCorp can ever spoof them. And suppose we are also directly connected to the UtahCom carrier.
It's all nice and pretty! By looking at the picture we can know that all the traffic with source IPs in 192.0.2.0/24 and 2001:db8::/48 must surely be delivered via UtahCom. Fighting IP spoofing is trivial! We can easily block source IPss 192.0.2.0/24 and 2001:db8::/48 coming from any other provider, right?
Unfortunately, this is not how the internet works.
One day FooCorp decided to get a second internet connection, from say IdahoCom. Cloudflare is also connected to IdahoCom and...
... at this moment all our IP spoofing assumptions break. Traffic sourced from 192.0.2.0/24, 2001:db8::/48 can now be sent EITHER via UtahCom OR IdahoCom! From Cloudflare - the receiver - point of view, we pretty much can't do any source IP filtering now!
Moreover, even if we saw packets from FooCorp IPs, sent from, say IndonesiaCom, we still can't really filter them. It is very possible that FooCorp just got yet more internet connectivity ... with a satelite connection I guess ... from Indonesia. Unlikely as it sounds, this is totally plasuible.
To add insult to injury, UtahCom's IP filtering logic is also not banal. While it definitely should not allow FooCorp to send traffic from arbitary source IPs, it can't really filter the packets from IPs 192.0.2.0/24, 2001:db8::/48 coming from other providers. For example, what if UtahCom and IdahoCom are interconnected, they could totally exchange 192.0.2.0/24, 2001:db8::/48 traffic between themselves!
Solving the problem
Solving IP spoofing is hard and must be done in the last mile (as close to the source of the traffic as possible). The internet community understood that years ago and wrote down the famous BCP38:
It digs deeper into what I explained above. Only the last mile ISPs can perform the source IP filtering and prevent IP spoofing. The internet carriers can't really do much about it.
Or can they?
Internet Carriers
It turns out internet carriers can help. There are two things they can do.
First, they can clean up their networks, and set their filtering right. While it's impossible to ensure that nobody else spoofed FooCorp's 192.0.2.0/24, 2001:db8::/48 ranges, it is totally possible to make sure that FooCorp can't spoof others!
It's not trivial though. In principle, the IP ranges of FooCorp can change over time, but such events should require informing carriers, presenting Letter of Authorization and validating IRR databases (see also RADb).
Next, while large Tier 1 and Tier 2 providers can't do source IP filtering, they can help with investigations. Remember - during large amplification attacks, the attacker spoofs the victim IP ranges. During attacks large carriers should help with debugging and be able to identify which of their paying customers pretended to be the victim!
To be able to do this some carriers maintain internal netflow/IPFIX infrastructure. The community should start asking the carriers tough questions - do you have visibility into who spoofed traffic? Do you do netflow internally and can you effectively use it?
I'm sure Github would like to know who faked their source IPs in order to generate 1.3Tbps amplification.
Internet Exchanges
Internet exchanges face the same attribution issue. When spoofing happens, it's often impossible to figure out which participant originated it. Internet exchanges should put together tools allowing participants to trace who originated the spoofed traffic.
Some internet exchanges do this already, but most don't.
Hardware vendors
A common technique to prohibit IP spoofing is called Strict uRPF. While it can't be used on the Tier 1 layer, since it assumes internet routing symmetry, uRPF definitely is useful in last-mile deployments.
Vendors should ship their hardware with secure defaults. Things like "DHCP Snooping" and "IP Source Verify" should be enabled by default on switches, and "Strict uRPF" on routers. Read more in this paper by Internet Society.
On the same note, if you are using Linux for routing you can turn on strict uRPF with sysctl:
for fname in /proc/sys/net/ipv4/conf/*/rp_filter; do\
echo 1 | sudo tee $fname; \
doneYou
Finally, everyone else can help the "Spoofer" project run by caida.org. "Spoofer" assesses networks by running an agent on your laptop, periodically probing the network.
Users, especially heavy travelers, might consider installing the agent software to help Caida to get greater coverage of end-user ISPs.
Final thoughts
In the past, the industry has struggled with DNS and NTP protocols being vulnerable to amplification attacks. These days we are fighting SSDP, CLDAP and Memcached amplifications. I'm certain in future there will be many more broken UDP protocols that could be used for amplification.
The lasting solution to the problem is to focus on fighting IP spoofing itself. Without IP spoofing, the internet would be a much calmer place.
As for now, attackers doing IP spoofing are invincible. Normal internet participants have no tools to track them. Without attribution it's hard to respond to malicious activity, for example with legal measures. The war is very much asymmetric in favor of the spoofers. To change that we must start actively fighting the spoofing problem.
I'm going to claim that all the large attacks require IP spoofing. A notable exception are the large GRE Mirai attacks that didn't employ spoofing of any kind. ↩︎
This is a simplification. Direct attacks don't strictly require IP spoofing, but very much benefit from it. Without IP spoofing the mitigation would be rather straightforward - just block the abusive source IPs. To avoid this and cover their tracks, the attackers usually do employ IP spoofing though. ↩︎