CC BY 2.0 image by jim simonson

Then we published a set of utilities we are using to generate the BPF rules for our production iptables: the bpftools.

Today we are very happy to open source another component of the bpftools: our p0f BPF compiler!

p0f is a tool written by superhuman Michal Zalewski.The main purpose of p0f is to passively analyze and categorize arbitrary network traffic. You can feed p0f any packet and in return it will derive knowledge about the operating system that sent the packet.

One of the features that caught our attention was the concise yet explanatory signature format used to describe TCP SYN packets.

The p0f SYN signature is a simple string consisting of colon separated values. This string cleanly describes a SYN packet in a human-readable way. The format is pretty smart, skipping the varying TCP fields and keeping focus only on the essence of the SYN packet, extracting the interesting bits from it.

We are using this on daily basis to categorize the packets that we, at CloudFlare, see when we are a target of a SYN flood. To defeat SYN attacks we want to discriminate the packets that are part of an attack from legitimate traffic. One of the ways we do this uses p0f.

We want to rate limit attack packets, and in effect prioritize processing of other, hopefully legitimate, ones. The p0f SYN signatures give us a language to describe and distinguish different types of SYN packets.

For example here is a typical p0f SYN signature of a Linux SYN packet:

4:64:0:*:mss*10,6:mss,sok,ts,nop,ws:df,id+:0

while this is a Windows 7 one:

4:128:0:*:8192,8:mss,nop,ws,nop,nop,sok:df,id+:0

Not getting into details yet, but you can clearly see that there are differences between these operating systems. Over time we noticed that the attack packets are often different. Here are two examples of attack SYN packets:

4:255:0:0:*,0::ack+,uptr+:0
4:64:0:*:65535,*:mss,nop,ws,nop,nop,sok:df,id+:0

You can have a look at more signatures in p0f's README and signatures database.

It's not always possible to perfectly distinguish an attack from valid packets, but very often it is. This realization led us to develop an attack mitigation tool based on p0f SYN signatures. With this we can ask iptables to rate limit only the selected attack signatures.

But before we discuss the mitigations, let's explain the signature format.

CC BY-SA 3.0 image by Hyacinth at the English language Wikipedia

As mentioned, the p0f SYN signature is a colon-separated string with the following parts:

• IP version: the first field carries the IP version. Allowed values are 4 and 6.

• Initial TTL: assuming that realistically a packet will not jump through more than 35 hops, we can specify an initial TTL ittl (usual values are 255, 128, 64 and 32) and check if the packet's TTL is in the range (ittl, ittl - 35).

• IP options length: length of IP options. Although it's not that common to see options in the IP header (and so 0 is the typical value you would see in a signature), the standard defines a variable length field before the IP payload where options can be specified. A * value is allowed too, which means "not specified".

• MSS: maximum segment size specified in the TCP options. Can be a constant or *.

• Window Size: window size specified in the TCP header. It can be a expressed as:

• a constant c, like 8192

• a multiple of the MSS, in the c*mss format

• a multiple of a constant, in the %c format

• any value, as *

• Window Scale: window scale specified during the three way handshake. Can be a constant or *.

• TCP options layout: list of TCP options in the order they are seen in a TCP packet.

• Quirks: comma separated list of unusual (e.g. ACK number set in a non ACK packet) or incorrect (e.g. malformed TCP options) characteristics of a packet.

• Payload class: TCP payload size. Can be 0 (no data), + (1 or more bytes of data) or *.

The following common TCP options are recognised:

• nop: no-operation

• mss: maximum segment size

• ws: window scaling

• sok: selective ACK permitted

• sack: selective ACK

• ts: timestamp

• eol+x: end of options followed by x bytes of padding

p0f describes a number of quirks:

• df: don't fragment bit is set in the IP header

• id+: df bit is set and IP identification field is non zero

• id-: df bit is not set and IP identification is zero

• ecn: explicit congestion flag is set

• 0+: reserved ("must be zero") field in IP header is not actually zero

• flow: flow label in IPv6 header is non-zero

• seq-: sequence number is zero

• ack+: ACK field is non-zero but ACK flag is not set

• ack-: ACK field is zero but ACK flag is set

• uptr+: URG field is non-zero but URG flag not set

• urgf+: URG flag is set

• pushf+: PUSH flag is set

• ts1-: timestamp 1 is zero

• ts2+: timestamp 2 is non-zero in a SYN packet

• opt+: non-zero data in options segment

• exws: excessive window scaling factor (window scale greater than 14)

• linux: match a packet sent from the Linux network stack (IP.id field equal to TCP.ts1 xor TCP.seq_num). Note that this quirk is not part of the original p0f signature format; we decided to add it since we found it useful.

• bad: malformed TCP options

Given a p0f SYN signature, we want to pass it to iptables for mitigation. It's not obvious how to do so, but fortunately we are experienced in BPF bytecode since we are already using it to block DNS DDoS attacks.

We decided to extend our BPF infrastructure to support p0f as well, by building a tool to compile a p0f SYN signature into a BPF bytecode blob, which got incorporated into the bpftools project.

This allows us to use a simple and human readable syntax for the mitigations - the p0f signature - and compile it to a very efficient BPF form that can be used by iptables.

With a p0f signature running as BPF in the iptables we're able to distinguish attack packets with a very high speed and react accordingly. We can either hard -j DROP them or do a rate limit if we wish.

First you need to clone the cloudflare/bpftools GitHub repository:

$ git clone https://github.com/cloudflare/bpftools.git

Then compile it:

$ cd bpftools
$ make

With this you can run bpfgen p0f to generate a BPF filter that matches a p0f signature.

Here's an example where we take the p0f signature of a Linux TCP SYN packet (the one we introduced before), and by using bpftools we generate the BPF bytecode that will match this category of packets:

$ ./bpfgen p0f -- 4:64:0:*:mss*10,6:mss,sok,ts,nop,ws:df,id+:0
56,0 0 0 0,48 0 0 8,37 52 0 64,37 0 51 29,48 0 0 0,
84 0 0 15,21 0 48 5,48 0 0 9,21 0 46 6,40 0 0 6,
69 44 0 8191,177 0 0 0,72 0 0 14,2 0 0 8,72 0 0 22,
36 0 0 10,7 0 0 0,96 0 0 8,29 0 36 0,177 0 0 0,
80 0 0 39,21 0 33 6,80 0 0 12,116 0 0 4,21 0 30 10,
80 0 0 20,21 0 28 2,80 0 0 24,21 0 26 4,80 0 0 26,
21 0 24 8,80 0 0 36,21 0 22 1,80 0 0 37,21 0 20 3,
48 0 0 6,69 0 18 64,69 17 0 128,40 0 0 2,2 0 0 1,
48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 1,
28 0 0 0,2 0 0 5,177 0 0 0,80 0 0 12,116 0 0 4,
36 0 0 4,7 0 0 0,96 0 0 5,29 0 1 0,6 0 0 65536,
6 0 0 0,

If this looks magical, use the -s flag to see the explanation on what's going on:

$ ./bpfgen -s p0f -- 4:64:0:*:mss*10,6:mss,sok,ts,nop,ws:df,id+:0
; ip: ip version
; (ip[8] <= 64): ttl <= 64
; (ip[8] > 29): ttl > 29
; ((ip[0] & 0xf) == 5): IP options len == 0
; (tcp[14:2] == (tcp[22:2] * 10)): win size == mss * 10
; (tcp[39:1] == 6): win scale == 6
; ((tcp[12] >> 4) == 10): TCP data offset
; (tcp[20] == 2): olayout mss
; (tcp[24] == 4): olayout sok
; (tcp[26] == 8): olayout ts
; (tcp[36] == 1): olayout nop
; (tcp[37] == 3): olayout ws
; ((ip[6] & 0x40) != 0): df set
; ((ip[6] & 0x80) == 0): mbz zero
; ((ip[2:2] - ((ip[0] & 0xf) * 4) - ((tcp[12] >> 4) * 4)) == 0): payload len == 0
;
; ipver=4
; ip and (ip[8] <= 64) and (ip[8] > 29) and ((ip[0] & 0xf) == 5) and (tcp[14:2] == (tcp[22:2] * 10)) and (tcp[39:1] == 6) and ((tcp[12] >> 4) == 10) and (tcp[20] == 2) and (tcp[24] == 4) and (tcp[26] == 8) and (tcp[36] == 1) and (tcp[37] == 3) and ((ip[6] & 0x40) != 0) and ((ip[6] & 0x80) == 0) and ((ip[2:2] - ((ip[0] & 0xf) * 4) - ((tcp[12] >> 4) * 4)) == 0)

l000:
    ld       #0x0
l001:
    ldb      [8]
l002:
    jgt      #0x40, l055, l003
l003:
    jgt      #0x1d, l004, l055
l004:
    ldb      [0]
l005:
    and      #0xf
l006:
    jeq      #0x5, l007, l055
l007:
    ldb      [9]
l008:
    jeq      #0x6, l009, l055
l009:
    ldh      [6]
l010:
    jset     #0x1fff, l055, l011
l011:
    ldxb     4*([0]&0xf)
l012:
    ldh      [x + 14]
l013:
    st       M[8]
l014:
    ldh      [x + 22]
l015:
    mul      #10
l016:
    tax
l017:
    ld       M[8]
l018:
    jeq      x, l019, l055
l019:
    ldxb     4*([0]&0xf)
l020:
    ldb      [x + 39]
l021:
    jeq      #0x6, l022, l055
l022:
    ldb      [x + 12]
l023:
    rsh      #4
l024:
    jeq      #0xa, l025, l055
l025:
    ldb      [x + 20]
l026:
    jeq      #0x2, l027, l055
l027:
    ldb      [x + 24]
l028:
    jeq      #0x4, l029, l055
l029:
    ldb      [x + 26]
l030:
    jeq      #0x8, l031, l055
l031:
    ldb      [x + 36]
l032:
    jeq      #0x1, l033, l055
l033:
    ldb      [x + 37]
l034:
    jeq      #0x3, l035, l055
l035:
    ldb      [6]
l036:
    jset     #0x40, l037, l055
l037:
    jset     #0x80, l055, l038
l038:
    ldh      [2]
l039:
    st       M[1]
l040:
    ldb      [0]
l041:
    and      #0xf
l042:
    mul      #4
l043:
    tax
l044:
    ld       M[1]
l045:
    sub      x
l046:
    st       M[5]
l047:
    ldxb     4*([0]&0xf)
l048:
    ldb      [x + 12]
l049:
    rsh      #4
l050:
    mul      #4
l051:
    tax
l052:
    ld       M[5]
l053:
    jeq      x, l054, l055
l054:
    ret      #65536
l055:
    ret      #0

For example, consider we want to block SYN packets generated by the hping3 tool.

First, we need to recognize the p0f SYN signature. Here it is, we know that one off the top of our heads:

4:64:0:0:*,0::ack+:0

(notice: unless you use the -L 0 option, hping3 will send SYN packets with the ACK number set, interesting, isn't it?)

Now, we can use the bpftools to get BPF bytecode that will match the naughty packets:

$ ./bpfgen p0f -- 4:64:0:0:*,0::ack+:0
39,0 0 0 0,48 0 0 8,37 35 0 64,37 0 34 29,48 0 0 0,
84 0 0 15,21 0 31 5,48 0 0 9,21 0 29 6,40 0 0 6,
69 27 0 8191,177 0 0 0,80 0 0 12,116 0 0 4,
21 0 23 5,48 0 0 6,69 21 0 128,80 0 0 13,
69 19 0 16,64 0 0 8,21 17 0 0,40 0 0 2,2 0 0 3,
48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 3,
28 0 0 0,2 0 0 7,177 0 0 0,80 0 0 12,116 0 0 4,
36 0 0 4,7 0 0 0,96 0 0 7,29 0 1 0,6 0 0 65536,
6 0 0 0,

This bytecode can then be passed to iptables:

$ sudo iptables -A INPUT -p tcp --dport 80 -m bpf --bytecode "39,0 0 0 0,48 0 0 8,37 35 0 64,37 0 34 29,48 0 0 0,84 0 0 15,21 0 31 5,48 0 0 9,21 0 29 6,40 0 0 6,69 27 0 8191,177 0 0 0,80 0 0 12,116 0 0 4,21 0 23 5,48 0 0 6,69 21 0 128,80 0 0 13,69 19 0 16,64 0 0 8,21 17 0 0,40 0 0 2,2 0 0 3,48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 3,28 0 0 0,2 0 0 7,177 0 0 0,80 0 0 12,116 0 0 4,36 0 0 4,7 0 0 0,96 0 0 7,29 0 1 0,6 0 0 65536,6 0 0 0," -j DROP

And here's how it would look in iptables:

$ sudo iptables -L INPUT -v
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    6   240            tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80match bpf 0 0 0 0,48 0 0 8,37 35 0 64,37 0 34 29,48 0 0 0,84 0 0 15,21 0 31 5,48 0 0 9,21 0 29 6,40 0 0 6,69 27 0 8191,177 0 0 0,80 0 0 12,116 0 0 4,21 0 23 5,48 0 0 6,69 21 0 128,80 0 0 13,69 19 0 16,64 0 0 8,21 17 0 0,40 0 0 2,2 0 0 3,48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 3,28 0 0 0,2 0 0 7,177 0 0 0,80 0 0 12,116 0 0 4,36 0 0 4,7 0 0 0,96 0 0 7,29 0 1 0,6 0 0 65536,6 0 0 0

While defending from DDoS attacks is sometimes fun, most often it's a mundane repetitive job. We are constantly working on improving our automatic DDoS mitigation system, but we do not believe there is a strong reason to keep it all secret. We want to help others fighting attacks. Maybe if we all worked together one day we could solve the DDoS problem for all.

Releasing our code open source is an important part of CloudFlare. This blog post and the p0f BPF compiler are part of our effort to open source our DDoS mitigations. We hope others affected by SYN floods will find it useful.

Do you enjoy playing with low level networking bits? Are you interested in dealing with some of the largest DDoS attacks ever seen?If so you should definitely have a look at the opened positions in our London, San Francisco, Singapore, Champaign (IL) and Austin (TX) offices!

After submitting the pull request, the netmap maintainers told us that the patch was interesting, but they would prefer something more configurable instead of a tailored custom mode.

After an exchange of ideas and some more work, our patch just got merged to mainline netmap.

Before our patch netmap used to be an all-or-nothing deal. That is: there was no way to put a network adapter partially in netmap mode. All of the queues would have to be detached from the host network stack. Even a netmap mode called “single ring pair” didn't help.

Our final patch is extended and more generic, while still supporting the simple functionality of our original single-rx-queue mode.

First we modified netmap to leave queues that are not explicitly requested to be in netmap mode attached to the host stack. In this way, if a user requests a pair of rings (for example using nm_open(“netmap:eth0-4”)) it will actually get a reference to both the number 4 RX and TX rings, while keeping the other rings attached to the kernel stack.

But since the NIC is still partially connected to the host stack, a new problem arises: what should we do with packets that are going to be transmitted by the operating system to a TX ring which is in netmap mode? The solution is simple: just move them to the RX host ring. In this way we can access these packets from netmap simply by opening the interface again in netmap mode and asking for its software ring pair.

Last, for simpler use cases we needed a way to ask for only the RX rings, without the TX counterpart - we do not need TX rings for our specific use case. To achieve this we introduced a couple of flags, NR_TX_RINGS_ONLY and NR_RX_RINGS_ONLY (which translate to /T and /R when we are using nm_open()) to request only TX or RX rings.

With these changes, the only line we needed to edit in our code was the netmap interface name passed to nm_open(). This:

snprintf(nm_if, sizeof(nm_if) “netmap:%s~%d”, if_name, ring_nr);

becomes this:

snprintf(nm_if, sizeof(nm_if), “netmap:%s-%d/R”, iface_name, ring_nr);

and everything kept working as expected!

You can follow these instructions to build a test program under Linux. In this example we are using the ixgbe driver.

The test program source code is available on github:

• 2015-12-nm-single-rx-queue/main.c

First clone the test application and the netmap repository:

$ git clone https://github.com/cloudflare/cloudflare-blog
$ cd cloudflare-blog/2015-12-nm-single-rx-queue
$ git clone https://github.com/luigirizzo/netmap deps

build it:

$ make

build and load netmap:

$ cd deps/netmap/LINUX
$ ./configure --kernel-sources=/path/to/kernel/sources --driver=ixgbe
$ make
$ sudo insmod netmap.ko
$ sudo insmod ixgbe/ixgbe.ko

and launch the application:

$ sudo ./nm-single-rx-queue eth0 1

We would like to thanks Luigi and Giuseppe for their great help shaping the final patch and their work on netmap.

CC BY-SA 2.0 image by Binary Koala

At CloudFlare we are constantly dealing with large packet floods. Our network constantly receives a large volume of packets, often coming from many, simultaneous attacks. In fact, it is entirely possible that the server which just served you this blog post is dealing with a many-million packets per second flood right now.

Since the Linux Kernel can't really handle a large volume of packets, we need to work around it. During packet floods we offload selected network flows (belonging to a flood) to a user space application. This application filters the packets at very high speed. Most of the packets are dropped, as they belong to a flood. The small number of "valid" packets are injected back to the kernel and handled in the same way as usual traffic.

It’s important to emphasize that the kernel bypass is enabled only for selected flows, which means that all other packets go to the kernel as usual.

This setup works perfectly on our servers with Solarflare network cards - we can use the ef_vi API to achieve the kernel bypass. Unfortunately, we don’t have this functionality on our servers with Intel IXGBE NIC’s.

This is when Netmap comes in.

Over the last few months we’ve been thinking hard about how to achieve bypass for selected flows (aka: bifurcated driver) on non-Solarflare network cards.

We’ve considered PF_RING, DPDK and other custom solutions, but sadly all of them take over the whole network card. Eventually we decided that the best way would be to patch Netmap with the functionality we need.

We chose Netmap because:

• It’s fully open source and released under a BSD license.

• It has a great NIC-agnostic API.

• It’s very fast: can reach line rate easily.

• The project is well maintained and reasonably mature.

• The code is very high quality.

• The driver-specific modifications are trivial: most of the magic happens in the shared Netmap module. It’s easy to add support for new hardware.

Usually, when a network card goes into the Netmap mode, all the RX queues get disconnected from the kernel and are available to the Netmap applications.

We don't want that. We want to keep most of the RX queues back in the kernel mode, and enable Netmap mode only on selected RX queues. We call this functionality: "single RX queue mode".

The intention was to expose a minimal API which could:

• Open a network interface in "a single RX queue mode".

• This would allow netmap applications to receive packets from that specific RX queue.

• While leaving all the other queues attached to the host network stack.

• On demand add or remove RX queues from the "single RX queue mode".

• Eventually remove the interface from the Netmap mode and reattach the RX queues to the host stack.

The patch to Netmap is awaiting code review and is available here:

• https://github.com/luigirizzo/netmap/pull/87

The minimal program receiving packets from eth3 RX queue #4 would look like:

d = nm_open("netmap:eth3~4", NULL, 0, 0);
while (1) {
    fds = {fds: d->fd, events: POLLIN};
    poll(&fds, 1, -1);

    ring = NETMAP_RXRING(d->nifp, 4);
    while (!nm_ring_empty(ring)) {
        i   = ring->cur;
        buf = NETMAP_BUF(ring, ring->slot[i].buf_idx);
        len = ring->slot[i].len;
        //process(buf, len)
        ring->head = ring->cur = nm_ring_next(ring, i);
    }
}

This code is very close to a Netmap example program. Indeed the only difference is the nm_open() call, which uses the new syntax netmap:ifname~queue_number.

Once again, when running this code only packets arriving on the RX queue #4 will go to the netmap program. All other RX and TX queues will be handled by the Linux kernel network stack.

You can find a more complete example here:

• https://github.com/jibi/nm-single-rx-queue

In multiqueue network cards, any packet can end up in almost any RX queue due to RSS. This is why before enabling the single RX mode it is necessary to make sure only the selected flow goes to the Netmap queue.

To do so it is necessary to:

• Modify the indirection table to ensure no new RSS-hashed packets will go there.

• Use flow steering to specifically direct some flows to the isolated queue.

• Work around RFS - make sure no other application is running on the CPU Netmap will run on.

For example:

$ ethtool -X eth3 weight 1 1 1 1 0 1 1 1 1 1
$ ethtool -K eth3 ntuple on
$ ethtool -N eth3 flow-type udp4 dst-port 53 action 4

Here we are setting the indirection table to prevent traffic from going to RX queue #4. Then we are enabling flow steering to enqueue all UDP traffic with destination port 53 into queue #4.

Here's how to run it with the IXGBE NIC. First grab the sources:

$ git clone https://github.com/jibi/netmap.git
$ cd netmap
$ git checkout -B single-rx-queue-mode
$ ./configure --drivers=ixgbe --kernel-sources=/path/to/kernel

Load the netmap-patched modules and setup the interface:

$ insmod ./LINUX/netmap.ko
$ insmod ./LINUX/ixgbe/ixgbe.ko
$ # Distribute the interrupts:
$ (let CPU=0; cd /sys/class/net/eth3/device/msi_irqs/; for IRQ in *; do \
  echo $CPU > /proc/irq/$IRQ/smp_affinity_list; let CPU+=1
         done)
$ # Enable RSS:
$ ethtool -K eth3 ntuple on

At this point we started flooding the interface with 6M short UDP packets. htop shows the server being totally busy with handling the flood:

To counter the flood we started Netmap. First, we needed to edit the indirection table, to isolate the RX queue #4:

$ ethtool -X eth3 weight 1 1 1 1 0 1 1 1 1 1
$ ethtool -N eth3 flow-type udp4 dst-port 53 action 4

This caused all the flood packets to go to RX queue #4.

Before putting an interface in Netmap mode it is necessary to turn off hardware offload features:

$ ethtool -K eth3 lro off gro off

Finally we launched the netmap offload:

$ sudo taskset -c 15 ./nm_offload eth3 4
[+] starting test02 on interface eth3 ring 4
[+] UDP pps: 5844714
[+] UDP pps: 5996166
[+] UDP pps: 5863214
[+] UDP pps: 5986365
[+] UDP pps: 5867302
[+] UDP pps: 5964911
[+] UDP pps: 5909715
[+] UDP pps: 5865769
[+] UDP pps: 5906668
[+] UDP pps: 5875486

As you see the netmap program on a single RX queue was able to receive about 5.8M packets.

For completeness, here's an htop showing only a single core being busy with Netmap:

We would like to thank Pavel Odintsov who suggested the possibility of using Netmap this way. He even prepared the initial hack we based our work on.

We would also like to thank Luigi Rizzo, for his Netmap work and great feedback on our patches.

At CloudFlare our application stack is based on open source software. We’re grateful to so many open source programmers for their awesome work. Whenever we can we try to contribute back to the community - we hope "the single RX Netmap mode" will be useful to others.

You can find more CloudFlare open source here.