This morning a hacker was able to access a customer's account on CloudFlare and change that customer's DNS records. The attack was the result a compromise of Google's account security procedures that allowed the hacker to eventually access to my CloudFlare.com email addresses, which runs on Google Apps. While we are still working with Google to investigate the details, we wanted to highlight it here to make people aware that they too may be vulnerable to similar attacks and provide a full accounting of what happened.
Hack a Long Time Coming
This attack appears to have begun in mid-May. It appears an account request was sent to Gmail for my personal email address. Google's procedure asks for a number of questions to attempt to verify account ownership. We're not clear on how the process works, but it appears that weeks after the process was initiated, the hacker somehow convinced Google's account recovery systems to add a fraudulent recovery email address to my personal Gmail account. The password used on my personal Gmail account was 20+ characters long, highly random, and not used by me on any other services so it's unlikely it was dictionary attacked or guessed.
Once the recovery email address was added, the hacker could then reinitiate the password recovery process and get reset instructions sent to the fraudulent email address. Those instructions were then used to reset my personal email this morning.
Google Apps and Privilege Escalation
Like thousands of other companies, CloudFlare uses Google Apps for email. When we first established CloudFlare.com's email address, I listed my personal email address as a recovery email for my account. The hacker was able to use Google's password recovery and have the password reset sent to my personal email for my CloudFlare.com address. Surprisingly, all CloudFlare.com accounts use two-factor authentication. We are still working with Google to understand how the hacker was able to reset the password without providing a valid two-factor authentication token.
Once the attacker had access to my CloudFlare.com email account, the hacker was able to access our Google Apps administrative panel. The hacker appears to have targeted a particular customer, and initiated a password reset request for the customer's CloudFlare.com account. We sent a copy of these requests to an administrative email account for debugging purposes and, ironically, to watch for invalid password reset requests. The hacker was able to access this account in Google Apps and verify the password reset. At that point, the attacker was able to log into the customer's CloudFlare account and change DNS settings to temporarily redirect the site.
Working With Google to Resolve
We were aware of the incident immediately. We have senior contacts at Google who we worked with in order to regain control of the Google Apps accounts (both my personal Gmail account and my CloudFlare.com account). We were able to revert the change to the customer's account. We manually reviewed all other password reset requests and DNS changes. There were no other CloudFlare.com accounts that were accessed or altered.
To ensure that no other accounts can be compromised, we have invalidated all the password reset logs. We have also removed copies of password reset requests from being set to any administrative email accounts in case our Google Apps account is compromised in the future. From our investigations, it appears that at no time was our database accessed or any additional client data exposed. It appears this was, in effect, a very elaborate and sophisticated attack targeting one particular customer's login information.
Protecting Yourself
My personal email address has been removed from any association with CloudFlare. I've also added two-factor authentication to my personal Gmail account -- something that this incident highlights the importance of. I would recommend if you are using Gmail or Google apps, you take the following steps as soon as possible:
Add two-factor authentication to your account by following the steps here;
Ensure your password on your email account is extremely strong and not used on any other services; and
Change any password recovery email to an account that you do not use for anything else and cannot easily be guessed by a determined hacker.
The final puzzle we don't yet know the answer to is how the hacker was able to bypass Google's two-factor authentication on CloudFlare.com email address. That is troubling. That should have prevented this attack, even if the attacker had the password, so it remains concerning to us that it did not. We are working with Google to understand how two-factor authentication was disabled. As we learn more, we'll update this post.
**Update (Saturday, June 2, 2012, 7:40 GMT):**Just received notice from Google that they tracked down the issue core issue that allowed a compromise of the two-factor authentication system. Google reports that they discovered a "subtle flaw affecting not 2-step verification itself, but the account recovery flow for some accounts. We've now blocked that attack vector to prevent further abuse." That's great news. I want to reiterate that the Google Security team has, at all times throughout this incident, been responsive and attentive to the issue. In my opinion, they are the model of security on the Internet and we continue to trust them to power email for CloudFlare.com.
Update (Saturday, June 2, 2012, 19:37 GMT): We have found no evidence of unauthorized access to CloudFlare's core systems or other customers accounts. We continue to work with Google to understand the nature of how the Google App's platform was breached. In a review of the contents of the email accounts that were compromised, we discovered some customers' API keys were present. In order to ensure they could not be used as an attack vector, we reset all customer API keys and disabled the process that would previously email them in certain cases to CloudFlare administrator accounts. If you're using an app like the CloudFlare WordPress plugin, you'll need to reenter your new API key.
We've received questions some questions from customers about credit card numbers. CloudFlare's payment systems are designed to never see any credit card numbers. Credit card data is sent directly to a secure payment processor without ever passing through CloudFlare's servers. This is designed to protect sensitive account information even in the case of a full breach by a fully privileged administrator.
Update (Monday, June 4, 2012, 1:40 GMT): Working with Google we believe we have discovered the vulnerability that allowed the hacker to access my personal Gmail account, which was what began the chain of events. It appears to have involved a breach of AT&T's systems that compromised the out-of-band verification. The upshot is that if an attacker knows your phone number and your phone number is listed as a possible recovery method for your Google account then, at best, your Google account may only be as secure as your voicemail PIN. In this case, we believe AT&T was compromised, potentially through social engineering of their support staff, allowing the hacker to bypass even the security of the PIN. We have removed all phone numbers as authorized Google account recovery methods. We are following up with AT&T to get more details.
Update (Thursday, June 7, 2012, 1:30 GMT): We've gotten a clearer picture of how AT&T's systems were circumvented. The morning of June 1, AT&T's customer support receive a call from the hacker impersonating me. AT&T's logs show that the hacker was not able to answer the account's official security question, but the customer support agent verified the account with the last four digits of my social security number. What's strange about that is the account is a corporate account and should not contain my SSN. If the hacker had CloudFlare's EIN, under AT&T's policies that should not have been sufficient to verify the account. As the senior customer support manager said as he was reading through the logs, "This is odd. This is very odd." The hacker asked the voicemail box to be redirected to the phone number (347) 291-1346. That is a landline or VoIP line controlled by Bandwidth.com. That, subsequently, allowed the hacker to fool Google's voice authentication system into leaving the account recovery PIN on my voicemail.
AT&T's Fraud and Criminal Investigations Research Team is continuing their investigation. In the meantime, we have added additional security to our AT&T account. While it is not a well advertised option, it is possible for businesses and individuals to add a 4 - 20 digit passcode which restricts all changes to the account unless the passcode is known. If you are interested in this option, you will need to complete a form provided by AT&T. First level customer service agents cannot add the passcode, and many agents appear to not be aware of the option. If you call to add it, ask to talk to a supervisor and follow up to confirm that the passcode has been properly added.
Finally, I'm happy to report that, as of last night, Google appears to have removed the voice verification option from the Gmail account recovery process. Given how easy it appears to redirect voicemail, voice calls should not be considered a secure out-of-band channel. While AT&T has assured me that there is no way to redirect SMS messages, we have removed any verification channel that relies on a mobile provider network. Instead, our two-factor authentication uses the Google Authenticator app which is free and can be installed on most smart phones. This ensures that there is a completely out-of-band authentication system that does not pass through a potentially insecure carrier's network.
Update (Tuesday, June 26, 2012, 21:40 GMT): It appears that the FBI has arrested most, if not all, of the individuals involved with the attack.