To cap off 2018 we decided hit the road (and then head to the airport) with our Real World Serverless event series in San Francisco, Austin, London, Singapore, Sydney, and Melbourne. It was a great time sharing serverless application development insights we’ve discovered over the past year as well as demonstrating how to build applications with new services like our key value store, Cloudflare Workers KV.

Below is a recording from our Singapore Real World Serverless event. It included three talks about Serverless technology featuring Tim Obezuk, Stanley Tan, and Remy Guercio from Cloudflare. They spoke about the fundamentals of serverless technology, twelve factors of serverless application development, and achieving no ops at scale with network-based serverless.

If you’d like to join us in person to talk about serverless, we’ll be announcing 2019 event locations starting in the new year.

Fundamentals of Serverless Technology - Tim Obezuk (0:00-13:56)

Tim explores the anatomy of Cloudflare’s serverless technology, Cloudflare Workers, and how they can be used to improve availability, build faster websites and save costs. Workers allows you to run Javascript from 150+ data centers around the world.

The Serverless Twelve Factors - Stanley Tan (13:56-22:46)

Developers all know the benefits of the Twelve-Factor App methodology. It is now the industry standard for building modern web app services. Let’s take a look at how this applies to a serverless platform.

Achieving No Ops at Scale with Network-Based Serverless - Remy Guercio (22:46-49:21)

While most major serverless platforms have done an effective job of abstracting the concept of a single server or group of servers, they have yet to make it as easy to deploy globally as it is to deploy to a specific region. Building global applications with region-based serverless providers still requires a significant effort to set up both frontend load balancing and backend data replication. Let’s explore how network-based serverless providers are helping developers build applications of all sizes with a true No Ops mentality.

Check out our Workers recipes we have listed on our docs here »

Today, v2 of Pwned Passwords was released as part of the Have I Been Pwned service offered by Troy Hunt. Containing over half a billion real world leaked passwords, this database provides a vital tool for correcting the course of how the industry combats modern threats against password security.

I have written about how we need to rethink password security and Pwned Passwords v2 in the following post: How Developers Got Password Security So Wrong. Instead, in this post I want to discuss one of the technical contributions Cloudflare has made towards protecting user information when using this tool.

Cloudflare continues to support Pwned Passwords by providing CDN and security functionality such that the data can easily be made available for download in raw form to organisations to protect their customers. Further, as part of the second iteration of this project, I have also worked with Troy on designing and implementing API endpoints that support anonymised range queries to function as an additional layer of security for those consuming the API, that is visible to the client.

This contribution allows for Pwned Passwords clients to use range queries to search for breached passwords, without having to disclose a complete unsalted password hash to the service.

Over time, the industry has realised that complex password composition rules (such as requiring a minimum number of special characters) have done little to improve user behaviour in making stronger passwords; they have done little to prevent users from putting personal information in passwords, avoiding common passwords or prevent the use of previously breached passwords[1]. Credential Stuffing has become a real threat recently; usernames and passwords are obtained from compromised websites and then injected into other websites until you find user accounts that are compromised.

This fundamentally works because users reuse passwords across different websites; when one set of credentials is breached on one site, this can be reused on other websites. Here are some examples of how credentials can be breached from insecure websites:

• Websites which don't use rate limiting or challenge login requests can have a user's log-in credentials breached using brute force attacks of common passwords for a given user,

• database dumps from hacked websites can be taken offline and the password hashes can be cracked; modern GPUs make this very efficient for dictionary passwords (even with algorithms like Argon2, PBKDF2 and BCrypt),

• many websites continue not to use any form of password hashing, once breached they can be captured in raw form,

• Proxy Attacks or hijacking a web server can allow for capturing passwords before they're hashed.

This becomes a problem with password reuse; having obtained real life username/password combinations, they can be injected into other websites (such as payment gateways, social networks, etc) until access is obtained to more accounts (often of a higher value than the original compromised site).

Under recent NIST guidance, it is a requirement, when storing or updating passwords, to ensure they do not contain values which are commonly used, expected or compromised[2]. Research has found that 88.41% of users who received a fear appeal later set unique passwords, whilst only 4.45% of users who did not receive a fear appeal would set a unique password[3].

Unfortunately, there are a lot of leaked passwords out there; the downloadable raw data from Pwned Passwords currently contains over 30 GB in password hashes.

The key problem in checking passwords against the old Pwned Passwords API (and all similar services) lies in how passwords are checked; with users being effectively required to submit unsalted hashes of passwords to identify if the password is breached. The hashes must be unsalted, as salting them makes them computationally difficult to search quickly.

Currently there are two choices that are available for validating whether a password is or is not leaked:

• Submit the password (in an unsalted hash) to a third-party service, where the hash can potentially be stored for later cracking or analysis. For example, if you make an API call for a leaked password to a third-party API service using a WordPress plugin, the IP of the request can be used to identify the WordPress installation and then breach it when the password is cracked (such as from a later disclosure); or,

• download the entire list of password hashes, uncompress the dataset and then run a search to see if your password hash is listed.

Needless to say, this conflict can seem like being placed between a security-conscious rock and an insecure hard place.

Academic computer scientists have considered the problem of how two (or more) parties can validate the intersection of data (from two or more unequal sets of data either side already has) without either sharing information about what they have. Whilst this work is exciting, unfortunately these techniques are new and haven't been subject to long-term review by the cryptography community and cryptographic primitives have not been implemented in any major libraries. Additionally (but critically), PSI implementations have substantially higher overhead than our k-Anonymity approach (particularly for communication[4]). Even the current academic state-of-the-art is not with acceptable performance bounds for an API service, with the communication overhead being equivalent to downloading the entire set of data.

Instead, our approach adds an additional layer of security by utilising a mathematical property known as k-Anonymity and applying it to password hashes in the form of range queries. As such, the Pwned Passwords API service never gains enough information about a non-breached password hash to be able to breach it later.

k-Anonymity is used in multiple fields to release anonymised but workable datasets; for example, so that hospitals can release patient information for medical research whilst withholding information that discloses personal information. Formally, a data set can be said to hold the property of k-Anonymity, if for every record in a released table, there are k − 1 other records identical to it.

By using this property, we are able to seperate hashes into anonymised "buckets". A client is able to anonymise the user-supplied hash and then download all leaked hashes in the same anonymised "bucket" as that hash, then do an offline check to see if the user-supplied hash is in that breached bucket.

In more concrete terms:

In essence, we turn the table on password derivation functions; instead of seeking to salt hashes to the point at which they are unique (against identical inputs), we instead introduce ambiguity into what the client is requesting.

Given hashes are essentially fixed-length hexadecimal values, we are able to simply truncate them, instead of having to resort to a decision tree structure to filter down the data. This does mean buckets are of unequal sizes but allows clients to query in a single API request.

This approach can be implemented in a trivial way. Suppose a user enters the password test into a login form and the service they’re logging into is programmed to validate whether their password is in a database of leaked password hashes. Firstly the client will generate a hash (in our example using SHA-1) of a94a8fe5ccb19ba61c4c0873d391e987982fbbd3. The client will then truncate the hash to a predetermined number of characters (for example, 5) resulting in a Hash Prefix of a94a8. This Hash Prefix is then used to query the remote database for all hashes starting with that prefix (for example, by making a HTTP request to example.com/a94a8.txt). The entire hash list is then downloaded and each downloaded hash is then compared to see if any match the locally generated hash. If so, the password is known to have been leaked.

As this can easily be implemented over HTTP, client side caching can easily be used for performance purposes; the API is simple enough for developers to implement with little pain.

Below is a simple Bash implementation of how the Pwned Passwords API can be queried using range queries (Gist):

#!/bin/bash

echo -n Password:
read -s password
echo
hash="$(echo -n $password | openssl sha1)"
upperCase="$(echo $hash | tr '[a-z]' '[A-Z]')"
prefix="${upperCase:0:5}"
response=$(curl -s https://api.pwnedpasswords.com/range/$prefix)
while read -r line; do
  lineOriginal="$prefix$line"
  if [ "${lineOriginal:0:40}" == "$upperCase" ]; then
    echo "Password breached."
    exit 1
  fi
done <<< "$response"

echo "Password not found in breached database."
exit 0

Hashes (even in unsalted form) have two useful properties that are useful in anonymising data.

Firstly, the Avalanche Effect means that a small change in a hash results in a very different output; this means that you can't infer the contents of one hash from another hash. This is true even in truncated form.

For example; the Hash Prefix 21BD1 contains 475 seemingly unrelated passwords, including:

lauragpe
alexguo029
BDnd9102
melobie
quvekyny

Further, hashes are fairly uniformally distributed. If we were to count the original 320 million leaked passwords (in Troy's dataset) by the first hexadecimal charectar of the hash, the difference between the hashes associated to the largest and the smallest Hash Prefix is ≈ 1%. The chart below shows hash count by their first hexadecimal digit:

Algorithm 1 provides us a simple check to discover how much we should truncate hashes by to ensure every "bucket" has more than one hash in it. This requires every hash to be sorted by hexadecimal value. This algorithm, including an initial merge sort, runs in roughly O(n log n + n) time (worst-case):

After identifying the Maximum Hash Prefix length, it is fairly easy to seperate the hashes into seperate buckets, as described in Algorithm 3:

This implementation was originally evaluated on a dataset of over 320 million breached passwords and we find the Maximum Prefix Length that all hashes can be truncated to, whilst maintaining the property k-anonymity, is 5 characters. When hashes are grouped together by a Hash Prefix of 5 characters, we find the median number of hashes associated with a Hash Prefix is 305. With the range of response sizes for a query varying from 8.6KB to 16.8KB (a median of 12.2KB), the dataset is usable in many practical scenarios and is certainly a good response size for an API client.

On the new Pwned Password dataset (with over half a billion) passwords and whilst keeping the Hash Prefix length 5; the average number of hashes returned is 478 - with the smallest being 381 (E0812 and E613D) and the largest Hash Prefix being 584 (00000 and 4A4E8).

Splitting the hashes into buckets by a Hash Prefix of 5 would mean a maximum of 16^5 = 1,048,576 buckets would be utilised (for SHA-1), assuming that every possible Hash Prefix would contain at least one hash. In the datasets we found this to be the case and the amount of distinct Hash Prefix values was equal to the highest possible quantity of buckets. Whilst for secure hashing algorithms it is computationally inefficient to invert the hash function, it is worth noting that as the length of a SHA-1 hash is a total of 40 hexadecimal characters long and 5 characters is utilised by the Hash Prefix, the total number of possible hashes associated with a Hash Prefix is 16^{35} ≈ 1.39E42.

It is important to note that where a user's password is already breached, an API call for a specific range of breached passwords can reduce the search candidates used in a brute-force attack. Whilst users with existing breached passwords are already vulnerable to brute-force attacks, searching for a specific range can help reduce the amount of search candidates - although the API service would have no way of determining if the client was or was not searching for a password that was breached. Using a deterministic algorithm to run queries for other Hash Prefixes can help reduce this risk.

One reason this is important is that this implementation does not currently guarantee l-diversity, meaning a bucket may contain a hash which is of substantially higher use than others. In the future we hope to use percentile-based usage information from the original breached data to better guarantee this property.

For general users, Pwned Passwords is usually exposed via web interface, it uses a JavaScript client to run this process; if the origin web server was hijacked to change the JavaScript being returned, this computation could be removed (and the password could be sent to the hijacked origin server). Whilst JavaScript requests are somewhat transparent to the client (in the case of a developer), this may not be depended on and for technical users, non-web client based requests are preferable.

The original use-case for this service was to be deployed privately in a Cloudflare data centre where our services can use it to enhance user security, and use range queries to complement the existing transport security used. Depending on your risks, it's safer to deploy this service yourself (in your own data centre) and use the k-anonymity approach to validate passwords where services do not themselves have the resources to store an entire database of leaked password hashes.

I would strongly recommend against storing the range queries used by users of your service, but if you do for whatever reason, store them as aggregate analytics such that they cannot be linked back to any given user's password.

Going forward, as we test this technology more, Cloudflare is looking into how we can use a private deployment of this service to better offer security functionality, both for log-in requests to our dashboard and for customers who want to prevent against credential stuffing on their own websites using our edge network. We also seek to consider how we can incorporate recent work on the Private Set Interesection Problem alongside considering l-diversity for additional security guarantees. As always; we'll keep you updated right here, on our blog.

1. Campbell, J., Ma, W. and Kleeman, D., 2011. Impact of restrictive composition policy on user password choices. Behaviour & Information Technology, 30(3), pp.379-388. ↩︎

2. Grassi, P. A., Fenton, J. L., Newton, E. M., Perlner, R. A., Regenscheid, A. R., Burr, W. E., Richer, J. P., Lefkovitz, N. B., Danker, J. M., Choong, Y.-Y., Greene, K. K., and Theofanos, M. F. (2017). NIST Special Publication 800-63B Digital Identity Guidelines, chapter Authentication and Lifecycle Management. National Institute of Standards and Technology, U.S. Department of Commerce. ↩︎

3. Jenkins, Jeffrey L., Mark Grimes, Jeffrey Gainer Proudfoot, and Paul Benjamin Lowry. "Improving password cybersecurity through inexpensive and minimally invasive means: Detecting and deterring password reuse through keystroke-dynamics monitoring and just-in-time fear appeals." Information Technology for Development 20, no. 2 (2014): 196-213. ↩︎

4. De Cristofaro, E., Gasti, P. and Tsudik, G., 2012, December. Fast and private computation of cardinality of set intersection and union. In International Conference on Cryptology and Network Security (pp. 218-231). Springer, Berlin, Heidelberg. ↩︎

Both in our real lives, and online, there are times where we need to authenticate ourselves - where we need to confirm we are who we say we are. This can be done using three things:

• Something you know

• Something you have

• Something you are

Passwords are an example of something you know; they were introduced in 1961 for computer authentication for a time-share computer in MIT. Shortly afterwards, a PhD researcher breached this system (by being able to simply download a list of unencrypted passwords) and used the time allocated to others on the computer.

As time has gone on; developers have continued to store passwords insecurely, and users have continued to set them weakly. Despite this, no viable alternative has been created for password security. To date, no system has been created that retains all the benefits that passwords offer as researchers have rarely considered real world constraints[1]. For example; when using fingerprints for authentication, engineers often forget that there is a sizable percentage of the population that do not have usable fingerprints or hardware upgrade costs.

In the 1970s, people started thinking about how to better store passwords and cryptographic hashing started to emerge.

Cryptographic hashes work like trapdoors; whilst it's easy to hash a password, it's far harder to turn that "hash" back into the original output (or computationally difficult for an ideal hashing algorithm). They are used in a lot of things from speeding up searching from files, to the One Time Password generators in banks.

Passwords should ideally use specialised hashing functions like Argon2, BCrypt or PBKDF2, they are modified to prevent Rainbow Table attacks.

If you were to hash the password, p4$$w0rd using the SHA-1 hashing algorithm, the output would be 6c067b3288c1b5c791afa04e12fb013ed2e84d10. This output is the same every time the algorithm is run. As a result, attackers are able to create Rainbow Tables which contain the hashes of common passwords and then this information is used to break password hashes (where the password and hash is listed in a Rainbow Table).

Algorithms like BCrypt essentially salt passwords before they hash them using a random string. This random string is stored alongside the password hash and is used to help make the password harder to crack by making the output unique. The hashing process is repeated many times (defined by a difficulty variable), each time adding the random salt onto the output of the hash and rerunning the hash computation.

For example; the BCrypt hash $2a$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy starts with $2a$10$ which indicates the algorithm used is BCrypt and contains a random salt of N9qo8uLOickgx2ZMRZoMye and a resulting hash of IjZAgcfl7p92ldGxad68LJZdL17lhWy. Storing the salt allows the password hash to be regenerated identically when the input is known.

Unfortunately; salting is no longer enough, passwords can be cracked quicker and quicker using modern GPUs (specialised at doing the same task over and over). When a site suffers a security breach, users passwords can be taken offline in database dumps in order to be cracked offline.

Additionally; websites that fail to rate limit login requests or use captchas, can be challenged by Brute Force attacks. For a given user, an attacker will repeatedly try different (but common) passwords until they gain access to a given users account.

Sometimes sites will lock users out after a handful of failed login attempts, attacks can instead be targeted to move on quickly to a new account after the most common set of a passwords has been attempted. Lists like the following (in some cases with many, many more passwords) can be attempted to breach an account:

The industry has tried to combat this problem by requiring password composition rules; requiring users comply to complex rules before setting passwords (requiring a minimum amount of numbers or punctuation symbols). Research has shown that this work hasn't helped combat the problem of password reuse, weak passwords or users putting personal information in passwords.

Whilst it may seem that this is only a bad sign for websites that store passwords weakly, Credential Stuffing makes this problem even worse.

It is common for users to reuse passwords from site to site, meaning a username and password from a compromised website can be used to breach far more important information - like online banking gateways or government logins. When a password is reused - it takes just one website being breached, to gain access to others that a users has credentials for.

This Is Not Fine - The Nib

There are fundamentally three things that need to be done to fix this problem:

• Good UX to improve User Decisions

• Improve Developer Education

• Eliminating reuse of breached passwords

Before discussing the things we're doing, I wanted to briefly discuss what you can do to help protect yourself now. For most users, there are three steps you can immediately take to help yourself.

Use a Password Manager (like 1Password or LastPass) to set random, unique passwords for every site. Additionally, look to enable Two-Factor Authentication where possible; this uses something you have, in addition to the password you know, to validate you. This will mean, alongside your password, you have to enter a short-lived password from a device like your phone before being able to login to any site.

Two-Factor Authentication is supported on many of the worlds most popular social media, banking and shopping sites. You can find out how to enable it on popular websites at turnon2fa.com. If you are a developer, you should take efforts to ensure you support Two Factor Authentication.

Set a secure memorable password for your password manager; and yes, turn on Two-Factor Authentication for it (and keep your backup codes safe). You can find additional security tips (including tips on how to create a secure main password) in my blog post: Simple Cyber Security Tips.

Developers should look to abolish bad practice composition rules (and simplify them as much as possible). Password expiration policies do more harm than good, so seek to do away with them. For further information refer to the blog post by the UK's National Cyber Security Centre: The problems with forcing regular password expiry.

Finally; Troy Hunt has an excellent blog post on passwords for users and developers alike: Passwords Evolved: Authentication Guidance for the Modern Era

Developers should seek to build a culture of security in the organisations where they work; try and talk about security, talk about the benefits of challenging malicious login requests and talk about password hashing in simple terms.

If you're working on an open-source project that handles authentication; expose easy password hashing APIs - for example the password_hash, password_​needs_​rehash & password_verify functions in modern PHP versions.

We know that complex password composition rules are largely ineffective, and recent guidance has followed suit. A better alternative to composition rules is to block users from signing up with passwords which are known to have been breached. Under recent NIST guidance, it is a requirement, when storing or updating passwords, to ensure they do not contain values which are commonly used, expected or compromised[2].

This is easier said than done, the recent version of Troy Hunt's Pwned Passwords database contains over half a billion passwords (over 30 GB uncompressed). Whilst developers can use API services to check if a password is reused, this requires either sending the raw password, or the password in an unsalted hash. This can be especially problematic when multiple services handle authentication in a business, and each has to store a large quantity of passwords.

This is a problem I've started looking into recently; as part of our contribution to Troy Hunt's Pwned Passwords database, I have designed a range search API that allows developers to check if a password is reused without needing to share the password (even in hashed form) - instead only needing to send a short segment of the cryptographic hash used. You can find more information on this contribution in the post: Validating Leaked Passwords with k-Anonymity.

Version 2 of Pwned Passwords is now available - you can find more information on how it works on Troy Hunt's blog post "I've Just Launched Pwned Passwords, Version 2".

1. Bonneau, J., Herley, C., Van Oorschot, P.C. and Stajano, F., 2012, May. The quest to replace passwords: A framework for comparative evaluation of web authentication schemes. In Security and Privacy (SP), 2012 IEEE Symposium on (pp. 553-567). IEEE. ↩︎

2. Grassi, P. A., Fenton, J. L., Newton, E. M., Perlner, R. A., Regenscheid, A. R., Burr, W. E., Richer, J. P., Lefkovitz, N. B., Danker, J. M., Choong, Y.-Y., Greene, K. K., and Theofanos, M. F. (2017). NIST Special Publication 800-63B Digital Identity Guidelines, chapter Authentication and Lifecycle Management. National Institute of Standards and Technology, U.S. Department of Commerce. ↩︎

If you’ve done SEO research, you may have come across an admixture of legitimate SEO practices, outdated optimizations, and misguided advice. You might have read that using the keyword meta tag in your HTML will help your SEO (it won’t), that there’s a specific number of instances a keyword should occur on a webpage (there isn’t), or that buying links will improve your rankings (it likely won’t and will get the site penalized). Let’s sift through the noise and highlight some dos and don’ts for performance-based SEO in 2018.

Nearly every year since its inception, SEO is declared dead. It is true that the scope of best practices for search engines has narrowed over the years as search engines have become smarter, and much of the benefit from SEO can be experienced by following these two rules:

1. Create good content

2. Don’t be creepy

Beyond the fairly obvious, there are a number of tactics that can help improve the importance with which a website is evaluated inside Google, Bing and others. This blog will focus on optimizing for Google, though the principles and practices likely apply to all search engines.

The short answer is, no. When asked whether or not Cloudflare can damage search rankings, John Mueller from Google stated CDNs can work great for both users and search engines when properly configured. This is consistent with our findings at Cloudflare, as we have millions of web properties, including SEO agencies, who use our service to improve both performance and SEO.

Yes, it can. Since at least 2010, Google has publicly stated that site speed affects your Google ranking. While most sites at that time were not affected, times have changed and heavier sites with frontend frameworks, images, CMS platforms and/or a slew of other javascript dependencies are the new normal. Google promotes websites that result in a good user experience, and slow sites are frustrating and penalized in rankings as a result.

The cost of slow websites on user experience is particularly dramatic in mobile, where limited bandwidth results in further constraints. Aside from low search rankings, slow loading sites result in bad outcomes; research by Google indicates 53% of mobile sites are abandoned if load time is more than 3 seconds. Separate research from Google using a deep neural network found that as a mobile site’s load time goes from 1 to 7 seconds, the probability of a visitor bouncing increases 113%. The problems surrounding page speed increase the longer a site takes to load; mobile sites that load in 5 seconds earn 2x more ad revenue than those that take 19 seconds to load (the average time to completely load a site on a 3G connection).

A number of free and verified tools are available for checking a website’s performance. Based on Google’s research, you can estimate the number of visitors you will lose due to excessive loading time on mobile. Not to sound click baitey, but the results may surprise you.

As more web traffic continues to shift to mobile, mobile optimization must be prioritized for most websites. Google has announced that in July 2018 mobile speed will also affect SEO placement. If you want to do more research on your site’s overall mobile readiness, you can check to see if your site is mobile friendly.

If you’re technically-minded and use Chrome, you can pop into the Chrome devtools and click on the audits tab to access Lighthouse, Chrome’s built in analysis tool.

Other key metrics used for judging your site's performance include FCP and DCL speeds. First Contentful Paint (FCP) measures the first moment content is loaded onto the screen of the user, answering the user’s question: “is this useful?”. The other metric, DOM Content Loaded (DCL), measures when all stylesheets have loaded and the DOM tree is able to be rendered. Google provides a tool for you to measure your website’s FCP and DCL speeds relative to other sites.

Generally speaking, there is no cause for concern as shared hosts shouldn’t hurt your SEO, even if some of the sites on the shared host are less reputable. In the unlikely event you find yourself as the only legitimate website on the host that is almost entirely spam, it might be time to rethink your hosting strategy.

If your site is down when it’s crawled, it may be temporarily pulled from results. This is why service interruptions such as getting DDoSed during peak purchases times can be more damaging. Typically a site’s ranking will recover when it comes back online. If it’s down for an entire day it may take up to a few weeks to recover.

Everybody likes to win, but playing outside the rules can have consequences. For websites that attempt to circumvent Google’s guidelines in an attempt to trick the search algorithms and web crawlers, a perilous future awaits. Here are a few things that you should make sure you avoid.

Permitting user-generated spam - sometimes unmoderated comment sections run amok with user generated spam ads, complete with links to online pharmacies and other unrelated topics. Leaving these types of links in place lowers the quality of your content and may subject you to penalization. Having trouble handling a spam situation? There are strategies you can implement.

Link schemes - while sharing links with reputable sources is still a legitimate tactic, excessively sharing links is not. Likewise, purchasing large bundles of links in an attempt to boost SEO by artificially passing PageRank is best avoided. There are many link schemes, and if you’re curious whether or not you’re in violation, look at Google’s documentation. If you feel like you might’ve made questionable link decisions in the past and you want to undo them, you can disavow links that point to your site, but use this feature with extreme caution.

Doorway pages - By creating many pages that optimize for specific search phrases, but ultimately point to the same page, some sites attempt to saturate all the search terms around a particular topic. While this might be tempting strategy to gain a lot of SEO very quickly, it may result in all pages losing rank.

Scraping content - In an attempt to artificially build content, some websites will scrape content from other reputable sources and call it their own. Aside from the fact that this behavior can get a site flagged by the Panda algorithm for unrelated or excessive content, it is also in violation of the guidelines and can result in penalization or removal of a website from results.

Hidden text and links - by hiding text inside a webpage so it’s not visible to users, some websites will try to artificially increment the amount of content they have on their site or the amount of instances a keyword occurs. Hiding text behind an image, setting a font size to zero, using CSS to position an element off of the screen, or the classic “white text on a white background” are all tactics to be avoided.

Sneaky redirects - as the name implies, it’s possible to surreptitiously redirect users from the result that they were expecting onto something different. Split cases can also occur where a desktop version of the site will be directed to the intended page while the mobile will be forwarded to full-screen advertising.

Cloaking - by attempting to show different content to search engines and users, some sites will attempt to circumvent the processes a search engine has in place to filter out low value content. While cloaking might have a cool name, it’s in violation and can result in rank reduction or listing removal.

There are number of sources that can be considered authoritative when it comes to Google SEO. John Mueller, Gary Illyes and (formerly) Matt Cutts, collectively represent a large portion of the official voice of Google search and provide much of the official SEO best practices content. Aside from the videos, blogs, office hours, and other content provided by these experts, Google also provides the Google webmaster blog and Google search console which house various resources and updates.

Last but not least, if you have web properties currently on Cloudflare there are technical optimizations you can make to improve your SEO.

Read that blog post to learn about how to configure your website, and for those who are not able to do that, how to disable caching for certain URIs to prevent this type of attacks. Since our previous blog post, we have looked for but have not seen any large scale attacks like this in the wild.

Today, we have released a tool to help our customers make sure only assets that should be cached are being cached.

Recall that the Web Cache Deception attack happens when an attacker tricks a user into clicking a link in the format of http://www.example.com/newsfeed/foo.jpg, when http://www.example.com/newsfeed is the location of a dynamic script that returns different content for different users. For some website configurations (default in Apache but not in nginx), this would invoke /newsfeed with PATH_INFO set to /foo.jpg. If http://www.example.com/newsfeed/foo.jpg does not return the proper Cache-Control headers to tell a web cache not to cache the content, web caches may decide to cache the result based on the extension of the URL. The attacker can then visit the same URL and retrieve the cached content of a private page.

The proper fix for this is to configure your website to either reject requests with the extra PATH_INFO or to return the proper Cache-Control header. Sometimes our customers are not able to do that (maybe the website is running third-party software that they do not fully control), and they can apply a Bypass Cache Page Rule for those script locations.

Photo by Henry Hustava / Unsplash

The new Cache Deception Armor Page Rule protects customers from Web Cache Deception attacks while still allowing static assets to be cached. It verifies that the URL's extension matches the returned Content-Type. In the above example, if http://www.example.com/newsfeed is a script that outputs a web page, the Content-Type is text/html. On the other hand, http://www.example.com/newsfeed/foo.jpg is expected to have image/jpeg as Content-Type. When we see a mismatch that could result in a Web Cache Deception attack, we will not cache the response.

There are some exceptions to this. For example if the returned Content-Type is application/octet-stream we don't care what the extension is, because that's typically a signal to instruct the browser to save the asset instead of to display it. We also allow .jpg to be served as image/webp or .gif as video/webm and other cases that we think are unlikely to be attacks.

This new Page Rule depends upon Origin Cache Control. A Cache-Control header from the origin or Edge Cache TTL Page Rule will override this protection.

Software engineering pro-tip:

Do not, I repeat, do not deploy this week. That is how you end up debugging a critical issue from your parent's wifi in your old bedroom while your spouse hates you for abandoning them with your racist uncle.

— Chris Albon (@chrisalbon) December 20, 2017

Whilst our Support and SRE teams operated on a schedule to ensure fingers were on keyboards; on Saturday, I headed out of the London bound for the Warwickshire countryside. Away from the barracks of the London tech scene, it didn't take long for the following conversation to happen:

• Family member: "So what do you do nowadays?"

• Me: "I work in Cyber Security."

• Family member: "There seems to be a new cyber attack every day on the news! What can I possibly do to keep myself safe?"

If you work in the tech industry, you may find a family member asking you for advice on cybersecurity. This blog post will hopefully save you from stuttering whilst trying to formulate advice (like I did).

The WannaCry Ransomware Attack was one of the most high-profile cyberattacks of 2017. In essence, ransomware works by infecting a computer, then encrypting files - preventing users from being able to access them. Users then see a window on their screen demanding payment with the promise of decrypting files. Multiple copycat viruses also sprung up, using the same exploit as WannaCry.

It is worth noting that even after paying, you're unlikely to see your files back (don't expect an honest transaction from criminals).

WannaCry was estimated to have infected over 300 000 computers around the world; this included high-profile government agencies and corporations, the UK's National Health Service was one notable instance of this.

Despite the wide-ranging impact of this attack, a lot of victims could have protected themselves fairly easily. Security patches had already been available to fix the bug that allowed this attack to happen and installing anti-virus software could have contained the spread of this ransomware.

For consumers; it is generally a good idea to install updates, particularly security updates. Platforms like Windows XP no longer receive security updates, and therefore shouldn't be used - regardless of how up-to-date they are on security patches.

Of course, it is also essential to back-up your most indispensable files, not just because of the damage security vulnerabilities.

It may not be Easter, but you certainly should not be putting all your eggs in one basket. For this reason; it is often not a good idea to use the same password across multiple sites.

Passwords have been around since 1961, however, no alternative has been found which keeps all their benefits; users continue to set passwords weakly, and website developers continue to store them insecurely.

When developers store computer passwords, they should do so in a way that they can check a computer password is correct but they can never know what the original password is. Unfortunately many websites (including some popular ones) implement internet security poorly. When they get hacked, a password dump can be leaked with everyone's emails/usernames alongside their passwords.

If the same email/username and password combination are used on multiple sites, hackers can automatically use the breached user data from one site to attempt logins against other websites you use online.

For this reason, it's absolutely critical to use a unique password across multiple sites. Password manager apps like LastPass or 1Password allow you to use unique randomly-generated passwords for each site but manage them from one encrypted wallet using a master password.

Simple passwords, based on personal information or using individual words in the dictionary, are far from safe too. Computers can repeatedly go through common passwords in order to crack them. Similarly, adding numbers and symbols (i.e. changing password to p4$$w0rd) will do little to help also.

When you have to choose a password you need to remember, you can create strong passwords from sentences. For example: "At Christmas my dog stole 2 pairs of Doc Martens shoes!" can become ACmds2poDMs! Passwords based on simple sentences can be long, but still easy to remember.

Another approach is to simply select four random dictionary words, for example: WindySoapLongBoulevard. (For obvious reasons, don't actually use that as your password.) Although this password uses solely letters, it is more secure than a shorter password that would also use numbers and symbols.

Authentication is how computers confirm you are who you say you are. Fundamentally, is done using either:

• Something you know

• Something you have

• Something you are

A password is an example of how you can log-in using "something you know"; if someone is able to gain access to that password, it's game-over for that online account.

Instead, it is possible to use "something you have" as well. This means, should your password be intercepted or disclosed, you still have another safeguard to protect your account.

In practice, this means that after entering your password onto a website, you may also be prompted for another code that you need to read off an app on your phone. This is known as Two-Factor Authentication.

Two-Factor Authentication is supported on many of the worlds most popular social media, banking and shopping sites.

When you browse to a website online, you may notice a lock symbol light up in your address bar. This indicates encryption is enabled when talking to the website, this is important in order to prevent interception.

When inputting personal information into websites, it is important you check this green lock appears and that the website address starts with "https://".

It is, however, important to double check the address bar you're putting your personal information into. Is it cloudflare.com or have you been redirected away to a dodgy website at cloudflair.com or cloudflare.net?

Despite how common encrypted web traffic has become, on many sites, it still remains relatively easy to strip away this encryption - by pointing internet traffic to a different address. I describe how this can be done in:Performing & Preventing SSL Stripping: A Plain-English Primer

It is also often good guidance to be careful about the links you see in emails; they legitimate emails from your bank, or just someone trying to capture your personal information from their fake "phishing" website that looks just like your bank? Just because someone has a little bit of information about you, doesn't mean they are who they say they are. When in doubt; void following links directly in emails, and check the validity of the email independently (such as by directly going to your banking website). A correct looking "to" address isn't enough to prove an email is coming from who it says it's from.

We always hear of new and innovative security vulnerabilities, but for most users, remembering a handful of simple security tips is enough to protect against the majority of security threats.

• As a rule-of-thumb, install the latest security patches

• Don't use obsolete software which doesn't provide security patches

• Use well-trusted anti-virus

• Back-up the files and folders you can't expect to lose

• Use a Password Manager to set random, unique passwords for every site

• Don't use common keywords or personal information as passwords

• Adding numbers and symbols to passwords often doesn't add security but impacts how easy they are to remember

• Enable Two-Factor Authentication on sites which support it

• Check the address bar when inputting personal information, make sure the connection is encrypted and the site address is correct

• Don't believe everything you see in your email inbox or trust every link sent through email; even if the sender has some information about you.

Finally, from everyone at Cloudflare, we wish you a wonderful and safe holiday season. For further reading, check out the Internet Mince Pie Database.

Brady Gentile from Cloudflare's product team wrote an App Developer Playbook, embedded within the developer documentation page. He decided to write it after he and his team conducted several app developer interviews, finding that many developers wanted to learn how to better promote their apps.

They wanted to help app authors out in the areas outside of developer core expertise. Social media posting, community outreach, email deployment, SEO, blog posting and syndication, etc. can be daunting.

I wanted to take a moment to highlight some of the tips from the App Developer Playbook because I think Brady did a great job of providing clear ways to approach promotional strategies.

Your app’s potential audience likely reads community-aggregated news sites such as HackerNews, Product Hunt, or reddit. Sharing your app across these websites is a great way for users to find your app.

For apps that are interesting to developers, designers, scientists, entrepreneurs, etc., be sure to share your work with the Hacker News community. Be sure to follow the official guidelines when posting and when engaging with the community. It may be tempting to ask your friends to upvote you, but honesty is the best policy, and the vote-ring detector will bury your post if you try to game it. Instead, if you don’t frontpage on the first try, consider re-posting on another day, with any of these options: the frontpage of your site, the blog post about the launch of your app, a demo of your app in action, a github repo. It may be worth taking into consideration the rate at which new posts are being added to /newest per minute or per hour, which impacts the likelihood of your post making it to the frontpage.

Since you’re sharing a project that people can play with, be sure to: 1) use “Show HN” and follow Show HN guidelines, and 2) be available to answer questions in the comments.

Be sure to start your title with the words ‘Show HN:’ (this indicates that you’ll be sharing something interesting that you’ve built with the HN community with a live demo people can try), then briefly explain your app within the same field. Rather than just use the name of your app, consider adding something informative, like the short description you use in your Cloudflare Apps marketplace tile. For instance, “Show HN: Trebble (embed voice and music on your site)” is more informative than “Show HN: Trebble” as a post title. Next, you’ll have the option of either submitting the URL of your app or explaining a little bit about yourself, the app, and pasting a link to the app itself.

Lastly, you should probably take the time to explain yourself and what you're all about in a first comment, as it helps build good rapport with the community. Block off some time on your calendar so you’re available to answer questions and engage with the community for however long your post is on the frontpage. In addition to gathering their valuable feedback, a signal that the app author is there (“Hi, I’m Name and I made this app to solve this problem -- I’d love to get your feedback.”) will often make your project more approachable and put a face on a product.

Product Hunt has released a blog post which outlines how to properly submit your app or product to their community. I highly recommended you review this post in its entirety prior to launching your Cloudflare App.

Submit a link to your app, along with some screenshots/videos, and a descriptive title for your post, and select a subreddit to post into. For the title of your post, you’ll want to use something descriptive about your app; for example you could say “I just built an app that does [X].”

If your app isn't relevant to the subreddit in which you post, it'll likely be removed by a moderator, so think carefully about which subreddits would find your app genuinely useful. I also recommend you take some time to engage with that community prior to posting your app, in part because their feedback is valuable, and in part so that you’re not a stranger. Here are two subreddits you should definitely include, though: Apps, Cloudflare.

One of the most important steps of the Cloudflare app deployment process is ensuring that both visitors browsing Cloudflare Apps and anyone doing a search on the web may quickly and easily find your app. By optimizing your Cloudflare app for discoverability, you’ll receive a greater number of views, installations, and revenue.

Your app’s title and short description are the first thing millions of website owners are going to see when coming across your Cloudflare app, whether it’s through browsing Cloudflare Apps or on a search engine. It’s important that an app’s title is unique, descriptive, and identifiable.

NoAdBlock is a great example.

Showcasing how your app might appear on a user’s website gives confidence to users thinking about previewing and installing. Include a variety of screenshots, showing multiple ways in which the software can be configured on a user’s website.

Read more about how to configure your full app description and categories in the App Developer Playbook.

Once your app has launched on Cloudflare Apps, it’s important that users are able to envision how your app will work for them and that they're easily able to use it.

Preview links allow you to generate a link to the install page for your app, which includes customization options for users to play around with.

Check out this preview for the Spotify app:

Make it easy and obvious for users. The Cloudflare Install Button is an interactive badge which can be embedded in any online assets, including websites and emails.

To use the full Cloudflare App install badge, you can paste the code listed in the Playbook onto your website or marketing page.You just need to replace [ appTitle ] and [ appId or appAlias ] with the appropriate details for your app. You can choose a standard button or customize it to your app.

Here's what NoAdBlock used:

A quick and easy way to announce your app’s availability is to notify your user base that the app is now available for them to preview and install. Read more in the Playbook on how to grow your user base before the launch.

Here's a good starting template for an email announcement:

Targeting users across multiple channels is a pretty easy way to ensure that website owners know your app is now available. Cloudflare can help you with this. Tag @Cloudflare in your posts, so your post will be retweeted and reshared.

Another way to promote the release of your app is by writing a blog post (or several) on your app’s website, delving into the features and benefits that your app brings to users. In addition to your launch post, you can enumerate the new features and bug fixes in a new and improved release, highlight different use cases from your own user base, or deep dive into a fascinating aspect of how you implemented your app.

Here's a well-written launch blog post, from the makers of Admiral.

Other blogs may help you with this as well. Syndication is a great way to gain significant exposure for your posts. Brainstorm a list of blogs facing the core audience for your app, and reach out and ask if you can contribute a guest blog post. If developers are the core audience, drop a line to community@cloudflare.com. I’d love to have a conversation about whether a guest post featuring your app would be right for the Cloudflare blog.

Again, this is just a glimpse into the guidance that the App Developer Playbook provides. Check it out and share it with your community of app developers.

Happy, productive app launching to you!

Earlier this year, Cloudflare Apps was launched so app developers may leverage our global network of 6 million+ websites, applications, and APIs. I’d like to take a moment to highlight Spotify, which was a launch partner for Cloudflare Apps, especially since they have elected to open source the code to their Cloudflare App.

Spotify Github repo »

Spotify is the leading digital service for streaming music, serving more than 140 million listeners.

Recently, Spotify launched a Cloudflare App to instantly and easily embed the Spotify player onto your website without having to copy / paste anything.

A musician who runs a site for their band - they can now play samples of new tracks on their tour calendar page and psych up their fans.

A game creator who wants to share their game's soundtrack with their fans.

An activewear company which wants to deliver popular running playlists to its customers.

Web properties that install the Spotify app have the ability to increase user engagement.

Add Spotify widgets to your web pages and let your users play tracks and follow Spotify profiles. Add a Spotify Play Button to your blog, website, or social page; all your fans have to do is hit “Play” to enjoy the music. You can create Play Buttons for albums, tracks, artists, or playlists.

When a logged-in Spotify user clicks the button on your page, the music will start playing in the Spotify player. If the user isn’t logged into their Spotify account, the play button will play a 30-second audio preview of the music and they will be prompted to login or sign up.

You can customize your button as well as link to any song or album you prefer in Spotify’s music catalog or to a playlist you’ve generated. Take a look at the preview of how the Spotify app would appear on a site here:

The Cloudflare App creator allows you to preview the app on your site without making any changes to your code.

In the left pane, you can see the install options where you can select what kind of widget you’d like displayed: a playlist, a track, or a follow button. You can customize the size, theme and position of the banner on your site. The “Pick a location” tool uses CSS selectors to allow you to pinpoint the location on your site where it’s displayed.

In the right pane, you can preview your choices, seeing what they’d look like on your website and experiment with placement and how it flows with the site. This is very similar to the tool that the app developer uses to test the app for how it behaves on a wide range of web properties.

Play with the Spotify Preview now »

Our friends at Spotify made their code available on GitHub. You can clone and fork the repository here. It’s a great way to get some practice developing Cloudflare Apps and to start with some basic scaffolding for your app.

Check out the documentation for Cloudflare Apps here.

Check out Cloudflare’s new App Developer Playbook, a step-by-step marketing guide for Cloudflare app developers here.

It’s with that in mind that we wanted to give Cloudflare users a more centralized location to gather, and to discuss all things Cloudflare. So we have launched a new Cloudflare Community at community.cloudflare.com.

It's for anyone and everyone who uses Cloudflare. Whether you are adding your first domain and don’t know what a name server is, or you are managing 1,000s of domains via API, or you are somewhere in between. In the Cloudflare Community you will be able to find tips, tricks, troubleshooting guidance, and recommendations.

We also think this will be a great way to get feedback from users on what’s working for them, what isn’t, and ways that we can make Cloudflare better. There will even be opportunities to participate in early access programs for new and evolving features.

Anyone can visit community.cloudflare.com and look around, soaking in all the information and expertise available, but in order to post questions or comments you must have a Cloudflare account. We wanted to keep this forum focused on using and improving Cloudflare, and not about questions from people who visit sites that use Cloudflare services. When users visit the forum and try to sign-in for the first time they will go through the usual Cloudflare login process, but there will be an added step for email verification. Once that is done users are good to go.

We’ve started off with some broad categories like Performance, Security, Product Feedback, etc., and created some tags to cover some of the more specific products and topics. In time, we could expand to more dedicated categories around things like SSL or WAF, but we didn’t want to separate things off too much up front. There is also a Meta category where members can direct questions or suggestions about the Community. So, just put your topic in the area that you think is best, throw on a tag or two if necessary, and we’ll all figure this out together. But don’t forget to search and see if someone’s already discussing it.

We aren’t degrading our presence on social media or other popular discussion sites. But we are hopeful that having a centralized location will make it even easier for people who want to get together and discuss their Cloudflare experiences to find the conversations they are looking for. So be sure to visit community.cloudflare.com and say hello!

A key Cloudflare feature is caching, which allows content to be served closer to the end user from our global network of data centers. Doing so improves the user's shopping experience and contributes to increasing the proportion of people completing a purchase (conversion rate).

For example:

• Walmart found improving page load time by 1 second increased their conversion rate by 2%

• Research for Amazon showed every 0.1 second of delay costs 1% of sales

• The Barack Obama campaign website saw an 80% page load time boost resulted in a 14% increase in donations

Cloudflare operates over 110 data centers around the world. When a website implements Cloudflare, visitor requests for the site will proxy through the nearest Cloudflare data center instead of connecting directly to the webserver hosting the site (origin). This means Cloudflare can store content such as images, JavaScript, CSS and HTML on our servers, speeding up access to those resources for end-users.

Most ecommerce websites rely on a backend database containing product descriptions and metadata such as prices. Without caching, each visit to a product page might involve several database requests to pull all the required data, which can introduce added latency to page load time, particularly on a busy website. Serving the website's homepage and product pages from Cloudflare's cache not only eliminates these costly database calls, but also reduces the load on your origin infrastructure.

To make the most of Cloudflare and to help maximize the speed of your website, serve as much content as possible from the Cloudflare cache.

By default, Cloudflare caches static content based on a fixed list of file extensions which includes assets such as images, CSS files and PDFs.

The reason Cloudflare only caches static content out of the box (and does not cache HTML content by default) is to avoid the risk of inappropriate data being cached. For example, if the shopping cart page is cached, then the next visitor might receive the cached version and see a cart with the incorrect contents. Therefore, while enabling more caching will let you make the most of Cloudflare, it requires careful and considered implementation.

Additional caching on Cloudflare can be enabled in one of two ways - using Page Rules or by sending cache headers from your origin. These two methods are explained in more detail here. In this blog post we’ll use Page Rules, but keep in mind you can use headers from your origin too.

A typical HTML page on an ecommerce website will contain static content (such as the product description) and dynamic content such as:

• a header section which varies according to the visitor’s logged in state - e.g. if the user is logged in, it may offer the user a “Logged in as..." message

• a basket section which populates as the user shops on the site

The user might have one or more session cookies to maintain these dynamic elements.

There are few ways to make the most of Cloudflare's caching, while taking into account the dynamic nature of ecommerce websites.

Note: the Bypass Cache on Cookie feature is only available on the Cloudflare Business and Enterprise plans

Many visitors to a site will be brand new, first time visitors - in other words, they won’t be logged in to the site and won’t have any items in their basket.

Serving their request from the Cloudflare cache means they can quickly view the page they’re looking for (whether the homepage or a specific product page). As they're a brand new visitor, the entire page can be served from the Cloudflare cache.

With most ecommerce platforms, as soon as the user logs in to the site or adds an item to basket, a relevant cookie is sent to the browser.

Cloudflare can cache the pages, but will bypass the cache should Cloudflare receive either of the cookies from the browser.

This is achieved by introducing a Page Rule with a “Bypass Cache on Cookie” setting:

In the above example, the Page Rule will cause all requests to the site to serve from cache, unless the web browser has sent a cookie named “loggedin” or “iteminbasket”.

Obviously every ecommerce platform is different, so always think through your settings and ensure you use the correct cookie values to ensure that there is no risk of private data (e.g. someone’s shopping basket) being served from cache and shown to another visitor.

A better solution would be to serve the entirety of the page from cache, but populate the dynamic elements using JavaScript / AJAX.

This means Cloudflare will serve the bulk of the page content and only small requests will pass (via Cloudflare) direct to origin to populate dynamic elements such as the basket contents.

To configure this, use a Page Rule with Cache Level “Cache Everything” for the static content and another Page Rule with Cache Level “Bypass” for the dynamic (AJAX) requests.

In this example, any requests going to www.example.com/ajax/basket_contents.php and www.example.com/ajax/logged-in-state.php would match the first Page Rule, which has cache level “Bypass” - Cloudflare will proxy the request but the request won't touch the Cloudflare cache.

Other requests, e.g. to www.example.com/products/product_page would not match the first Page Rule but would instead match the second “Cache Everything” Page Rule - thus the product page is served from the Cloudflare cache. Within that product page, the dynamic elements (such as the basket contents and the logged in state) are dynamically populated using the AJAX requests.

You should also consider introducing additional appropriate Page Rules for special pages such as the checkout pages - for example, you may wish to create a Page Rule that bypasses the cache for all the checkout pages.

Remember: only one Page Rule will execute for any given request, and Page Rules are processed in the order they exist in the Cloudflare control panel. Read over our Page Rules tutorial to better understand how they work.

Note: the Railgun feature is only available on the Cloudflare Business and Enterprise plans

Cloudflare’s Railgun technology optimizes the connection between Cloudflare and the website origin, for accelerating dynamic HTML content - content that can't be served from the Cloudflare cache.

Railgun helps in two ways:

• Establishing a persistent connection between Cloudflare and the website origin (to speed up initial connection times)

• Compressing the data that passes from the origin to Cloudflare by only sending content that has changed

Before Railgun:

After Railgun:

Railgun can happily be used in conjunction with the previously discussed caching methods.

If you’ve implemented method 1 (the bypass cache on cookie method) then Railgun will accelerate the requests which pass directly to origin due to the presence of the relevant bypass cache cookies.

Method 2 (caching everything on Cloudflare except AJAX calls to populate dynamic sections) is already more efficient than method 1. Railgun can still be used to further accelerate the AJAX requests that pass from Cloudflare to origin.

Railgun is a little more advanced as it requires installation of a small software package on (or very close to) the origin webserver to handle the compression. You can read more about Railgun here and find the installation documentation here.

Ideally a well optimized ecommerce website will leverage our caching service as much as possible - serving images, CSS and JavaScript from Cloudflare's network, in addition to as much static HTML content as possible. Adding our Railgun service to accelerate those inevitable non-cachable requests to the origin webserver will help create a fantastic, speedy shopping experience for your customers.

First, we'll explain the basics of the Web Cache Deception attack. For those who want a more in-depth explanation, Omer's original post is a great resource.

CC BY-SA 2.0 - image by shelleygibb

Imagine that you run the social media website example.com, and that each of your users has a newsfeed at example.com/newsfeed. When a user navigates to their newsfeed, the HTTP request generated by their browser might look something like this:

GET /newsfeed HTTP/1.1
Host: example.com
...

If you use Cloudflare, you don't want us to cache this request because if we did, some of your users might start seeing other users' newsfeeds instead of their own, which would be very bad. Luckily, as we'll explain below, this request won't be cached because the path in the request (the /newsfeed part) doesn't have a "cacheable file extension" (a file extension such as .jpg or .css that instructs Cloudflare that it's OK to cache the request).

The trouble begins if your website is configured to be flexible about what kinds of paths it can handle. In particular, the issue arises when requests to a path that doesn't exist (say, /x/y/z) are treated as equivalent to requests to a parent path that does exist (say, /x). For example, what happens if you get a request for the nonexistent path /newsfeed/foo? Depending on how your website is configured, it might just treat such a request as equivalent to a request to /newsfeed. For example, if you're running the Django web framework, the following configuration would do just that because the regular expression ^newsfeed/ matches both newsfeed/ and newsfeed/foo (Django routes omit the leading /):

from django.conf.urls import url

patterns = [url(r'^newsfeed/', ...)]

And here's where the problem lies. If your website does this, then a request to /newsfeed/foo.jpg will be treated as the same as a request to /newsfeed. But Cloudflare, seeing the .jpg file extension, will think that it's OK to cache this request.

Now, you might be thinking, "So what? My website never has any links to /newsfeed/foo.jpg or anything like that." That's true, but that doesn't stop other people from trying to convince your users to visit paths like that. For example, an attacker could send this message to somebody:

Hey, check out this cool link! https://example.com/newsfeed/foo.jpg

If the recipient of the message clicks on the link, they will be taken to their newsfeed. But when the request passes through Cloudflare, since the path ends in `.jpg`, we will cache it. Then the attacker can visit the same URL themselves and their request will be served from our cache, exposing your user's sensitive content.

The best way to defend against this attack is to ensure that your website isn't so permissive, and never treats requests to nonexistent paths (say, /x/y/z) as equivalent to requests to valid parent paths (say, /x). In the example above, that would mean that requests to /newsfeed/foo or /newsfeed/foo.jpg wouldn't be treated as equivalent to requests to /newsfeed, but would instead result in some kind of error or a redirect to a legitimate page. If we wanted to modify the Django example from above, we could add a $ to the end of the regular expression to ensure only exact matches (in this case, a request to /newsfeed/foo will result in a 404):

We provide many settings that allow you to customize the way our cache will treat requests to your website. For example, if you have a Page Rule enabled for /newsfeed with the Cache Everything setting enabled (it's off by default), then we'll cache requests to /newsfeed, which could be bad. Thus, the best way to ensure that your website is secure is to understand the rules that our cache uses to determine whether or not a request should be cached.

When a request comes in to our network, we perform two phases of processing in order to determine whether or not to cache the origin's response to that request:

• In the eligibility phase, which is performed when a request first reaches our edge, we inspect the request to determine whether it should be eligible for caching. If we determine that it is not eligible, then we will not cache it. If we determine that it is eligible, then we proceed to a second disqualification phase.

• In the disqualification phase, which is performed after we've received a response from the origin web server, we inspect the response to determine whether any characteristics disqualify the response from being cached. If nothing disqualifies it, then the response will be cached.

_Configuring caching via a Page Rule_

Note that site-wide settings or Page Rules can affect this logic. Below, when we say "a setting applies" or "the setting is," we mean that either a global setting exists which applies to all requests or a Page Rule with the setting exists that applies to the given request (e.g., a Page Rule for /foo/* applies to requests to /foo/bar, /foo/baz, /foo/bar/baz, etc). Page Rules override global rules if both apply to a given request.

In the eligibility phase, we use characteristics of the request from the client to determine whether or not the request is eligible to be cached. If the request is not eligible, then it will not be cached. If the request is eligible, then we will perform more processing later in the disqualification phase.

The rules for eligibility are as follows:

• If the setting is Standard, Ignore Query String, or No Query String, then:

  • a request is eligible to be cached if the requested path ends in one of the file extensions listed in Figure 1 below

  • a request may be eligible to be cached (depending on performance-related decisions made by our edge) if the requested path ends in one of the file extensions listed in Figure 2 below

  • a request is eligible to be cached if the request path is /robots.txt

• If the setting is Cache Everything, then all requests are eligible to be cached.

In addition to the above rules, if either of the following two conditions hold, then any decision made so far about eligibility will be overridden, and the request will not be eligible to be cached:

• If the Cache on Cookie setting is enabled and the configured cookie is not present in a Cookie header, then the request is not eligible to be cached.

• If the Bypass Cache on Cookie setting is enabled and the configured cookie is present in a Cookie header, then the request is not eligible to be cached.

In the disqualification phase, which only occurs if a request has been marked as eligible, characteristics of the response from the origin web server can disqualify a request. If a request is disqualified, then the response will not be cached. If a request is not disqualified, then the response will be cached.

The rules for disqualification are as follows:

• If the setting is Standard, Ignore Query String, or No Query String, or if the setting is Cache Everything and no Edge Cache TTL is present, then:

  • A Cache-Control header in the response from the origin with any of the following values will disqualify a request, causing it not to be cached:

    • no-cache

    • max-age=0

    • private

    • no-store

  • An Expires header in the response from the origin indicating any time in the past will disqualify a request, causing it not to be cached.

• If the setting is Cache Everything and an Edge Cache TTL is present, then a request will never be disqualified under any circumstances, and will always be cached.

There is one further set of rules relating to the Set-Cookie header. The following rules only apply if a Set-Cookie header is present:

• If the setting is Standard, Ignore Query String, or No Query String, or if the setting is Cache Everything and an Edge Cache TTL is present, then the request will not be disqualified, but the Set-Cookie header will be stripped from the version of the response stored in our cache.

• If the setting is Cache Everything and no Edge Cache TTL is present, then the request will be disqualified, and it will not be cached. The Set-Cookie header will be stripped from the response that is sent to the client making the request.

class
css
jar
js
jpg
jpeg
gif
ico
png
bmp
pict
csv
doc
docx
xls
xlsx
ps
pdf
pls
ppt
pptx
tif
tiff
ttf
otf
webp
woff
woff2
svg
svgz
eot
eps
ejs
swf
torrent
midi
mid

_Figure 1: Always Cacheable File Extensions_

mp3
mp4
mp4v
mpg
mpeg
mov
mkv
flv
webm
wmv
avi
ogg
m4a
wav
aac
ogv
zip
sit
tar
7z
rar
rpm
deb
dmg
iso
img
msi
msp
msm
bin
exe
dll
ra
mka
ts
m4v
asf
mk3d
rm
swf

_Figure 2: Sometimes Cacheable File Extensions_

So there you have it. So long as you make sure to follow the advice above, and make sure that your site plays nicely with our cache, you should be secure against the Web Cache Deception attack.

The North American Network Operators Group (NANOG) is the loci of modern Internet innovation and the day-to-day cumulative network-operational knowledge of thousands and thousands of network engineers. NANOG itself is a non-profit membership organization; but you don’t need to be a member in order to attend the conference or join the mailing list. That said, if you can become a member, then you’re helping a good cause.

The next NANOG conference starts in a few days (February 6-8 2017) in Washington, DC. Nearly 900 network professionals are converging on the city to discuss a variety of network-related issues, both big and small; but all related to running and improving the global Internet. For this upcoming meeting, Cloudflare has three network professionals in attendance. Two from the San Francisco office and one from the London office.

With the conference starting next week, it seemed a great opportunity to introduce readers of the blog as to why a NANOG conference is so worth attending.

While it seems obvious how to do some network tasks (you unpack the spiffy new wireless router from its box, you set up its security and plug it in); alas the global Internet is somewhat more complex. Even seasoned professionals could do with a recap on how traceroute actually works, or how DNSSEC operates, or this years subtle BGP complexities, or be enlightened about Optical Networking. All this can assist you with deployments within your networks or datacenter.

If there’s one thing that keeps the Internet (a network-of-networks) operating, it’s peering. Peering is the act of bringing together two or more networks to allow traffic (bits, bytes, packets, email messages, web pages, audio and video streams) to flow efficiently and cleanly between source and destination. The Internet is nothing more than a collection of individual networks. NANOG provides one of many forums for diverse network operators to meet face-to-face and negotiate and enable those interconnections.

While NANOG isn’t the only event that draws networks together to discuss interconnection, it’s one of the early forums to support these peering discussions.

In this day-and-age we are brutally aware that security is the number-one issue when using the Internet. This is something to think about when you choose your email password, lock screen password on your laptop, tablet or smartphone. Hint: you should always have a lock screen!

At NANOG the security discussion is focused on a much deeper part of the global Internet, in the very hardware and software practices, that operate and support the underlying networks we all use on a daily basis. An Internet backbone (rarely seen) is a network that moves traffic from one side of the globe to the other (or from one side of a city to the other). At NANOG we discuss how that underlying infrastructure can operate efficiently, securely, and be continually strengthened. The growth of the Internet over the last handful of decades has pushed the envelope when it comes to hardware deployments and network-complexity. Sometimes it only takes one compromised box to ruin your day. Discussions at conferences like NANOG are vital to the sharing of knowledge and collective improvement of everyone's networks.

Above the hardware layer (from a network stack point of view) is the Domain Name System (DNS). DNS has always been a major subject of discussion within the NANOG community. It’s very much up to the operational community to make sure that when you type a website name into web browser or type in someone’s email address into your email program that there’s a highly efficient process to convert from names to numbers (numbers, or IP address, are the address book and routing method of the Internet). DNS has had its fair share of focus in the security arena and it comes down to network operators (and their system administrator colleagues) to protect DNS infrastructure.

Nearly everyone knows that bad news sells. It’s a fact. To be honest, the same is the case in the network operator community. However, within NANOG, those stories of disasters are nearly always told from a learning and improvement point of view. There’s simply no need to repeat a failure, no-one enjoys it a second time around. Notable stories have included subjects like route-leaks, BGP protocol hiccups, peering points, and plenty more.

We simply can’t rule out failures within portions of the network; hence NANOG has spent plenty of time discussing redundancy. The internet operates using routing protocols that explicitly allow for redundancy in the paths that traffic travels. Should a failure occur (a hardware failure, or a fiber cut), the theory is that the traffic will be routed around that failure. This is a recurring topic for NANOG meetings. Subsea cables (and their occasional cuts) always make for good talks.

While we learned twenty or more years ago how to type into Internet routers on the command line, those days are quickly becoming history. We simply can’t scale if network operational engineers have to type the same commands into hundreds (or thousands?) of boxes around the globe. We need automation. This is where NANOG has been a leader in this space. Cloudflare has been active in this arena and Mircea Ulinic presented our experience for Network Automation with Salt and NAPALM at the previous NANOG meeting. Mircea (and Jérôme Fleury) will be giving a follow-up in-depth tutorial on the subject at next week’s meeting.

The first NANOG conference was held June 1994 in Ann Arbor, Michigan and the conference has grown significantly since then. While it’s fun to follow the history, it’s maybe more important to realize that NANOG has covered a multitude of subjects since that start. Go scan the archives at nanog.org and/or watch some of the online videos.

Let’s not forget the advantages of spending time with other operators within a relaxed setting. After all, sometimes the big conversations happen when spending time over a beer discussing common issues. NANOG has long understood this and it’s clear that the Tuesday evening Beer ’n Gear social is set up specifically to let network geeks both grab a drink (soft drinks included) and poke around with the latest and greatest network hardware on show. The social is as much about blinking lights on shiny network boxes as it is about tracking down that network buddy.

Oh; there’s a fair number of vendor giveaways (so far there’s 15 hardware and software vendors signed up for next week’s event). After all, who doesn’t need a new t-shirt?

But there’s more to the downtime and casual hallway conversations. For myself (the author of this blog), I know that sometimes the most important work is done within the hallways during breaks in the meeting vs. standing in front of the microphone presenting at the podium. The industry has long recognized this and the NANOG organizers were one of the early pioneers in providing full-time coffee and snacks that cover the full conference agenda times. Why? Because sometimes you have to step out of the regular presentations to meet and discuss with someone from another network. NANOG knows its audience!

NANOG isn’t the only forum to discuss network operational issues, however it’s arguably the largest. It started off as a “North American” entity; however, in the same way that the Internet doesn’t have country barriers, NANOG meetings (which take place in the US, Canada and at least once in the Caribbean) have fostered an online community that has grown into a global resource. The mailing list (well worth reviewing) is a bastion of networking discussions.

In a different realm, the Internet Engineering Task Force (IETF) focuses on protocol standards. Its existence is why diverse entities can communicate. Operators participate in IETF meetings; however, it’s a meeting focused outside of the core operational mindset.

Central to the Internet’s existence is ICANN. Meeting three times a year at locations around the globe, it focused on the governance arena and in domain names and related items. Within the meetings there’s an excellent Tech Day.

In the numbers arena ARIN is an example of a regional routing registry (an RIR) that runs members meeting. An RIR deals with allocating resources like IP addresses and AS numbers. ARIN focuses on the North American area and sometimes holds its meetings alongside NANOG meetings.

ARIN’s counterparts in other parts of the world also hold meetings. Sometimes they simply focus on resource policy and sometimes they also focus on network operational issues. For example RIPE (in Europe, Central Asia and the Middle East) runs a five-day meeting that covers operational and policy issues. APNIC (Asia Pacific), AFRINIC (Africa), LACNIC (Latin America & Caribbean) all do similar variations. There isn’t one absolute method and that's a good thing. It’s worth pointing out that APNIC holds it’s members meetings once a year in conjunction with APRICOT which is the primary operations meeting in the Asia Pacific region.

While NANOG is somewhat focused on North America, there are also the regional NOGs. These regional NOGs are vital to help the education of network operators globally. Japan has JANOG, Southern Africa has SAFNOG, MENOG in the Middle East, AUSNOG & NZNOG in Australia & New Zealand, DENOG in Germany, PHNOG in the Philippines, and just to be different, the UK has UKNOF (“Forum” vs. “Group”). It would be hard to list them all; but each is a worthwhile forum for operational discussions.

Peering specific meetings also exist. Global Peering Forum, European Peering Forum, and Peering Forum de LACNOG for example. Those focus on bilateral meetings within a group of network operators or administrators and specifically focus on interconnect agreements.

In the commercial realms there’s plenty of other meetings that are attended by networks like Cloudflare. PTC and International Telecoms Week (ITW) are global telecom meetings specifically designed to host one-to-one (bilateral) meetings. They are very commercial in nature and less operational in focus.

As you would guess, you will find our network team at RIR meetings, sometimes at IETF meetings, sometimes at ICANN meetings, often at various regional NOG meetings (like SANOG in South East Asia, NoNOG in Norway, RONOG in Romania, AUSNOG/NZNOG in Australia/New Zealand and many other NOGs). We get around; however, we also run a global network and we need to interact with many many networks around the globe. These meetings provide an ideal opportunity for one-to-one discussions.

If you've heard something you like from Cloudflare at one of these operational-focused conferences, then check out our jobs listings (in various North American cities, London, Singapore, and beyond!)

On Wednesday, September 21st, 2016 at 10am PT/1pm ET join SANS Technology Institute instructor and courseware author, David Holzer, as well as CloudFlare Solutions Engineer, Matthew Silverlock, as they discuss best practices for adopting and deploying microservices securely. During the session they will cover:

• How microservices differ from SOA or monolithic architectures

• Best practices for adopting and deploying secure microservices for production use

• Avoiding continuous delivery of new vulnerabilities

• Limiting attack vectors on a growing number of API endpoints

• Protecting Internet-facing services from resource exhaustion

Don't miss this chance to learn from the pros. Register now!

CC BY 2.0 image by Chris Combe

522 error on CloudFlare indicates a connection issue between our edge server and the origin server. Most often the blame is on the origin server side - the origin server is slow, offline or encountering high packet loss. Less often the problem is on our side.

In the case I was debugging it was neither. The internet connectivity between CloudFlare and origin was perfect. No packet loss, flat latency. So why did we see a 522 error?

The root cause of this issue was pretty complex. After a lot of debugging we identified an important symptom: sometimes, once in thousands of runs, our test program failed to establish a connection between two daemons on the same machine. To be precise, an NGINX instance was trying to establish a TCP connection to our internal acceleration service on localhost. This failed with a timeout error.

Once we knew what to look for we were able to reproduce this with good old netcat. After a couple of dozen of runs this is what we saw:

$ nc 127.0.0.1 5000  -v
nc: connect to 127.0.0.1 port 5000 (tcp) failed: Connection timed out

The view from strace:

socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("127.0.0.1")}, 16) = -110 ETIMEDOUT

netcat calls connect() to establish a connection to localhost. This takes a long time and eventually fails with ETIMEDOUT error. Tcpdump confirms that connect() did send SYN packets over loopback but never received any SYN+ACKs:

$ sudo tcpdump -ni lo port 5000 -ttttt -S
00:00:02.405887 IP 127.0.0.12.59220 > 127.0.0.1.5000: Flags [S], seq 220451580, win 43690, options [mss 65495,sackOK,TS val 15971607 ecr 0,nop,wscale 7], length 0
00:00:03.406625 IP 127.0.0.12.59220 > 127.0.0.1.5000: Flags [S], seq 220451580, win 43690, options [mss 65495,sackOK,TS val 15971857 ecr 0,nop,wscale 7], length 0
... 5 more ...

Hold on. What just happened here?

Well, we called connect() to localhost and it timed out. The SYN packets went off over loopback to localhost but were never answered.

CC BY 2.0 image by akj1706

The first thought is about Internet stability. Maybe the SYN packets were lost? A little known fact is that it's not possible to have any packet loss or congestion on the loopback interface. The loopback works magically: when an application sends packets to it, it immediately, still within the send syscall handling, gets delivered to the appropriate target. There is no buffering over loopback. Calling send over loopback triggers iptables, network stack delivery mechanisms and delivers the packet to the appropriate queue of the target application. Assuming the target application has some space in its buffers, packet loss over loopback is not possible.

Under normal circumstances connections to localhost are not supposed to time out. There is one corner case when this may happen though - when the listening application does not call accept() fast enough.

When that happens, the default behavior is to drop the new SYN packets. If the listening socket has a full accept queue, then new SYN packets will be dropped. The intention is to cause push-back, to slow down the rate of incoming connections. The peers should eventually re-send SYN packets, and hopefully by that time the accept queue will be freed. This behavior is controlled by the tcp_abort_on_overflow sysctl.

But this accept queue overflow did not happen in our case. Our listening application had an empty accept queue. We checked this with the ss command:

$ ss -n4lt 'sport = :5000'
State      Recv-Q Send-Q  Local Address:Port    Peer Address:Port
LISTEN     0      128                 *:5000               *:*

The Send-Q column shows the backlog / accept queue size given to listen() syscall - 128 in our case. The Recv-Q reports on the number of outstanding connections in the accept queue - zero.

To recap: we are establishing connections to localhost. Most of them work fine but sometimes the connect() syscall times out. The SYN packets are being sent over loopback. Because it's loopback they are being delivered to the listening socket. The listening socket accept queue is empty, but we see no SYN+ACKs.

Further investigation revealed something peculiar. We noticed hundreds of CLOSE_WAIT sockets:

$ ss -n4t | head
State      Recv-Q Send-Q  Local Address:Port    Peer Address:Port
CLOSE-WAIT 1      0           127.0.0.1:5000       127.0.0.1:36599
CLOSE-WAIT 1      0           127.0.0.1:5000       127.0.0.1:36467
CLOSE-WAIT 1      0           127.0.0.1:5000       127.0.0.1:36154
CLOSE-WAIT 1      0           127.0.0.1:5000       127.0.0.1:36412
CLOSE-WAIT 1      0           127.0.0.1:5000       127.0.0.1:36536
...

CC BY 2.0 image by DaveBleasdale

Citing the Red Hat docs:

CLOSE_WAIT - Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed. This means the socket is waiting for the application to execute close(). A socket can be in CLOSE_WAIT state indefinitely until the application closes it. Faulty scenarios would be like a file descriptor leak: server not executing close() on sockets leading to pile up of CLOSE_WAIT sockets.

This makes sense. Indeed, we were able to confirm the listening application leaks sockets. Hurray, good progress!

The leaking sockets don't explain everything though.

Usually a Linux process can open up to 1,024 file descriptors. If our application did run out of file descriptors the accept syscall would return the EMFILE error. If the application further mishandled this error case, this could result in losing incoming SYN packets. Failed accept calls will not dequeue a socket from accept queue, causing the accept queue to grow. The accept queue will not be drained and will eventually overflow. An overflowing accept queue could result in dropped SYN packets and failing connection attempts.

But this is not what happened here. Our application hasn't run out of file descriptors yet. This can be verified by counting file descriptors in /proc/<pid>/fd directory:

$ ls /proc/` pidof listener `/fd | wc -l
517

517 file descriptors are comfortably far from the 1,024 file descriptor limit. Also, we earlier showed with ss that the accept queue is empty. So why did our connections time out?

The root cause of the problem is definitely our application leaking sockets. The symptoms though, the connection timing out, are still unexplained.

Time to raise the curtain of doubt. Here is what happens.

The listening application leaks sockets, they are stuck in CLOSE_WAIT TCP state forever. These sockets look like (127.0.0.1:5000, 127.0.0.1:some-port). The client socket at the other end of the connection is (127.0.0.1:some-port, 127.0.0.1:5000), and is properly closed and cleaned up.

When the client application quits, the (127.0.0.1:some-port, 127.0.0.1:5000) socket enters the FIN_WAIT_1 state and then quickly transitions to FIN_WAIT_2. The FIN_WAIT_2 state should move on to TIME_WAIT if the client received FIN packet, but this never happens. The FIN_WAIT_2 eventually times out. On Linux this is 60 seconds, controlled by net.ipv4.tcp_fin_timeout sysctl.

This is where the problem starts. The (127.0.0.1:5000, 127.0.0.1:some-port) socket is still in CLOSE_WAIT state, while (127.0.0.1:some-port, 127.0.0.1:5000) has been cleaned up and is ready to be reused. When this happens the result is a total mess. One part of the socket won't be able to advance from the SYN_SENT state, while the other part is stuck in CLOSE_WAIT. The SYN_SENT socket will eventually give up failing with ETIMEDOUT.

It all starts with a listening application that leaks sockets and forgets to call close(). This kind of bug does happen in complex applications. An example buggy code is available here. When you run it nothing will happen initially. ss will show a usual listening socket:

$ go build listener.go && ./listener &
$ ss -n4tpl 'sport = :5000'
State      Recv-Q Send-Q  Local Address:Port    Peer Address:Port
LISTEN     0      128                 *:5000               *:*      users:(("listener",81425,3))

Then we have a client application. The client behaves correctly - it establishes a connection and after a while it closes it. We can demonstrate this with nc:

$ nc -4 localhost 5000 &
$ ss -n4tp '( dport = :5000 or sport = :5000 )'
State      Recv-Q Send-Q  Local Address:Port    Peer Address:Port
ESTAB      0      0           127.0.0.1:5000       127.0.0.1:36613  users:(("listener",81425,5))
ESTAB      0      0           127.0.0.1:36613      127.0.0.1:5000   users:(("nc",81456,3))

As you see above ss shows two TCP sockets, representing the two ends of the TCP connection. The client one is (127.0.0.1:36613, 127.0.0.1:5000), the server one (127.0.0.1:5000, 127.0.0.1:36613).

The next step is to gracefully close the client connection:

$ kill `pidof nc`

Now the connections enter TCP cleanup stages: FIN_WAIT_2 for the client connection, and CLOSE_WAIT for the server one (if you want to read more about these TCP states here's a recommended read):

$ ss -n4tp
State      Recv-Q Send-Q  Local Address:Port    Peer Address:Port
CLOSE-WAIT 1      0           127.0.0.1:5000       127.0.0.1:36613  users:(("listener",81425,5))
FIN-WAIT-2 0      0           127.0.0.1:36613      127.0.0.1:5000

After a while FIN_WAIT_2 will expire:

$ ss -n4tp
State      Recv-Q Send-Q  Local Address:Port    Peer Address:Port
CLOSE-WAIT 1      0           127.0.0.1:5000       127.0.0.1:36613  users:(("listener",81425,5))

But the CLOSE_WAIT socket stays in! Since we have a leaked file descriptor in the listener program, the kernel is not allowed to move it to FIN_WAIT state. It is stuck in CLOSE_WAIT indefinitely. This stray CLOSE_WAIT would not be a problem if only the same port pair was never reused. Unfortunately, it happens and causes the problem.

To see this we need to launch hundreds of nc instances and hope the kernel will assign the colliding port number to one of them. The affected nc will be stuck in connect() for a while:

$ nc -v -4 localhost 5000 -w0
...

We can use the ss to confirm that the ports indeed collide:

SYN-SENT   0  1   127.0.0.1:36613      127.0.0.1:5000   users:(("nc",89908,3))
CLOSE-WAIT 1  0   127.0.0.1:5000       127.0.0.1:36613  users:(("listener",81425,5))

In our example the kernel allocated source address (127.0.0.1:36613) to the nc process. This TCP flow is okay to be used for a connection going to the listener application. But the listener will not be able to allocate a flow in reverse direction since (127.0.0.1:5000, 127.0.0.1:36613) from previous connections is still being used and remains with CLOSE_WAIT state.

The kernel gets confused. It retries the SYN packets, but will never respond since the other TCP socket is stick in the CLOSE_WAIT state. Eventually our affected netcat will die with unhappy ETIMEDOUT error message:

...
nc: connect to localhost port 5000 (tcp) failed: Connection timed out

If you want to reproduce this weird scenario consider running this script. It will greatly increase the probability of netcat hitting the conflicted socket:

$ for i in `seq 500`; do nc -v -4 -s 127.0.0.1 localhost 5000 -w0; done

A little known fact is that the source port automatically assigned by the kernel is incremental, unless you select the source IP manually. In such case the source port is random. This bash script will create a minefield of CLOSE_WAIT sockets randomly distributed across the ephemeral port range.

If there's a moral from the story it's to watch out for CLOSE_WAIT sockets. Their presence indicate leaking sockets, and with leaking sockets some incoming connections may time out. Presence of many FIN_WAIT_2 sockets says the problem is not on current machine but on the remote end of the connection.

Furthermore, this bug shows that it is possible for the states of the two ends of a TCP connection to be at odds, even if the connection is over the loopback interface.

It seems that the design decisions made by the BSD Socket API have unexpected long lasting consequences. If you think about it - why exactly the socket can automatically expire the FIN_WAIT state, but can't move off from CLOSE_WAIT after some grace time. This is very confusing... And it should be! The original TCP specification does not allow automatic state transition after FIN_WAIT_2 state! According to the spec FIN_WAIT_2 is supposed to stay running until the application on the other side cleans up.

Let me leave you with the tcp(7) manpage describing the tcp_fin_timeout setting:

tcp_fin_timeout (integer; default: 60)
      This specifies how many seconds to wait for a final FIN packet
      before the socket is forcibly closed.  This is strictly a
      violation of the TCP specification, but required to prevent
      denial-of-service attacks.

I think now we understand why automatically closing FIN_WAIT_2 is strictly speaking a violation of the TCP specification.

Do you enjoy playing with low level networking bits? Are you interested in dealing with some of the largest DDoS attacks ever seen?

If so you should definitely have a look at the open positions in our London, San Francisco, Singapore, Champaign (IL) and Austin (TX) offices!

In order to keep track of all our endpoints, we use a rich notation called JSON Hyper-Schema. These schemas are used to generate the complete HTML documentation that you can see at https://api.cloudflare.com. Today, we want to share a set of tools that we use in this process.

CC BY 2.0 image by Richard Martin

JSON Schema is a powerful way to describe your JSON data format. It provides complete structural validation and can be used for things like validation of incoming requests. JSON Hyper-Schema further extends this format with links and gives you a way describe your API.

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "number" },
    "address": {
      "type": "object",
      "properties": {
        "street_address": { "type": "string" },
        "city": { "type": "string" },
        "state": { "type": "string" },
        "country": { "type" : "string" }
      }
    }
  }
}

{
  "name": "John Doe",
  "age": 45,
  "address": {
    "street_address": "12433 State St NW",
    "city": "Atlanta",
    "state": "Georgia",
    "country": "United States"
  }
}

JSON Schema supports all simple data types. It also defines some special meta properties including title, description, default, enum, id, $ref, $schema, allOf, anyOf, oneOf, and more. The most powerful construct is $ref. It provides similar functionality to hypertext links. You can reference external schemas (external reference) or a fragment inside the current schema (internal reference). This way you can easily compose and combine multiple schemas together without repeating yourself.

JSON Hyper-Schema introduces another property called links where you define your API links, methods, request and response formats, etc. The best way to learn more about JSON Schemas is to visit Understanding JSON Schema. You can also visit the official specification website or wiki. If you want to jump straight into examples, try this.

CC BY 2.0 image by Tony Walmsley

We already have an open source library that can generate complete HTML documentation from JSON Schema files and Handlebars.js templates. It's called JSON Schema Docs Generator (JSDC). However, it has some drawbacks that make it hard to use for other teams:

• Complicated configuration

• It's necessary to rebuild everything with every change (slow)

• Templates cannot have their own dependencies

• All additional scripting must be in a different place

• It is hard to further customize it (splitting into sections, pages)

We wanted something more modular and extensible that addresses the above issues, while still getting ready-to-go output just with a few commands. So, we created a toolchain based on JSDC and modern JavaScript libraries. This article is not just a description for how to use these tools, but also an explanation of our design decisions. It is described in a bottom-up manner. You can skip to the bottom if you are not interested in the technical discussion and just want to get started using the tools.

JSON Schema files need to be preprocessed first. The first thing we have to do is to resolve their references ($ref). This can be quite a complex task since every schema can have multiple references, some of which are external (referencing even more schemas). Also, when we make a change, we want to only resolve schemas that need to be resolved. We decided to use Webpack for this task because a webpack loader has some great properties:

• It's a simple function that transforms input into output

• It can maintain and track additional file dependencies

• It can cache the output

• It can be chained

• Webpack watches all changes in required modules and their dependencies

Our loader uses the 3rd party JSON Schema Ref Parser library. It does not adhere to the JSON Schema specification related to id properties and their ability to change reference scope since it is ambiguous. However, it does implement the JSON Pointer and JSON Reference specifications. What does this mean? You can still combine relative (or absolute) paths with JSON Pointers and use references like:

 "$ref": "./product.json#/definitions/identifier"

but ids are simply ignored and the scope is always relative to the root. That makes reasoning about our schemas easier. That being said, a unique root id is still expected for other purposes.

Finally, we have resolved schemas. Unfortunately, their structure doesn't really match our final HTML documentation. It can be deeply nested, and we want to present our users with nice examples of API requests and responses. We need to do further transformations. We must remove some original properties and precompute new ones. The goal is to create a data structure that will better fit our UI components. Please check out the project page for more details.

You might be asking why we use another webpack loader and why this isn't part of our web application instead. The main reason is performance. We do not want to bog down browsers by doing these transformations repeatedly since JSON Schemas can be arbitrarily nested and very complex and the output can be precomputed.

With both of these webpack loaders, you can easily use your favorite JavaScript framework to build your own application. However, we want to make doc generation accessible even to people who don't have time to build their own app. So, we created a set of templates that match the output of json-schema-example-loader. These templates use the popular library React. Why React?

• It can be used and rendered server-side

• We can now bake additional features into components (e.g., show/hide...)

• It is easily composable

• We really really like it :)

doca-bootstrap-theme is a generic theme based on Twitter Bootstrap v3. We also have our private doca-cf-theme used by https://api.cloudflare.com. We encourage you to fork it and create your own awesome themes!

CC BY 2.0 image by Maia Coimbra

So, we have loaders and nice UI components. Now, it's time to put it all together. We have something that can do just that! We call it doca. doca is a command-line tool written in Node.js that scaffolds the whole application for you. It is actually pretty simple. It takes fine-tuned webpack/redux/babel based application, copies it into a destination of your choice, and does a few simple replacements.

Since all hard work is done by webpack loaders and all UI components live in a different theme package, the final app can be pretty minimal. It's not intended to be updated by the doca tool. You should only use doca once. Otherwise, it would just rewrite your application, which is not desirable if you made some custom modifications. For example, you might want to add React Router to create multi-page documentation.

doca contains webpack configs for development and production modes. You can build a completely static version with no JavaScript. It transforms the output of json-schema-example-loader into an immutable data structure (using Immutable.js). This brings some nice performance optimizations. This immutable structure is then passed to doca-bootstrap-theme (the default option). That's it.

This is a good compromise between ease of setup and future customization. Do you have a folder with JSON Schema files and want to quickly get index.html? Install doca and use a few commands. Do you need your own look? Fork and update doca-bootstrap-theme. Do you need to create more pages, sections, or use a different framework? Just modify the app that was scaffolded by doca.

One of the coolest features of webpack is hot module replacement. Once you save a file, you can immediately see the result in your browser. No waiting, refreshing, scrolling or lost state. It's mostly used in combination with React; however, we use it for JSON Schemas, too. Here's a demo:

It gets even better. It is easy to make a mistake in your schemas. No worries! You will be immediately prompted with a descriptive error message. Once it's fixed, you can continue with your work. No need to leave your editor. Refreshing is so yesterday!

The only prerequisite is to have Node.js v4+ on your system. Then, you can install doca with:

npm install doca -g

There are just two simple commands. The first one is doca init:

doca init [-i schema_folder] [-o project_folder] [-t theme_name]

It goes through the current dir (or schema_folder), looks for **/*.json files, and generates /documentation (or /project_folder). This command should be used only once when you need to bootstrap your project.

The second one is doca theme:

doca theme newTheme project

This gives a different theme (newTheme) to the project. It has two steps:

• It calls npm install newTheme --save inside of project

• It renames all doca-xxx-theme references to doca-newTheme-theme

This can make destructive changes in your project. Always use version control!

CC BY 2.0 image by Robert Couse-Baker

The best way how to start is to try our example. It includes two JSON Schemas.

git clone git@github.com:cloudflare/doca.git
cd doca/example
doca init
cd documentation
npm install
npm start
open http://localhost:8000

That's it! This results in a development environment where you can make quick changes in your schemas and see the effects immediately because of mighty hot reloading.

You can build a static production ready app with:

npm run build
open build/index.html

Or you can build it with no JavaScript using:

npm run build:nojs
open build/index.html

Do you need to add more schemas or change their order? Edit the file /schema.js.Do you want to change the generic page title or make curl examples nicer? Edit the file /config.js.

We're open sourcing a set of libraries that can help you develop and ship a rich RESTful API documentation. We are happy for any feedback and can't wait to see new themes created by the open source community. Please, gives us a star on GitHub. Also, if this work interests you, then you should come join our team!

CC BY 2.0 image by jim simonson

Then we published a set of utilities we are using to generate the BPF rules for our production iptables: the bpftools.

Today we are very happy to open source another component of the bpftools: our p0f BPF compiler!

p0f is a tool written by superhuman Michal Zalewski.The main purpose of p0f is to passively analyze and categorize arbitrary network traffic. You can feed p0f any packet and in return it will derive knowledge about the operating system that sent the packet.

One of the features that caught our attention was the concise yet explanatory signature format used to describe TCP SYN packets.

The p0f SYN signature is a simple string consisting of colon separated values. This string cleanly describes a SYN packet in a human-readable way. The format is pretty smart, skipping the varying TCP fields and keeping focus only on the essence of the SYN packet, extracting the interesting bits from it.

We are using this on daily basis to categorize the packets that we, at CloudFlare, see when we are a target of a SYN flood. To defeat SYN attacks we want to discriminate the packets that are part of an attack from legitimate traffic. One of the ways we do this uses p0f.

We want to rate limit attack packets, and in effect prioritize processing of other, hopefully legitimate, ones. The p0f SYN signatures give us a language to describe and distinguish different types of SYN packets.

For example here is a typical p0f SYN signature of a Linux SYN packet:

4:64:0:*:mss*10,6:mss,sok,ts,nop,ws:df,id+:0

while this is a Windows 7 one:

4:128:0:*:8192,8:mss,nop,ws,nop,nop,sok:df,id+:0

Not getting into details yet, but you can clearly see that there are differences between these operating systems. Over time we noticed that the attack packets are often different. Here are two examples of attack SYN packets:

4:255:0:0:*,0::ack+,uptr+:0
4:64:0:*:65535,*:mss,nop,ws,nop,nop,sok:df,id+:0

You can have a look at more signatures in p0f's README and signatures database.

It's not always possible to perfectly distinguish an attack from valid packets, but very often it is. This realization led us to develop an attack mitigation tool based on p0f SYN signatures. With this we can ask iptables to rate limit only the selected attack signatures.

But before we discuss the mitigations, let's explain the signature format.

CC BY-SA 3.0 image by Hyacinth at the English language Wikipedia

As mentioned, the p0f SYN signature is a colon-separated string with the following parts:

• IP version: the first field carries the IP version. Allowed values are 4 and 6.

• Initial TTL: assuming that realistically a packet will not jump through more than 35 hops, we can specify an initial TTL ittl (usual values are 255, 128, 64 and 32) and check if the packet's TTL is in the range (ittl, ittl - 35).

• IP options length: length of IP options. Although it's not that common to see options in the IP header (and so 0 is the typical value you would see in a signature), the standard defines a variable length field before the IP payload where options can be specified. A * value is allowed too, which means "not specified".

• MSS: maximum segment size specified in the TCP options. Can be a constant or *.

• Window Size: window size specified in the TCP header. It can be a expressed as:

• a constant c, like 8192

• a multiple of the MSS, in the c*mss format

• a multiple of a constant, in the %c format

• any value, as *

• Window Scale: window scale specified during the three way handshake. Can be a constant or *.

• TCP options layout: list of TCP options in the order they are seen in a TCP packet.

• Quirks: comma separated list of unusual (e.g. ACK number set in a non ACK packet) or incorrect (e.g. malformed TCP options) characteristics of a packet.

• Payload class: TCP payload size. Can be 0 (no data), + (1 or more bytes of data) or *.

The following common TCP options are recognised:

• nop: no-operation

• mss: maximum segment size

• ws: window scaling

• sok: selective ACK permitted

• sack: selective ACK

• ts: timestamp

• eol+x: end of options followed by x bytes of padding

p0f describes a number of quirks:

• df: don't fragment bit is set in the IP header

• id+: df bit is set and IP identification field is non zero

• id-: df bit is not set and IP identification is zero

• ecn: explicit congestion flag is set

• 0+: reserved ("must be zero") field in IP header is not actually zero

• flow: flow label in IPv6 header is non-zero

• seq-: sequence number is zero

• ack+: ACK field is non-zero but ACK flag is not set

• ack-: ACK field is zero but ACK flag is set

• uptr+: URG field is non-zero but URG flag not set

• urgf+: URG flag is set

• pushf+: PUSH flag is set

• ts1-: timestamp 1 is zero

• ts2+: timestamp 2 is non-zero in a SYN packet

• opt+: non-zero data in options segment

• exws: excessive window scaling factor (window scale greater than 14)

• linux: match a packet sent from the Linux network stack (IP.id field equal to TCP.ts1 xor TCP.seq_num). Note that this quirk is not part of the original p0f signature format; we decided to add it since we found it useful.

• bad: malformed TCP options

Given a p0f SYN signature, we want to pass it to iptables for mitigation. It's not obvious how to do so, but fortunately we are experienced in BPF bytecode since we are already using it to block DNS DDoS attacks.

We decided to extend our BPF infrastructure to support p0f as well, by building a tool to compile a p0f SYN signature into a BPF bytecode blob, which got incorporated into the bpftools project.

This allows us to use a simple and human readable syntax for the mitigations - the p0f signature - and compile it to a very efficient BPF form that can be used by iptables.

With a p0f signature running as BPF in the iptables we're able to distinguish attack packets with a very high speed and react accordingly. We can either hard -j DROP them or do a rate limit if we wish.

First you need to clone the cloudflare/bpftools GitHub repository:

$ git clone https://github.com/cloudflare/bpftools.git

Then compile it:

$ cd bpftools
$ make

With this you can run bpfgen p0f to generate a BPF filter that matches a p0f signature.

Here's an example where we take the p0f signature of a Linux TCP SYN packet (the one we introduced before), and by using bpftools we generate the BPF bytecode that will match this category of packets:

$ ./bpfgen p0f -- 4:64:0:*:mss*10,6:mss,sok,ts,nop,ws:df,id+:0
56,0 0 0 0,48 0 0 8,37 52 0 64,37 0 51 29,48 0 0 0,
84 0 0 15,21 0 48 5,48 0 0 9,21 0 46 6,40 0 0 6,
69 44 0 8191,177 0 0 0,72 0 0 14,2 0 0 8,72 0 0 22,
36 0 0 10,7 0 0 0,96 0 0 8,29 0 36 0,177 0 0 0,
80 0 0 39,21 0 33 6,80 0 0 12,116 0 0 4,21 0 30 10,
80 0 0 20,21 0 28 2,80 0 0 24,21 0 26 4,80 0 0 26,
21 0 24 8,80 0 0 36,21 0 22 1,80 0 0 37,21 0 20 3,
48 0 0 6,69 0 18 64,69 17 0 128,40 0 0 2,2 0 0 1,
48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 1,
28 0 0 0,2 0 0 5,177 0 0 0,80 0 0 12,116 0 0 4,
36 0 0 4,7 0 0 0,96 0 0 5,29 0 1 0,6 0 0 65536,
6 0 0 0,

If this looks magical, use the -s flag to see the explanation on what's going on:

$ ./bpfgen -s p0f -- 4:64:0:*:mss*10,6:mss,sok,ts,nop,ws:df,id+:0
; ip: ip version
; (ip[8] <= 64): ttl <= 64
; (ip[8] > 29): ttl > 29
; ((ip[0] & 0xf) == 5): IP options len == 0
; (tcp[14:2] == (tcp[22:2] * 10)): win size == mss * 10
; (tcp[39:1] == 6): win scale == 6
; ((tcp[12] >> 4) == 10): TCP data offset
; (tcp[20] == 2): olayout mss
; (tcp[24] == 4): olayout sok
; (tcp[26] == 8): olayout ts
; (tcp[36] == 1): olayout nop
; (tcp[37] == 3): olayout ws
; ((ip[6] & 0x40) != 0): df set
; ((ip[6] & 0x80) == 0): mbz zero
; ((ip[2:2] - ((ip[0] & 0xf) * 4) - ((tcp[12] >> 4) * 4)) == 0): payload len == 0
;
; ipver=4
; ip and (ip[8] <= 64) and (ip[8] > 29) and ((ip[0] & 0xf) == 5) and (tcp[14:2] == (tcp[22:2] * 10)) and (tcp[39:1] == 6) and ((tcp[12] >> 4) == 10) and (tcp[20] == 2) and (tcp[24] == 4) and (tcp[26] == 8) and (tcp[36] == 1) and (tcp[37] == 3) and ((ip[6] & 0x40) != 0) and ((ip[6] & 0x80) == 0) and ((ip[2:2] - ((ip[0] & 0xf) * 4) - ((tcp[12] >> 4) * 4)) == 0)

l000:
    ld       #0x0
l001:
    ldb      [8]
l002:
    jgt      #0x40, l055, l003
l003:
    jgt      #0x1d, l004, l055
l004:
    ldb      [0]
l005:
    and      #0xf
l006:
    jeq      #0x5, l007, l055
l007:
    ldb      [9]
l008:
    jeq      #0x6, l009, l055
l009:
    ldh      [6]
l010:
    jset     #0x1fff, l055, l011
l011:
    ldxb     4*([0]&0xf)
l012:
    ldh      [x + 14]
l013:
    st       M[8]
l014:
    ldh      [x + 22]
l015:
    mul      #10
l016:
    tax
l017:
    ld       M[8]
l018:
    jeq      x, l019, l055
l019:
    ldxb     4*([0]&0xf)
l020:
    ldb      [x + 39]
l021:
    jeq      #0x6, l022, l055
l022:
    ldb      [x + 12]
l023:
    rsh      #4
l024:
    jeq      #0xa, l025, l055
l025:
    ldb      [x + 20]
l026:
    jeq      #0x2, l027, l055
l027:
    ldb      [x + 24]
l028:
    jeq      #0x4, l029, l055
l029:
    ldb      [x + 26]
l030:
    jeq      #0x8, l031, l055
l031:
    ldb      [x + 36]
l032:
    jeq      #0x1, l033, l055
l033:
    ldb      [x + 37]
l034:
    jeq      #0x3, l035, l055
l035:
    ldb      [6]
l036:
    jset     #0x40, l037, l055
l037:
    jset     #0x80, l055, l038
l038:
    ldh      [2]
l039:
    st       M[1]
l040:
    ldb      [0]
l041:
    and      #0xf
l042:
    mul      #4
l043:
    tax
l044:
    ld       M[1]
l045:
    sub      x
l046:
    st       M[5]
l047:
    ldxb     4*([0]&0xf)
l048:
    ldb      [x + 12]
l049:
    rsh      #4
l050:
    mul      #4
l051:
    tax
l052:
    ld       M[5]
l053:
    jeq      x, l054, l055
l054:
    ret      #65536
l055:
    ret      #0

For example, consider we want to block SYN packets generated by the hping3 tool.

First, we need to recognize the p0f SYN signature. Here it is, we know that one off the top of our heads:

4:64:0:0:*,0::ack+:0

(notice: unless you use the -L 0 option, hping3 will send SYN packets with the ACK number set, interesting, isn't it?)

Now, we can use the bpftools to get BPF bytecode that will match the naughty packets:

$ ./bpfgen p0f -- 4:64:0:0:*,0::ack+:0
39,0 0 0 0,48 0 0 8,37 35 0 64,37 0 34 29,48 0 0 0,
84 0 0 15,21 0 31 5,48 0 0 9,21 0 29 6,40 0 0 6,
69 27 0 8191,177 0 0 0,80 0 0 12,116 0 0 4,
21 0 23 5,48 0 0 6,69 21 0 128,80 0 0 13,
69 19 0 16,64 0 0 8,21 17 0 0,40 0 0 2,2 0 0 3,
48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 3,
28 0 0 0,2 0 0 7,177 0 0 0,80 0 0 12,116 0 0 4,
36 0 0 4,7 0 0 0,96 0 0 7,29 0 1 0,6 0 0 65536,
6 0 0 0,

This bytecode can then be passed to iptables:

$ sudo iptables -A INPUT -p tcp --dport 80 -m bpf --bytecode "39,0 0 0 0,48 0 0 8,37 35 0 64,37 0 34 29,48 0 0 0,84 0 0 15,21 0 31 5,48 0 0 9,21 0 29 6,40 0 0 6,69 27 0 8191,177 0 0 0,80 0 0 12,116 0 0 4,21 0 23 5,48 0 0 6,69 21 0 128,80 0 0 13,69 19 0 16,64 0 0 8,21 17 0 0,40 0 0 2,2 0 0 3,48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 3,28 0 0 0,2 0 0 7,177 0 0 0,80 0 0 12,116 0 0 4,36 0 0 4,7 0 0 0,96 0 0 7,29 0 1 0,6 0 0 65536,6 0 0 0," -j DROP

And here's how it would look in iptables:

$ sudo iptables -L INPUT -v
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    6   240            tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80match bpf 0 0 0 0,48 0 0 8,37 35 0 64,37 0 34 29,48 0 0 0,84 0 0 15,21 0 31 5,48 0 0 9,21 0 29 6,40 0 0 6,69 27 0 8191,177 0 0 0,80 0 0 12,116 0 0 4,21 0 23 5,48 0 0 6,69 21 0 128,80 0 0 13,69 19 0 16,64 0 0 8,21 17 0 0,40 0 0 2,2 0 0 3,48 0 0 0,84 0 0 15,36 0 0 4,7 0 0 0,96 0 0 3,28 0 0 0,2 0 0 7,177 0 0 0,80 0 0 12,116 0 0 4,36 0 0 4,7 0 0 0,96 0 0 7,29 0 1 0,6 0 0 65536,6 0 0 0

While defending from DDoS attacks is sometimes fun, most often it's a mundane repetitive job. We are constantly working on improving our automatic DDoS mitigation system, but we do not believe there is a strong reason to keep it all secret. We want to help others fighting attacks. Maybe if we all worked together one day we could solve the DDoS problem for all.

Releasing our code open source is an important part of CloudFlare. This blog post and the p0f BPF compiler are part of our effort to open source our DDoS mitigations. We hope others affected by SYN floods will find it useful.

Do you enjoy playing with low level networking bits? Are you interested in dealing with some of the largest DDoS attacks ever seen?If so you should definitely have a look at the opened positions in our London, San Francisco, Singapore, Champaign (IL) and Austin (TX) offices!

This makes it very easy to get unit test coverage, but there's no simple way to get coverage data for tests that you run against the main version of your program, like end-to-end tests.

The proper fix would involve adding -cover preprocessing support to go build, and exposing the coverage profile maybe as a runtime/pprof.Profile, but as of Go 1.6 there’s no such support. Here instead is a hack we've been using for a while in the test suite of RRDNS, our custom Go DNS server.

We create a dummy test that executes main(), we put it behind a build tag, compile a binary with go test -c -cover and then run only that test instead of running the regular binary.

Here's what the rrdns_test.go file looks like:

// +build testrunmain

package main

import "testing"

func TestRunMain(t *testing.T) {
	main()
}

We compile the binary like this

$ go test -coverpkg="rrdns/..." -c -tags testrunmain rrdns

And then when we want to collect coverage information, we execute this instead of ./rrdns (and run our test battery as usual):

$ ./rrdns.test -test.run "^TestRunMain$" -test.coverprofile=system.out

You must return from main() cleanly for the profile to be written to disk; in RRDNS we do that by catching SIGINT. You can still use command line arguments and standard input normally, just note that you will get two lines of extra output from the test framework.

Finally, since you probably also run unit tests, you might want to merge the coverage profiles with gocovmerge (from issue #6909):

$ go get github.com/wadey/gocovmerge
$ gocovmerge unit.out system.out > all.out
$ go tool cover -html all.out

If finding creative ways to test big-scale network services sounds fun, know that we are hiring in London, San Francisco and Singapore.

(You can play with this in the Go Playground here)

package main

import "fmt"

func main() {
	for i := 0; i < 10; i++ {
		fmt.Printf("%d ", i)
	}
}

It's output is easy to predict:

0 1 2 3 4 5 6 7 8 9

If you decided that it would be nice to run those fmt.Printfs concurrently using goroutines you might be surprised by the result. Here's a version of the code that runs each fmt.Printf in its own goroutine and uses a sync.WaitGroup to wait for the goroutines to terminate.

package main

import (
	"fmt"
    "runtime"
	"sync"
)

func main() {
	runtime.GOMAXPROCS(runtime.NumCPU())
    
	var wg sync.WaitGroup
	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func() {
			fmt.Printf("%d ", i)
			wg.Done()
		}()
	}
	
	wg.Wait()
}

(This code is in the Go Playground here). If you're thinking concurrently then you'll likely predict that the output will be the numbers 0 to 9 in some random order depending on precisely when the 10 goroutines run.

But the output is actually:

10 10 10 10 10 10 10 10 10 10

Why?

Because each of those goroutines is sharing the single variable i across the ten closures generated by the func() used for each goroutine.

The output from the goroutines will depend on the value of i when they start running. In the example, above they didn't actually start running until the loop had terminated and i had the value 10.

This programmer error can have other weird effects depending on the variable that's being shared across the goroutine closures.

To solve this the simplest solution is to create a new variable, a parameter to the func() and pass i into the function call. Like this:

package main

import (
	"fmt"
    "runtime"
	"sync"
)

func main() {
	runtime.GOMAXPROCS(runtime.NumCPU())
    
	var wg sync.WaitGroup
	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func(i int) {
			fmt.Printf("%d ", i)
			wg.Done()
		}(i)
	}
	
	wg.Wait()
}

(The code for that is here). That works correctly.

This is such a common gotcha that it's also in the FAQ.

Image credit: jantik

Over the last few weeks a number of companies have seen their password databases leaked onto the web and found that despite having made some effort to protect them many of the passwords were easily uncovered. Unfortunately, the disclosure of password databases is an ugly reality of the Internet; entire forums are dedicated to hackers who collaborate to uncover passwords from files and specialized password cracking software is easy to obtain.

To understand password storage it's best to go back to basics and some history.

The simplest way to store a password is just to store it in a database. When a customer tries to log in and types in the password 'supersecret' that string is compared with the password in the database and the customer is or is not allowed in.

Of course, storing passwords in the clear (or in plain text) is very dangerous. If the database is compromised then the passwords can be read and every account can be broken into. Despite this danger there are many companies that store passwords in plain text. Some attempt to encrypt the password and then decrypt it when you log in. Although that's slightly better than a plain text password in the database, it only adds a small hurdle for a hacker: they just have to take the database and the encryption key and since the key is almost certainly on the same machine as the database it becomes trivial to do.

Despite the poor security offered by encrypted or plain text passwords, many companies still use them. One sure fire way to find out whether a site you are using does this is to ask for a password reset: if the company is able to email you your old password then it was stored insecurely.

If you're following along and are new to password security you may be asking yourself: how can you test someone's password when they want to log in if you don't store it in some way? It does seem like an unsovable conundrum until you discover the cryptographic hash function (which I'll just shorten to hash function).

A hash function takes some string (such as a password) and turns it into a long number. In doing so it ensures two things: it's not possible to do the reverse (you can't take the number and run the algorithm backwards to get the string) and the number it generates is unique (i.e. there are no two strings that have the same number).

(Aside: I've simplified things a little in the previous paragraph. "not possible" should really be "infeasible" (i.e. you'd need to have more computers than there are on the planet to find the string) and "unique" should be "vanishingly improbable that two different strings will have the same number").

Hash functions work by taking the string to be hashed and scrambling the bits over and again to produce a number. One popular hash function is SHA-1. The SHA-1 hash of the password 'supersecret' is a761ce3a45d97e41840a788495e85a70d1bb3815 (the numbers are so long that they are typically written like this in hexadecimal instead of decimal. In decimal that number is 955,582,595,971,963,915,918,670,633,711,507,401,334,868,097,045). The SHA-1 hash of 'Supersecret' (note that capital S) is 1b417472fc8e2a0a4d44ed43f874309ca4069099 (as you can see it's totallydifferent).

Hash functions are used for many purposes such as checking that the contents of a file haven't changed. When you download a file from the Internet its hash might also be sent so that your computer can check that no bits in the file have been accidentally flipped in transmission.

Hash functions are also often used in password systems because instead of storing the password, you can simple store the hash. Since the hash can't be easily reversed the stored hash is a secure way of keeping the password. When a visitor comes to the site the hash of the password they entered is calculated and compared with the hash in the database. Since the hashes are unique they'll only be able to log in with the right password.

Image credit: ToGa Wanderings

Unfortunately, simply using a hash function like this is dangerous. Over the last few weeks a number of prominent Internet companies have found that their password databases have been cracked even though they 'hashed' their passwords. To see why, try Googling a761ce3a45d97e41840a788495e85a70d1bb3815. You might be surprised to find that the first result tells you that that's the SHA-1 hash of 'supersecret'.

The problem with simple hash functions is that hackers simply get a dictionary and compute all the hashes of all the possible passwords made from the dictionary. These massive databases of precomputed hashes are called rainbow tables. If a password database leaks then the hackers just look up the hashes in the rainbow table. The hashes that aren't found in the rainbow table correspond to those users who created long, complex passwords that weren't precomputed in this way. That's one reason why picking a long, complex password matters: hackers won't have already computed its hash.

Even though the hash function itself couldn't be reversed, it was possible to create a table of precomputed password hashes (especially for poorly chosen passwords).

The way around rainbow tables is with something called salt. Let's suppose you've picked the password 'supersecret' and company X is going to use SHA-1 to hash the password. Instead of simply hashing the password, company X picks a random salt (a random string of characters) that's unique to you (such as '$f2%38h##f23'). Instead of computing SHA-1(supersecret) they compute SHA-1(supersecret$f2%38h##f23) and get 33438b91ce09e695923 2f698b7939e6ee1d0712a. Try Googling that and you won't get anyresults.

Image credit: stlbites

Since each user has some random salt applied to the hash, rainbow tables are useless. It's not possible to precompute the hashes of all the possible passwords with all the possible salt values.

Until recently a 'salted hash' like this was how CloudFlare stored user passwords.

Unfortunately, password cracking techniques benefit enormously from two things: Moore's Law and the speed of hash functions. Hash functions weren't originally designed for protecting passwords, they were designed to check the integrity of data by detecting changes (notice how just changing from s to S in supersecret dramatically changed the SHA-1 hash above) and for that reason they were designed to be fast, very fast.

As computers have increased in speed with Moore's Law the speed of hash functions has made it possible to do away with rainbow tables and start attacking passwords directly even when salted. When a password database leaks, password cracking software is able to compute millions of passwords per second applying the unique salt to each password and checking the resulting hash value. The software literally tries out combinations of words and letters and computes the hash for each one.

That means that only long, complex passwords are safe with a saltedhash.

Image credit: 4nitsirk

The solution is to use a hash function that's slow. If the hash function itself is slow then it slows down cracking software. If the speed can be chosen so that over time the hash function can be made slower, then the hash function can be slowed down so that password cracking doesn't get easier.

Happily, hash functions with just that property have been invented specifically to help keep passwords safe. We recently upgraded our entire password database to use bcrypt. bcrypt is just like a normal hash function but it has an additional parameter: as well as being fed the password and some random salt it's fed a cost. The cost tells the hash function how hard to work in computing the hash (and thus determines how long it will take).

Over time the cost can be increased (it's just a number) to keep pace with faster and faster computers and keep passwords safe by making the hash function slower and slower.

Just like all aspects of security, password storage needs to be reviewed from time to time. As we've seen recently many companies don't take the time to upgrade their password security leading to serious problems.

And, of course, users can help out too: password cracking relies partly on the algorithms used to store the passwords and partly on the complexity of the password. Make sure to choose a long, complex password and don't use it on any other site.

CloudFlare has partnered with a number of CloudFlare Certified Partners to make it simple for website owners that want a faster and safer website. Since signing up for CloudFlare through a hosting partner is different than signing up for CloudFlare directly, we wanted to provide some quick tips to help you get the most out of your CloudFlare experience.

1. You do not need to change your name servers when activating through a hosting partner. You would still manage your DNS entries at your hosting provider or registrar.

2. CloudFlare can only be enabled for CNAME records when activating through a hosting partner. To enable CloudFlare on your root domain (yourdomain.com), which is an A record, you need to have your hosting partner set a 301 redirect from your root domain to www. Not only will the redirect help accelerate and protect the root domain, this will also make the statistics in your CloudFlare account accurate. Note: If you have a naked domain, 'yourdomain.com', and you don't want your visitors to go to 'www.yourdomain.com', then you need to signup directly with CloudFlare.

3. What you should do if you see any of the following error messages after enabling CloudFlare:

"Host Not Configured to Serve Web Traffic" error message will appear on the first request to your site after activating through a partner, then will go away after a few minutes. If it lasts for more than 10 minutes,then contact your hosting provider and our support teams will work together to resolve.

"CloudFlare-nginx 502 Bad Gateway": This is an issue on the CloudFlare network. We deal with these quickly (less than 10 minutes). We publish all announcements regarding our network status on @CloudFlareSys.

"Website is Unavailable": Either your server is offline and we don't have a copy of your site in cache or something on the origin server is blocking CloudFlare's IPs.

If your server is online, then work with your hosting provider to find out what could be blocking CloudFlare's IPs on your server. The most common culprit is a security solution like a firewall like CSF or IP tables. As soon as the block is removed, the error page will disappear.

SSLIf you have SSL on the domain(s), you will need to upgrade to a Pro account. The cost for a Pro account is $20.00 per month for the first website and $5.00 for each additional site. In addition to the SSL support, you will also receive additional security and performance benefits.

Note: You will find the option to upgrade to Pro in your CloudFlare account.

Development ModeIf you are making changes to the static content on your website, temporarily bypass CloudFlare's cache so any changes appear immediately. You can find Development Mode either right in your hosting provider's control panel or by logging in to your CloudFlare account under CloudFlare Settings.

PageRulesPageRules gives you more powerful performance and configuration options, including:Advanced Caching ConfigurationsExcluding URLS from CloudFlare's default caching and security optionsSetting URL forwards and redirects

CloudFlare has developed web content optimization features called Rocket Loader and Auto Minify. Both Rocket Loader and Auto Minify are designed to load your site's resources even faster than the default CloudFlare configuration.

Rocket LoaderRocket Loader will speed up the delivery of your pages by automatically asynchronously loading your JavaScript resources. Rocket Loader works well for websites that have a lot of ads, widgets or plugins.

Auto MinifyRemoves all unnecessary characters from HTML, CSS, and JavaScript to reduce file size.

Note: Both of these features are still in beta. If you encounter any issues, such as a broken plugin or JavaScript not working properly, then please turn the feature off and report any bugs to our team.

To turn on Rocket Loader and Auto Minify, you need to log in to your CloudFlare account and go to CloudFlare Settings.

IPv6 GatewayMake your website IPv6 compatible, by turning on the CloudFlare IPv6 gateway.

The CloudFlare Support Center has answers to a number of questions. Searching our knowledge base is the fastest way to get a quick response to the majority of questions. Don't see the answer to your question? Please contact CloudFlare.

We frequently post about product updates, early beta access to new features, system issues, and giveaways, so we recommend that you follow us on Facebook, Twitter or Google+:

• Facebook

• Twitter

• Google+

Thank you for joining CloudFlare in partnership with your hosting provider.