Our Blog

When To Use No Follow vs. Robots.txt

Websites show up in Google’s results page because Google’s web crawlers have indexed the website. When they index the site, they take note of the title tags, image tags, content, headers, and of course links. There are times, however, when someone may not want a portion of their website crawled or indexed.

No-follow

No-follow tags are placed in the code of websites next to links that the webmaster does not want Google to “follow.” Why would someone not want Google to follow a link? Aren’t links good for SEO? Well it depends on the link, especially now in light of Penguin 2.0.

Internal links, or a link that brings to a different page on the same website, pass what is called “page rank”.  For example, the homepage of a website typically ranks the highest SEO-wise, and each link from the home page to each internal page passes some of that ranking value to the internal page. Sometimes, you do not want to pass some of this ranking value to the other pages. An example of this is the contact page. If someone is on your website and wants to contact you, that is great and you should certainly make it easy for them to do so. But your contact page does not necessarily need to rank highly in search engine results, so links to this page can be no-follow.

The main reason the no-follow tag was created is comment spamming containing external links on blogs. Bloggers usually allow comments on their sites in order to engage readers and keep people interested by allowing them to participate in the website. Unfortunately, a lot of spammers use this as a way to build links in a dishonest way, often including a link to a website that is also spam. These links also often go to gambling and adult websites, which most bloggers do not want to be associated with. Additionally, links should go to and come from relevant sites, so if the blog isn’t about gambling a link to a gambling site isn’t considered relevant. Bloggers typically code all links no-follow on blog comments due to these reasons.

Matt Cutts, head of Google’s Webspam team, has stated that just because the no-follow tag is placed on a link, doesn’t necessarily mean Google’s crawlers won’t still follow the link. He has likened it to removing a restroom sign. It may be harder to find, but that doesn’t mean it’s impossible. This has led some webmasters to doubt whether these links really are truly not affecting SEO. While they do not pass actual page rank, there are other factors Google looks at when determining where a site shows up on the search results.

Robots.txt

To really be sure Google isn’t crawling a link you do not want crawled, the robots.txt file is used. This is placed in the code anywhere a webmaster does not want Google to index. Besides spammy links, why would you not want Google to index something on your site? More complicated websites have private folders or may have content that does not provide additional information. By preventing the crawlers from indexing information that isn’t important to the site, the crawler is more likely to go deeper into your site and index more of what you do want indexed. Because the crawler indexes the site deeper and more quickly, the results will show up in search engines more quickly.

There is much more to building websites and getting to the first page of Google results than many people realize. While blocking portions of your sites from Google’s crawlers may seem counter-intuitive, this is sometimes the best way to get the portions of the website you really want to promote to be deemed the most important by Google.