NoIndex, NoFollow and Robots.txt

An interesting interview between Eric Enge and Matt Cutts sheds light on the distinction between the NoIndex, NoFollow and Robots.txt. These are the staples when trying to sculpt PageRank,  so that it accrues only to your most important pages or avoiding excessive duplicate content. The interview transcript reveals lots of useful information but it is helpful to summarise that information which I have done below:

The noindex and nofollow attributes can be combined.
To complicate matters further, the attributes can be used in different ways. Either:

• a) as part of the link, for example: <a rel=”nofollow, noindex” href=”somelink.html”>A link that I don’t want the search engines to see</a> This method is a less reliable way to control the page because there may other links which allow a page to be crawled and indexed.
• b) in the head sections of the page using the <meta name=”robots” content=”noindex, nofollow” /> tag. This means any link pointing to the page will have the nofollow or noindex applied.

The problem then is that robots.txt can cause PageRank to be lost to pages that do not get followed. The same thing happens for pages that use the meta noindex follow and the meta noindex nofollow.

The best solution to avoid duplicate content is to apply the rel=”nofollow” on all links to a page that you don’t want the links to be crawled and PageRank will not flow through the links. With complicated website with many duplicated pages, this is not an ideal fix but it is better than nothing.

That being said, these modifications are an afterthought. As one needs some PageRank before one can sculpt it.