Essential Web Site Maintainance

There are certain things that one should do to your website to make it easier for the search engines to crawl your web site. While they don’t directly increase your ranking they can make it easy for search engines to find their way around  which may give your pages the edge and will most definitely give you useful information which can help you make the right decisions in your SEO campaign.

Google Webmaster Tools

Webmaster Tools is very useful diagnostic tool to find out what Google knows about your web site. You will need to set up a Google account in order to use it, however once you have registered you can use it for all the Google services.

It will give you an overview of your website alerting you to any errors or problems that Google encountered crawling your site such as pages not found, content problems. In addition, it will give statistics for how often Google is crawling your site and help you to see what keywords users are searching for and your position for these queries as well as which sites are linking to your site.The sitemap section will alert you to any problems encountered with your sitemap.

Google Analytics

Google Analytics is a good way of tracking your visitors and answering questions such as: Where did they come from? What pages did they visit? How long did they stay?

Robots.txt

This is one way to tell search engines which pages they are allowed to index. I have known people to change their robots.txt disallowing them from crawling your site. If you are having problems getting indexed it is one thing that you can check before assuming that you have been banned.

The robots.txt can be written in Notepad or any text editor. The basic structure shows the name of the user agent and the directories that it is not allowed to spider.

For example:

This would allow any search engine to spider anything.

This would stop all search-engine from spidering the site, so look out for that one. If there is no robots.txt then the search-engines can spider follow any link in your content. To stop a particular robot then you would  use:

You should prevent robots from accessing any administrator code for example login pages or cgi-bin folders as a security measure.

This will prevent search engines access to the folders listed. Individual pages can also be added to the list.

Sitemap XML

It is a good idea to include both an XML sitemap. This is tells Googlebot what pages are in your site and it will allow give another way to crawl the site and hopefully spider all your pages.

Sitemaps can be created automatically using online XML sitemap generators, which will spider your web site and write the XML for you.

A similar XML file can be created with a list of urls in your website or submitted as a text file to the Yahoo search engine. It is important that the sitemap files are kept upto date when new pages are created.

A little known benefit of using a sitemap is that it is a way of telling Google about your preference in terms of the URL you would like indexing. The URL in the sitemap is the URL which Google will consider the most important if there are duplicate versions of the page, which is a common occurence on many dynamic web sites.