Search Engine Optimization (SEO) Tools #2 - Robots.txt Generator

by Scott Allen

Robots.txt is an essential but highly ignored element of web design and SEO. Robots.txt is a text file that gets placed in the root of your web site, one for each web domain (yoursite.com). All the search engine spiders (and other good bots) check it to see what directories of your site they are allowed to index and which ones they are restricted from visiting. The bad bots look in it to see what directories are off limits and they head straight to your off-limit files to see what’s hidden there. We’ll go more into how to get rid of the bad bots later (through use of bot traps and dynamic robots.txt files), but for now, here is a great tool that will let you generate a quick robots.txt file for your site.

Reposted from http://www.seochat.com/seo-tools/robots-generator/:

Use this tool to generate a simple robots.txt file for your website. This file allows you to hide files or directories that you don’t wish the search engine spiders to find. This generator tool is designed to create the text for the file for you; you can then make changes afterward should you deem it necessary.

Generating a robot file with this tool is ideal if you wish to block certain directories or files from search engines.

To use the generator tool, enter the required information here and click the button. You will then be shown the text for the file. Copy this to a file called robots.txt and place on the root of your website (in the same place as your home page).

Imposing Restrictions
You may impose restrictions on which web pages to disallow indexing. By default, most users will want to allow all directories except their /cgi-bin/ directory, which commonly holds scripts, and their images directory /images/. To enable all web pages, select Yes to “Enable All Webpages,” then enter each web page or directory path in the exclusion box, one per line.

Example: “http://www.yourdomain.com/cgi-bin/” (Excludes the /cgi-bin/ directory)
Example: “http://www.yourdomain.com/images/” (Excludes the /images/ directory)
Example: “http://www.yourdomain.com/welcome.html” (Excludes the /welcome.html web page)

Tags:
robots.txt | seo | webgeek

If you enjoyed this post, make sure you subscribe to the RSS feed!

 Email This to a Friend

 Print This Post

Related Posts:

Detect User-Agents: Cloak and Dagger for Web Sites - Part 2

How to Build Bot Traps in PHP

Contact Form Generator

Web 2.0 is Not a Style

Web Site Security - Bot Traps