Simple robots.txt file
New webmasters tends to ask very important questions that experienced webmasters tends to forget how difficult such matters can be for new webmasters. A young webmaster recently found creating robots.txt file a challenging task, so a simple rough and ready robots.txt file was created for the young webmasters, here is a what a robots.txt file should contain for the benefits of those who may need some info on robots.txt
# robots.txt for www.SiteName
User-agent: *
Disallow:
The robots.txt above allows all robots to index all parts of your site, here is what you need to do if you want robots to leave some parts of your site alone. On the third like which says “Disallow:” just add the folder you wish robots to leave a alone, lets say you cgi-bin folder, just add this:
/cgi-bin/
so the new robots.txt that allows robots to every part of your website except the cgi-bin folder would look like this:
# robots.txt for www.SiteName
User-agent: *
Disallow: /cgi-bin/
This is some text prior to the author information. You can change this text from the admin section of WP-Gravatar To change this standard text, you have to enter some information about your self in the Dashboard -> Users -> Your Profile box. Read more from this author
