Simple robots.txt file
April 20, 2007 · Print This Article
New webmasters tends to ask very important questions that experienced webmasters tends to forget how difficult such matters can be for new webmasters. A young webmaster recently found creating robots.txt file a challenging task, so a simple rough and ready robots.txt file was created for the young webmasters, here is a what a robots.txt file should contain for the benefits of those who may need some info on robots.txt
# robots.txt for www.SiteName
User-agent: *
Disallow:
The robots.txt above allows all robots to index all parts of your site, here is what you need to do if you want robots to leave some parts of your site alone. On the third like which says “Disallow:” just add the folder you wish robots to leave a alone, lets say you cgi-bin folder, just add this:
/cgi-bin/
so the new robots.txt that allows robots to every part of your website except the cgi-bin folder would look like this:
# robots.txt for www.SiteName
User-agent: *
Disallow: /cgi-bin/
This is some text prior to the author information. You can change this text from the admin section of WP-Gravatar Test Bio for Temi Read more from this author
Comments
One Response to “Simple robots.txt file”
Got something to say?
[...] did a blogpost about robots.txt files here. You can also generate it automatically this days with help from Google via your Google webmaster [...]