Select Page

New webmasters tends to ask very important questions that experienced webmasters tends to forget how difficult such matters can be for new webmasters. A young webmaster recently found creating robots.txt file a challenging task, so a simple rough and ready robots.txt file was created for the young webmasters, here is a what a robots.txt file should contain for the benefits of those who may need some info on robots.txt

# robots.txt for www.SiteName
User-agent: *
Disallow:

The robots.txt above allows all robots to index all parts of your site, here is what you need to do if you want robots to leave some parts of your site alone. On the third like which says “Disallow:”  just add the folder you wish robots to leave a alone, lets say you cgi-bin folder, just add this:
/cgi-bin/

so the new robots.txt that allows robots to every part of your website except the cgi-bin folder would look like this:

# robots.txt for www.SiteName
User-agent: *
Disallow: /cgi-bin/