What is the Robot Text File?
Skip | 14 December, 2005 21:12
The robot text file is used to allow or disallow specific or all search
engine spider’s access to web folders, pages or files that you don't
want spidered or indexed.
Why would you want to do this?
You may have created a personnel page for company employees that you don't want listed in a search engine. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.
How do you use it?
You need to upload it to the root of your web site or it will not work. If you don't have access to the root directory (on Cpanel hosting accounts, this directory is named /WWW/ or /public_html/ ) then you will need to use a Meta tag to disallow access.
You need to include both the user agent and a file or folder to disallow.
What does it look like?
It's really nothing more than a "Notepad" type .txt file named "robots.txt"
The basic syntax is
User-agent: spiders name here
Disallow:/ filename here
If you use
User-agent: *
The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.
To disallow an entire directory use
Disallow:/mydirectory/
To disallow an individual file use
Disallow:/file.htm
You have to use a separate line for each disallow. You cannot you for example use
Disallow:/file1.htm,file2.html
You should use
Use-agent/*
Disallow:/file1.htm
Disallow:/file2.htm
For a list of spider names visit http://www.robotstxt.org/wc/active/html/
Make sure you use the right syntax. If you don't it will not work.
There are many robots.txt tutorials on the web, including the site listed above. Before you try to implement one, be sure to double check with that site.
Lastly, if you are a true paranoid person, you are best to password protect any directory or files you don't want the public or any spiders to see. There is no better way to keep files secure than to password protect them.
Why would you want to do this?
You may have created a personnel page for company employees that you don't want listed in a search engine. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.
How do you use it?
You need to upload it to the root of your web site or it will not work. If you don't have access to the root directory (on Cpanel hosting accounts, this directory is named /WWW/ or /public_html/ ) then you will need to use a Meta tag to disallow access.
You need to include both the user agent and a file or folder to disallow.
What does it look like?
It's really nothing more than a "Notepad" type .txt file named "robots.txt"
The basic syntax is
User-agent: spiders name here
Disallow:/ filename here
If you use
User-agent: *
The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.
To disallow an entire directory use
Disallow:/mydirectory/
To disallow an individual file use
Disallow:/file.htm
You have to use a separate line for each disallow. You cannot you for example use
Disallow:/file1.htm,file2.html
You should use
Use-agent/*
Disallow:/file1.htm
Disallow:/file2.htm
For a list of spider names visit http://www.robotstxt.org/wc/active/html/
Make sure you use the right syntax. If you don't it will not work.
There are many robots.txt tutorials on the web, including the site listed above. Before you try to implement one, be sure to double check with that site.
Lastly, if you are a true paranoid person, you are best to password protect any directory or files you don't want the public or any spiders to see. There is no better way to keep files secure than to password protect them.
Comments for post
Menu
calendar
| « | October 2008 | » | ||||
|---|---|---|---|---|---|---|
| Su | Mo | Tu | We | Th | Fr | Sa |
| 1 | 2 | 3 | 4 | |||
| 5 | 6 | 7 | 8 | 9 | 10 | 11 |
| 12 | 13 | 14 | 15 | 16 | 17 | 18 |
| 19 | 20 | 21 | 22 | 23 | 24 | 25 |
| 26 | 27 | 28 | 29 | 30 | 31 | |
RSS Feed