diff options
| author | brian <brian@unknown> | 1998-01-26 08:12:58 +0100 |
|---|---|---|
| committer | brian <brian@unknown> | 1998-01-26 08:12:58 +0100 |
| commit | bbc3f6d277e3b53cc23ebd07663087dba79ef941 (patch) | |
| tree | 4b3114d195abf400640643099dcf649996968a00 /docs | |
| parent | PR: (diff) | |
| download | apache2-bbc3f6d277e3b53cc23ebd07663087dba79ef941.tar.xz apache2-bbc3f6d277e3b53cc23ebd07663087dba79ef941.zip | |
PR:
tsk tsk, randy. Can't find this on covalent.net either.
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@80015 13f79535-47bb-0310-9956-ffa450edef68
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/manual/misc/howto.html | 9 |
1 files changed, 8 insertions, 1 deletions
diff --git a/docs/manual/misc/howto.html b/docs/manual/misc/howto.html index 98a1843e83..a7d6b38c3c 100644 --- a/docs/manual/misc/howto.html +++ b/docs/manual/misc/howto.html @@ -130,7 +130,14 @@ is then used by a <em>search engine</em> to help locate information.</P> <P><code>robots.txt</code> provides a means to request that robots limit their activities at the site, or more often than not, to leave the site alone.</P> -<P>When the first robots were developed, they had a bad reputation for sending hundreds/thousands of requests to each site, often resulting in the site being overloaded. Things have improved dramatically since then, thanks to <A HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html"> Guidelines for Robot Writers</A>, but even so, some robots may <A HREF="http://www.zyzzyva.com/robots/alert/">exhibit unfriendly behavior</A> which the webmaster isn't willing to tolerate, and will want to stop.</P> +<P>When the first robots were developed, they had a bad reputation for +sending hundreds/thousands of requests to each site, often resulting +in the site being overloaded. Things have improved dramatically since +then, thanks to <A +HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html"> +Guidelines for Robot Writers</A>, but even so, some robots may exhibit +unfriendly behavior which the webmaster isn't willing to tolerate, and +will want to stop.</P> <P>Another reason some webmasters want to block access to robots, is to stop them indexing dynamic information. Many search engines will use the |