I'm not certain whether this is "on topic" or not. I'm going to
take the chance and ask.
Is it possible to use "robots.txt" to totally block certain robots
from a domain?
I have a domain from which certain sites have been gone for 2 years.
However, Scooter insists on probing for them. It wouldn't be bad
except that I have a perl script that sends an email whenever a
robot reports broken links. Scooter just found all those sites
which have been gone for 2 years and my perl script sent me ... all
at once ... 181 emails.
At least the other robots don't hit so hard in one shot!
Phew! I feel better now. If this isn't a topic for here, does
anyone know where to look?
--
Raquel
============================================================
Men never do evil so completely and cheerfully as when they do it
with religious conviction.
--Blaise Pascal
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
take the chance and ask.
Is it possible to use "robots.txt" to totally block certain robots
from a domain?
I have a domain from which certain sites have been gone for 2 years.
However, Scooter insists on probing for them. It wouldn't be bad
except that I have a perl script that sends an email whenever a
robot reports broken links. Scooter just found all those sites
which have been gone for 2 years and my perl script sent me ... all
at once ... 181 emails.
At least the other robots don't hit so hard in one shot!
Phew! I feel better now. If this isn't a topic for here, does
anyone know where to look?
--
Raquel
============================================================
Men never do evil so completely and cheerfully as when they do it
with religious conviction.
--Blaise Pascal
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org