There used to be a third party module, something like mod_limit or mod_bwlimit, that lets you limit connection rates.<div><br></div><div>That said, the developer in me says "cache". :)<br><div><br></div><div>Sean<br>
<br><div class="gmail_quote">On Fri, Sep 17, 2010 at 4:08 PM, Gilles Detillieux <span dir="ltr"><<a href="mailto:grdetil@scrc.umanitoba.ca">grdetil@scrc.umanitoba.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Every once in a while, some doofus points a web crawler at our web site<br>
and, ignoring the disallowed areas in our robots.txt file, starts<br>
crawling through some of our cgi-bin scripts at a rate of 4 to 8 hits a<br>
second. This is particularly annoying with some of the more processor<br>
and disk intensive CGI programs, such as man2html, which also happens to<br>
generate lots of links back to itself.<br>
<br>
Is there anything I can set up in Apache to throttle back and slow down<br>
remote hosts when they start hitting hard on cgi-bin? I don't want to<br>
do anything that would adversely affect legitimate users, nor make<br>
important things like the manual pages hard to find by removing any<br>
public links to them. But when a client starts making 10 or more GET<br>
requests on /cgi-bin in a 5 second period, it would be nice if I could<br>
get the server to progressively add longer and longer delays before<br>
servicing these requests, to keep the load down and prevent the server<br>
from thrashing.<br>
<br>
I'd appreciate any tips.<br>
<br>
Thanks,<br>
Gilles<br>
<font color="#888888"><br>
--<br>
Gilles R. Detillieux E-mail: <<a href="mailto:grdetil@scrc.umanitoba.ca">grdetil@scrc.umanitoba.ca</a>><br>
Spinal Cord Research Centre WWW: <a href="http://www.scrc.umanitoba.ca/" target="_blank">http://www.scrc.umanitoba.ca/</a><br>
Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 0J9 (Canada)<br>
_______________________________________________<br>
Roundtable mailing list<br>
<a href="mailto:Roundtable@muug.mb.ca">Roundtable@muug.mb.ca</a><br>
<a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
</font></blockquote></div><br><br clear="all"><br>-- <br>Sean Walberg <<a href="mailto:sean@ertw.com">sean@ertw.com</a>> <a href="http://ertw.com/">http://ertw.com/</a><br>
</div></div>