I'm not sure this is possible with Apache alone (anyone correct me if I'm wrong), but I have an idea.<br><br>You didn't mention which language the CGI scripts are written in, but assuming that you can edit them, you could create a temporary file when at their top, delete it at the end, and skip the execution if the file exists. That way, only one execution can be made at a time. I don't know how well it fits your requirements, but that's the first thing that comes to mind.<br>
<br>Kind regards,<br>Helgi Hrafn Gunnarsson<br><a href="mailto:helgi@binary.is">helgi@binary.is</a><br><br><div class="gmail_quote">On Fri, Sep 17, 2010 at 4:08 PM, Gilles Detillieux <span dir="ltr"><<a href="mailto:grdetil@scrc.umanitoba.ca">grdetil@scrc.umanitoba.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Every once in a while, some doofus points a web crawler at our web site<br>
and, ignoring the disallowed areas in our robots.txt file, starts<br>
crawling through some of our cgi-bin scripts at a rate of 4 to 8 hits a<br>
second. This is particularly annoying with some of the more processor<br>
and disk intensive CGI programs, such as man2html, which also happens to<br>
generate lots of links back to itself.<br>
<br>
Is there anything I can set up in Apache to throttle back and slow down<br>
remote hosts when they start hitting hard on cgi-bin? I don't want to<br>
do anything that would adversely affect legitimate users, nor make<br>
important things like the manual pages hard to find by removing any<br>
public links to them. But when a client starts making 10 or more GET<br>
requests on /cgi-bin in a 5 second period, it would be nice if I could<br>
get the server to progressively add longer and longer delays before<br>
servicing these requests, to keep the load down and prevent the server<br>
from thrashing.<br>
<br>
I'd appreciate any tips.<br>
<br>
Thanks,<br>
Gilles<br>
<font color="#888888"><br>
--<br>
Gilles R. Detillieux E-mail: <<a href="mailto:grdetil@scrc.umanitoba.ca">grdetil@scrc.umanitoba.ca</a>><br>
Spinal Cord Research Centre WWW: <a href="http://www.scrc.umanitoba.ca/" target="_blank">http://www.scrc.umanitoba.ca/</a><br>
Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 0J9 (Canada)<br>
_______________________________________________<br>
Roundtable mailing list<br>
<a href="mailto:Roundtable@muug.mb.ca">Roundtable@muug.mb.ca</a><br>
<a href="http://www.muug.mb.ca/mailman/listinfo/roundtable" target="_blank">http://www.muug.mb.ca/mailman/listinfo/roundtable</a><br>
</font></blockquote></div><br>