Why the 1MB fetcher limit

[Christian Holtje]

On Jul 31, 2008, at 4:29 PM, Jonathan Daugherty wrote:
>> Why limit
>> responses to 1MB?  Was this an arbitrary size, or was there some  
>> logic
>> to it?  Why was this added in the first place?  Was the absence of
>> such a limit causing real problems?
>
> As Andrew pointed out, we're using it to prevent accidental DoS.  We
> did have a few cases come up in the wild.  So yes, it was causing
> problems for some RPs.  In some cases it may also avoid problems when
> attempting to parse identity URL responses with PCRE.  I can
> understand why Bad Behavior might consider that a typical bot feature,
> but I don't think it makes sense to remove it from the libraries.

But how does it prevent against DoS?  I can write a web server that  
sends gigs of crap and ignores RANGE headers.  What you want is the  
receiver to close the connection after 1MB.  Don't depend on the sender.

Ciao!