Re: Interesting spamming tool: a robot to grab e-mail addresses

At 4:30 pm +0100 30/3/98, Richard Kettlewell wrote:
>Simon Wilkinson writes:
>>>> Bull's Eye Gold is the PREMIER email address collection tool.
>[...]
>>
>> Here's an idea (presuming it ignores robots.txt files, and forges its
>> User-Agent to look like a popular browser) Use wpoison (or similar) to
>> construct an infinitely deep area of your web tree. List this area in
>> your robots.txt file.
>>
>> If you see more than (x) hits to this area in a certain time from a
>> certain IP address then set up access control measures to block that
>> IP address from accessing your server. Perhaps return a message
>> telling them why this is happened, and how to have their access
>> re-enabled. You could do all of this automatically, so the admin
>> wouldn't have to do anything about it.
>
>If the scanner was run by a user of an ISP which allocated addresses
>dynamically then you'd end up blocking people who had done you no
>harm.  Which would be bad.
>...

An alternative approach might be to try to mire the robot as it tries to
search you. If you detect a certain flurry of access that is characteristic
of a robot search you could progressively slow down your responses to
requests from this address (halve the speed each time perhaps?) and trickle
back low-grade rubbish as slow as possible, but just enough to stop the
robot going away. Once the robot did actually stop probing you could return
to normal for following legitimate users (perhaps after, say, half an hour
or a day or something), but perhaps set more aggressive thresholds for this
address for the next week/month?