[anti-abuse-wg] Re: update on netsecdb project

On Tue, Apr 06, 2010 at 03:57:30PM +0200, Frank Gadegast wrote:
> most blacklist do not care to block a person, they block IPs.
> If those IPs are dynamic, its up to the provider how to deal
> with that.
I'm just noting that almost all of the spam nowadays is sent out
from dynamic IP addresses by bots and that these are used to send
out a number of spams per bot which tends to decrease (obviously to
make detection and blocking harder). These finding are not in dispute
in the community AFAIK.


> I doubt your results.
> They are probably based only on open blacklists.
We've used SpamHaus XBL which specifically targets bots. I don't see
how a non-open blacklist could be used in a scientific paper as
nobody would be able to check the results - anything could be
claimed. Our claims have been verified by peer review, been published
in a prestiguous journal and been quite popular (at least according
to download counts, provided these have been correctly counted by
Elsevier).

One reason for a seemingly good performance in detecting bots via
blacklists could be if you blocked whole network ranges instead of
single IPs. This would make it possible to block whole ISPs (mostly
those who don't care about bots in their ranges), but also
significantly increases the FP rate by blocking legitimate traffic.
Not all users from "bad" ISPs are necessarily "bad" themselves.

Hidden costs of such as system can be quite high and are costly to
analyze. I'm still taking a few hours each month to manually analyze a
random sample of incoming spam for false positives but very few companies
do. In fact when I worked for a spam filter company for a few months and
did the same for a 24h sample, I found out that their actual FP rate was
ten times(!) higher than their previously estimated value based on
explicit customer feedback.

Feel free to read our paper and download our systems, run them on
your own data and check our results.


> Its that easy to track every bot, specially for the access providers,
> if their own IPs get abused.
Indeed. But since some access providers make money off lots of bot
traffic, it might be hard to convince them to stop this. If we have
to wait till all access providers have software to detect bots, we
are likely to wait a long time...


> > So here's to hoping the spammers die out from the current crisis and
> > we can switch off all our spamfilters...
> I disagree here.
So you want the spammers to survive?  One of the hindrances in my work
has been that - because spam filters work extremely well and the
costs of FPs are easily overlooked - a lot of companies profit from the
status quo: not only spam filter companies, but also free email
services, ISPs with traffic-dependent fees (mobile may be upcoming
here), anti-virus companies, IT security firms etc..  They are not
interested in a permanent solution and indirectly contribute to a
prolonging of the current situation. I had some first-hand
encounters with this mindset.


> I still not get how RIPE can accept criminal or even ignorant members.
> Criminality cant be part of the "free internet".
If it cannot, it can no longer be free. There is a price to pay for
freedom and it is exactly that. Also, AFAIK RIPE was never designed
for that and has no legal way to enforce their rules even if they wanted
to. You can't expect a technical governing body to take the role of
world internet criminality police without additional resources. It
would be far too much like legislation, judgment and execution in one
organization, and that's clearly _not_ my definition of freedom.

Best,
  Alex
-- 
Dr. Alexander K. Seewald

Seewald Solutions
www.seewald.at
Tel. +43(664)1106886
Fax. +43(1)2533033/2764