[anti-abuse-wg] Fail2ban usage, was Draft Minutes - AA-WG @ RIPE80

Hi Jordi and all,

TL;DR:  Fail2ban can deal with missing or non-responding abuse teams
         automatically, without the need to load RIPE with extra costs.


In the draft minutes I read:

     Jordi said he thinks it will work because smaller providers use more and
     more Open Source tools and it's very common to use Fail2ban. He uses it
     himself, and it takes a couple of hours to implement that. So, he
     disagreed, but pointed out there there are lots of different opinions on
     the matter.


I can confirm that abuse reporting by email works.  When I started reporting I 
noticed some ISPs were receiving lots or reports each day.  In some cases, the 
frequency suddenly dropped.  Most likely, that's the result of the ISP starting 
to work on my reports and clean up.

Based on such evidence, I recently changed my reporting script.  Now, I don't 
use Fail2ban; I use ipqbdb, which works in a similar way.  It features an 
abuserdap utility which looks up abuse addresses.  It takes as argument an 
exclusion-file, which I manually fill with the addresses that seem to be 
permanently bouncing.  Currently, the utility returns no address if either no 
address is found in RDAP, or all the addresses found there are also found in 
the exclusion file.  (See bash snippet below).

Like Fail2ban, ipqbdb bans addresses for a limited time.  Wrong passwords 
deserve a particularly short time period, because they can be given by legit 
users.  However, users coming from IP addresses not supported by a responding 
abuse team can be safely banned for a longer period.  I do one month.


On Tue 07/Jul/2020 10:33:58 +0200 PP wrote:
> The complaint to RIPE mechanism should only be an escalation mechanism when the 
> ISP does not respond.


Besides costs, that would make RIPE behave different than other LIRs.  I log 
how many RDAP lookup fail.  Most of them are in LACNIC and APNIC.  Figures are 
as follows:

Total RDAP lookups           99,   3.03% of which failed
Total RDAP lookups          107,   5.61% of which failed
Total RDAP lookups          102,   3.92% of which failed
Total RDAP lookups          140,  17.14% of which failed
Total RDAP lookups          125,   6.40% of which failed
Total RDAP lookups          115,   8.70% of which failed
Total RDAP lookups          127,   7.09% of which failed
Total RDAP lookups          113,   4.42% of which failed
Total RDAP lookups          415,  21.93% of which failed
Total RDAP lookups         1542,  39.49% of which failed
Total RDAP lookups         1996,  49.10% of which failed
Total RDAP lookups         1297,  55.05% of which failed
Total RDAP lookups          242,  31.40% of which failed
Total RDAP lookups          125,  40.80% of which failed
Total RDAP lookups          149,  43.62% of which failed
Total RDAP lookups           89,  30.34% of which failed
Total RDAP lookups           55,  18.18% of which failed
Total RDAP lookups           53,  18.87% of which failed
Total RDAP lookups           61,   9.84% of which failed
Total RDAP lookups           64,  25.00% of which failed
Total RDAP lookups         1259,  49.80% of which failed
Total RDAP lookups         1725,  60.46% of which failed
Total RDAP lookups         1746,  64.83% of which failed
Total RDAP lookups          643,  62.99% of which failed
Total RDAP lookups           73,   5.48% of which failed
Total RDAP lookups          148,   8.11% of which failed
Total RDAP lookups          163,  11.04% of which failed
Total RDAP lookups          155,  21.94% of which failed

The relevant snippet of code is below:

         let rdap_lookup++
         readarray -t <<< "$(abuserdap -x $XCLUDE -vs $rdap_url 2>> $RDAP_LOG)"
         rcpt=${MAPFILE[0]}
         if test -z "$rcpt"; then
             let rdap_failed++
             # since Tue 19 May 2020, ban for 1 month.  Don't use -l here!!
             ibd-ban -i $key -c 0 -t 2592000 -r "IP without abuse team"
         fi
         lastline="Recipient found in ${MAPFILE[1]}"

     # [...]

     if [ "$rdap_lookup" -gt 0 ]; then
         printf 'Total RDAP lookups     %8d, %6.2f%% of which failed\n' \
             "$rdap_lookup" "$(echo "100*$rdap_failed/$rdap_lookup"| bc -l)"
     fi


Best
Ale
--