[atlas] RIPE Atlas SMTP Measurement

(Coffee is to blame for the length of this)
It's worth bearing in mind that the internet is a large place, populated 
with large, larger, very large, enormous, gargantuan....and extremely 
tiny, extremely budget-limited, personal, private servers that cannot 
afford the charges for non-productive traffic. The internet is not just 
Proofpoint or Sendgrid or "Meta" or etc.; it is also vast numbers of 
people in second- and third-world countries trying to engage with the 
world and learn, and who literally can't afford excess, unproductive bytes 
hitting their server that will cost them money.

I used to be the sole systems admin for a site that sent variously 
anywhere from a million to eleven million emails per day (yes, all opt-in, 
legitimate email for our customers, just for the record), so I've spent a 
fair bit of my life going zen analyzing logs for hours and hours. I'm now 
retired and run my own personal mail and webservers, on one t3.micro on 
AWS that costs me a few hundred bucks over the course of three years 
commitment, and a free server on each of GCE and Oracle, the former of 
which also costs me a few bucks a month in traffic charges.

I'm not poor, and can afford these charges. So I don't block any RIPE 
Atlas traffic (to my knowledge), as I don't have (or need) extreme 
restrictions on incoming traffic. I do occasionally sift through my logs 
to find new pointless traffic that's 'annoying' my systems, and may set 
something up to block them. I run a couple of stratum one timeservers from 
home, part of the ntp pool project, and regularly have to block nitwits 
(some of them ISP's themselves) who hammer my little primary RPI with 
multiple connections per minute. I have to block those malefactors simply 
because they interfere with legitimate traffic. There are countless ways 
that 'internet hooligans' and/or the clueless can cause harm with useless 
traffic.

There are potentially millions of individuals trying to contribute for the 
benefit of the internet who don't have the lack of concern for costs that 
I have. Thus why there are so many Raspberry Pi probes out there in 
non-first-world nations; and likely no small number of tiny personal 
mailservers running on a shared 56k line, the cost of which is _just_ 
bearable to maintain.

There is no one-size-fits-all on the internet. It is common - and very 
human - to view the internet and world through our own perspective.

There are other perspectives.

Okay, that's my coffee-fueled bloviation for the day, and with that, I 
acknowledge the significant digression involved...so I'll shut up now.

Cheers!

On 9/20/2022 13:04 PM, Simon Brandt via ripe-atlas wrote:
> There's literally no danger for you (recipient), if the sender 
> terminates the connection before the an e-mail has been successfully 
> transmitted. No need to ban the ip address. You would only risk to block 
> a legitimate ip address which might have trouble sending e-mails to you. 
> If you want to do this because you don't want to see such connections in 
> you logs that's fine. But i do not share your opinion, that this is a 
> common configuration.
>
> However, we have digressed from the topic.
>
> BR,
> Simon
>
>
>
> On 20.09.22 21:47, Eric Kuhnke wrote:
>> No legitimate incoming SMTP traffic comes from IPs that abort 
>> connections to my mx prior to attempting to deliver mail, so however I 
>> choose to declutter my log files has absolutely zero real world impact 
>> in deliverablity of legitimate incoming mail. Nor does it hurt anybody 
>> at the other end whatever the connect-and-do-nothing software at the 
>> other side.
>>
>> On Tue, Sep 20, 2022, 12:37 PM <ripe.net at toppas.net> wrote:
>>
>>     >> because common configurations of fail2ban [...] absolutely will
>>     ban your IP [...] after multiple connection attempts and no
>>     successful mail transfer
>>
>>     I would consider this a heavy misconfiguration. Please explain to
>>     me what incomplete SMTP connections have in common with spammers,
>>     virus/worm/trojan compromised hosts or open relay searching bots.
>>     Those bad senders WANT to _successfully_ deliver mails to you. They
>>     will never abort the connection on purpose. For example: bots which
>>     search for open relays ALWAYS try to send mails with a foreign
>>     sender and recipient domain. That's how you discover them. But as
>>     suggested, the Atlas SMTP check should not send E-Mails at all, not
>>     even send MAIL FROM: or RCPT TO: command.
>>
>>     You will not achieve mitigation of inbound spam/malware/phishing
>>     traffic by blocking IP addresses of hosts from incomplete SMTP
>>     sessions. Usually, incomplete SMTP sessions indicate a
>>     misconfiguration. For example: forced TLS enabled, but expired
>>     certificate or no matching cipher suites. But that is no reason to
>>     ban the senders! I think you have to go a little bit deeper in your
>>     logs and consider why mailtransfer was not successfull, before
>>     blocking ip addresses.
>>
>>     I am no expert for fail2ban, but as far is i know, i searches for
>>     failed login attempts. So that affects mostly authenticated SMTP
>>     connections (client E-Mail submission on tcp/465 or tcp/587),
>>     right? This should not concern us here.
>>
>>     I work with enterprise mailgateway solutions for years (mostly
>>     Proofpoint), but i have never encountered what you describe.
>>
>>     Reject or throttle because of too much connections at the same
>>     time? Yes.
>>     Reject or throttle because of too much non-existing recipient
>>     adresses? Yes.
>>     Reject or throttle because both sender and recipient domain is
>>     foreign? Yes.
>>     Reject or throttle because of bad reputation (known spammer or TOR
>>     exit node ip address)? Yes.
>>
>>     But not because of incomplete SMTP connections. From my point of
>>     view, I can not confirm that this common behaviour.
>>
>>     BR,
>>     Simon
>>
>>
>>     On 20.09.22 19:22, Eric Kuhnke wrote:
>>>     I would discourage anyone from relying upon the data from
>>>     "probing" the MX and SMTP daemons for a domain name no matter what
>>>     port they run on, because common configurations of fail2ban used
>>>     with postfix and others absolutely will ban your IP at the host
>>>     operating system level (iptables or other) after multiple
>>>     connection attempts and no successful mail transfer or
>>>     authentication.
>>>
>>>     a probe of smtpd will look not much different from the many things
>>>     out there on the internet already maliciously probing smtpd trying
>>>     to find open relays.
>>>
>>>
>>>
>>>
>>>     On Tue, 20 Sept 2022 at 09:22, Simon Brandt via ripe-atlas
>>>     <ripe-atlas at ripe.net> wrote:
>>>
>>>         Hi Michel,
>>>
>>>         Currently, HTTP and SSL are separate measurements. But for
>>>         SMTP it will probably not work this way... Encryption is not
>>>         mandatory for SMTP and thus negotiated between client and
>>>         server every time on port 25. Ports 465 and 587 are used for
>>>         Client Email Submission. You could run some measurements for
>>>         these ports as well, but for the beginning, i would focus on
>>>         server2server communication.
>>>
>>>         So here's what i think:
>>>
>>>         What we could measure:
>>>         - General reachability/availability of the e-mail service
>>>         - Response time of the remote server: time between connection
>>>         establish and first SMTP response (220 service ready)
>>>         - Which enhanced status codes are offered by the server?
>>>         - Forward/Reverse DNS matching?
>>>         - SMTP banner matching the DNS name?
>>>             - if not: what is it?
>>>         - Does the remote server offer encryption (250-STARTTLS)
>>>         - Which cipher settings are offered by the server (SSL/TLS
>>>         Version, Key Exchange Algorithms, Encryption Algorithms,
>>>         Hashing Algorithms)
>>>             - alternatively: Which cipher setting has been negotiated
>>>         between probe and server?
>>>         - Can we successfully establish a TLS connection?
>>>         - Certificate Check: is the server-certificate valid and does
>>>         it match the hostname?
>>>         - Forced Encryption: MTA-STS available and 'enforced' or
>>>         'testing' or 'report'?
>>>         - Forced Authentication: DANE available and check successfull?
>>>
>>>
>>>         What we should not do:
>>>         - send MAIL FROM: command
>>>         - send RCPT TO: command
>>>         - send DATA command
>>>         - measure e-mail delivery/roundtrip time, etc.
>>>         - Sending e-mails at all
>>>
>>>         The Atlas probe should quit the connection after the data is
>>>         collected. An actual e-mail should not be send. The target
>>>         mailserver would count this session as "incomplete" or
>>>         "aborted" which is totally fine. If someone would want to
>>>         monitor what happens after a mailserver has successfully
>>>         accepted a testmail, he should better use a monitoring
>>>         service/solution. We measure the INTERnet, not what comes
>>>         after (Intra) :)
>>>
>>>         I don't expect any "spam" problems. Since the Atlas probes
>>>         wouldn't send e-mails, there's nothing a spamfilter could
>>>         analyze. The only thing that could theoretically happen, is
>>>         that the probes ip addresses are flagged as bad by services
>>>         like Spamhaus etc. and thus be listed on DNSBL/IPBL, but i
>>>         really don't see a reason why that should happen. There
>>>         wouldn't be any activity originating from the probes which
>>>         could be classified as bad. IP addresses are normally only
>>>         blacklisted, if they send unwanted mails like spam. Also:
>>>         there are a lot of "check you mailserver" or "check your
>>>         SSL/TLS" websites. The RIPE Atlas probes would behave the same
>>>         way. No big deal.
>>>
>>>         Maybe we can think of an "extended" SMTP measurement where
>>>         RIPE Atlas sends actual e-mails, but that would require (in my
>>>         opinion), that the person who is creating the measurement
>>>         somehow provides proof, to be the owner of the target mailserver.
>>>
>>>
>>>         BR,
>>>         Simon
>>>
>>>
>>>         On 15.09.22 12:03, Michel Stam wrote:
>>>>         Hello Simon,
>>>>
>>>>         Thank you for the suggestion.
>>>>
>>>>         I have a couple of questions to get a better idea:
>>>>
>>>>           * Can you maybe describe what a SMTP measurement would look
>>>>             like?
>>>>               o Simple EHLO/HELO
>>>>               o Sending an email to a designated address (which would
>>>>                 then validate that the SMTP server is capable of
>>>>                 relaying etc.)
>>>>           * How would DNSBL or other spam prevention techniques fit
>>>>             into this?
>>>>           * What would the result be?
>>>>               o Delay until mail received
>>>>               o Response time by the actual mail server
>>>>               o Using the Received: headers to get a “traceroute”
>>>>                 like result.
>>>>           * What about the more uncommon ports such as 565
>>>>             (SMTP+SSL/TLS) or 587 (mail submission port).
>>>>           * How can we prevent this implementation from having RIPE
>>>>             Atlas be identified as a spam bot network?
>>>>
>>>>         Best regards,
>>>>
>>>>         Michel
>>>>
>>>>>         On 3 Sep 2022, at 14:48, Simon Brandt via ripe-atlas
>>>>>         <ripe-atlas at ripe.net> wrote:
>>>>>
>>>>>         Hello,
>>>>>
>>>>>         i'd like to start a discussion about having a RIPE Atlas
>>>>>         SMTP measurements.
>>>>>         First of all: yes, i know there's a big obstacle for such a
>>>>>         measurement type. A lot of probes are deployed using enduser
>>>>>         internet-connections like dsl, cable, etc. with
>>>>>         dynamic/eyeball IP addresses. Those IP spaces are usually
>>>>>         blocked by SMTP servers as approach to reduce spam mails.
>>>>>         For Example: by using blocklists like Spamhaus PBL. So a
>>>>>         proper SMTP measurement wouldn't work.
>>>>>
>>>>>         BUT we could have an easy way for RIPE Atlas probe hosters
>>>>>         to signalize, that their probe is eligible for SMTP
>>>>>         measurements:
>>>>>
>>>>>         Step 1: enable "Simple DNS Entry"
>>>>>         Step 2: create a matching reverse DNS record for the probes
>>>>>         IP address
>>>>>
>>>>>         Everybody who is able so configure a reverse DNS record for
>>>>>         his probes IP address, is most likely using a
>>>>>         non-dynamic/non-home ip address space e.g. a datacenter or
>>>>>         office network. If an ISP provides the option for his
>>>>>         customers to configure a reverse DNS record, it's most
>>>>>         likely a "business-customer" subnet which can be used to run
>>>>>         mailservers. After Step 1+2 are done, the RIPE Atlas
>>>>>         platform would easily be able to verify if forward-confirmed
>>>>>         reverse DNS is successful, and if so, automatically enable
>>>>>         that probe for SMTP measurements. Alternatively: probe
>>>>>         hosters choose their own Forward-confirmed reverse DNS name
>>>>>         and submit it on the RIPE Atlas website.
>>>>>
>>>>>         Also: if we would have STMP measurements, forward-confirmed
>>>>>         reverse DNS should be mandatory for anchors, or is it already?
>>>>>
>>>>>         Why should we have SMTP measurements?
>>>>>
>>>>>         Besides general reachability checks, we could also evaluate
>>>>>         SMTP response codes. But the most important thing for me is
>>>>>         this: the SMTP protocol is old. Very old. From a security
>>>>>         point of view, it's absolutely outdated. Most security
>>>>>         features have been added years after the initial RfC. Thus,
>>>>>         those security features are optional. Mandatory SMTP
>>>>>         encryption is not provided by the SMTP RfC. So both sides
>>>>>         have to signalize, that they are capable of encryption using
>>>>>         the STARTTLS command. An attacker (man-in-the-middle) can
>>>>>         perform a downgrade attack by suppressing the STARTTLS
>>>>>         command. So both sides are forced to fallback and
>>>>>         communicate unencrypted. RIPE Atlas would be a really good
>>>>>         tool to identify such attacks, by monitor/measure the
>>>>>         (enhanced) status codes of a target.
>>>>>
>>>>>         But there's more!
>>>>>         I see a two-sided model here. Either use the RIPE Atlas SMTP
>>>>>         measurements to monitor/measure your own mailserver by alot
>>>>>         of other RIPE probes out there, OR probe hosters could run
>>>>>         SMTP measurements originating from their own probe to find
>>>>>         out, if their own IP address is currently blocked by other
>>>>>         mailservers.
>>>>>
>>>>>
>>>>>         What do you think?
>>>>>
>>>>>
>>>>>         BR,
>>>>>         Simon
>>>>>         -- 
>>>>>         ripe-atlas mailing list
>>>>>         ripe-atlas at ripe.net
>>>>>         https://mailman.ripe.net/
>>>>
>>>
>>>         -- 
>>>         ripe-atlas mailing list
>>>         ripe-atlas at ripe.net
>>>         https://mailman.ripe.net/
>>>
>>
>
>

-- 
Paul Theodoropoulos
anastrophe.com <https://www.anastrophe.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ripe/mail/archives/ripe-atlas/attachments/20220920/ca7de1a1/attachment.html>