[anti-abuse-wg] 2019-04 Review Phase (Validation of "abuse-mailbox")

Hi Jordi,

Please find below the answer to your questions:
 - We have provided our own data from 2019 to be used as a comparison. I do see why the numbers derived from our data would be a gross overestimation and why any other RIR set of data would provide a more accurate estimation. Even if this number is a bit higher or lower, it does not change the impact that we are stating in our impact analysis.

I had a quick look at the reported numbers from APNIC and if we use that set of data for the comparison, the percentage of estimated manual workload would actually be higher. I did not include them as I do not feel they are relevant.

 - The data from 2020 can not be correlated with the data from 2019 before the end of the year. 2019 was the implementation year so we are not using the same rhythm in 2020.

 - A slow start would not be possible in this case as it takes us one year to validate 7.5% of the abuse mailboxes using the same process. We can not reliably estimate how many years would be needed to finalise the implementation if it left on a best effort basis. Plus until the implementation is complete we have the current policy still in effect.


We are happy to provide any statistics requested, however I would not like to turn the review phase into a discussion between the proposer and the RIPE NCC about a better implementation plan, more efficient work or a smarter deployment. I do not feel this is within the scope of the impact analysis or the review phase.

The goal of the impact analysis is for the RIPE NCC to provide relevant supporting information to facilitate the discussions about the proposal and provide some projections about the possible impact if it were to be accepted. Thus, I would like the community to discuss this proposal.

Everything that we mentioned in the impact analysis has been carefully considered and it is the RIPE NCC's estimated impact if the proposal is approved.

We have tried to provide accurate projections, providing an explanation on how we estimate these numbers and what we based the estimation on.

 - We expect the creation of 64,000 tickets per given year.
 - A considerable amount of these tickets will need to be checked manually. (in 2019, this was ~30%. In reality it can be higher or lower)
 - We estimate 10 times the amount of workload that is currently spent on abuse-c validation.
 - We expect this to be a regular workload. Not a one time thing.
 - The workload exceeds the current capacity of Registration Services and a significant increase in FTEs would be required.

Full Impact Analysis:
https://www.ripe.net/participate/policies/proposals/2019-04#impact-analysis

Kind regards,
--
Petrit Hasani
Policy Officer
RIPE NCC





> On 20 Jul 2020, at 20:42, JORDI PALET MARTINEZ via anti-abuse-wg <anti-abuse-wg at ripe.net> wrote:
> 
> Hi Petrit,
> 
> Tks for the quick response!
> 
> Responding in-line below.
> 
> 
> El 20/7/20 20:07, "Petrit Hasani" <phasani at ripe.net> escribió:
> 
>    Hi Jordi,
> 
>    Thank you for your feedback. I will try to address each point, please do let me know if I miss something.
> 
>    I would like to start with your last comment about implementation in other RIRs, which may help clarify some of your other concerns as well.
>    The Impact Analysis is based on the current RIPE NCC policies and procedures. The RIPE NCC will manually follow up with the members to try and fix the incorrect details (not validated). This manual follow up generates the mentioned workload. This manual follow up may not be happening in other regions which allows for a more automated approach.
> 
> [Jordi] I guess I miss-explained my point on this. The other RIR that already implemented a more complex policy that this proposal is APNIC. What I'm saying is that it may be some correlation between their figures of % of how much manual work they need to do if the validation is failing, because I think the figures that you have extrapolated from the actual validation, are really overestimated. I fully understand that it will be anyway and estimation, because regional differences, but I really think that reading the IA with those numbers, and making very clear that is an estimation, can provide a very wrong/biased view.
> 
>    1) The current validation process expresses the RIPE NCCs understanding of policy proposal 2017-02. Our understanding was shared in the Impact Analysis at the time (the validation process was included as well) and the policy was approved based on that understanding. Our view is that the purpose of RIPE-705 is being fulfilled under the current validation process.
> 
> [Jordi] Yes, I understand that, but doesn't preclude that I still think, even if I'm not native English speaker, that is not a correct interpretation of literal English text. "required and intended" say all. If you require something to intend to be able to send abuse reports, and the intent is not fulfilled because the mailbox doesn't work ...
> 
>    However, we are not claiming in this impact analysis that the out of 92.5% of the email addresses currently validated automatically, there does not exist the possibility that a certain % does not reach the intended party due to email address belonging to somebody else, an unchecked or a full mailbox, etc. This is the reason we stated on our Impact:
> 
>    "If this proposed policy change reaches consensus, an improvement of the registry data is expected as more “abuse-mailbox:” attributes will be current and correct."
> 
> [Jordi] We agree here, no doubt. I just wanted to stress the point that many folks in the community may still believe that we have over 92.5% correct abuse mailboxes, which is not the case (I said unless, but I understand its very difficult to have that data, unless we do a different validation ...).
> 
>    2) We have had only one full year validation (2019) so we can not provide accurate comparison before the end of 2020 on how has the situation has changed.
> 
> [Jordi] I don't recall right now how has been done that during the 2019, or even a bit before (if it started before), but my point was to have the numbers across the different months or quarters. So, if you did, for example, 25% of the validations each quarter, we could see differences with the 1st and 2nd quarter in 2020. I'm assuming that because that validation was yearly, the validated data from 1Q2019 has been repeated in 1Q2020, and so son. We could even see if the people that was manually called on for correcting invalid data is now not being called again in a given %. This will tell us that the amount of workload usually will be reduced.
> 
>    3+4) All current “abuse-mailbox:” attributes will be validated in batches.
> 
> [Jordi] This may be good enough, if you can adjust the "pause" in between batches to allow manual verifications and the size of the batches. An alternative is to have, as part of the automation, a distribution based on the week or month of the original contract for each LIR, etc. Many ways to optimize it, so the workload is evenly distributed as much as possible.
> 
>    5+6) I would like to clarify that “10 times the current level” refers to workload, not FTEs. We have not made a calculation on the exact number of FTEs needed. We only state that a significant number is expected due to the workload level.
> 
> [Jordi] Wow, that can bring to a very different interpretation and result ...
> 
>    We tried to explain in the Impact Analysis how we came up with the number ~32,000 tickets per validation round and estimated the number of tickets that would require a manual follow up (around 30% according to 2019’s figures). The number of ticket which need a manual follow up could improve over the years. However at this time the RIPE NCC can not reliably estimate it.
> 
> [Jordi] But then what you're missing is that the proposal gives the RIPE NCC the freedom to, in case of a huge number of manual validations are needed, instead of doing the 1st validation in 6 months, you may need 12 months, or even 18. Not an issue. No need to have more FTEs for that, if the board or whoever is in charge of taking the decision don't want to hire more staff (that was my goal on that part of the proposal -> you manage it at your own pace).
> 
> One more advantage of this manual validations is that it may help to discover LIRs, resources, etc., which even if they "pay the bills" automatically, are not really taking care of the resources ... People in charge let the company, that department doesn't exist (but admin pay all the yearly bills, this happens, I've seen it), etc. This was not my goal with the proposal, but I think it adds some additional value, I just realized it.
> 
>    I hope the explanation above is helpful. Please let us know if you have any other concerns.
> 
> [Jordi] Yes, thanks a lot, specially the FTE point.
> 
>    --
>    Petrit Hasani
>    Policy Officer
>    RIPE NCC
> 
> 
> 
> 
> 
>> On 20 Jul 2020, at 16:36, JORDI PALET MARTINEZ via anti-abuse-wg <anti-abuse-wg at ripe.net> wrote:
>> 
>> Hi Petrit,
>> 
>> Tks for the impact analysis!
>> 
>> However, I think there are some aspects not well covered.
>> 
>> 1) It is clear, unless you can provide stats about that, that we don't really know if the 92.5% of the automated validations check are *really* correct in the sense of being able to receive emails (due to mistakes, or on purpose), as some % may be reaching a null in-box, a mailbox that bounces because is full, a mailbox that bounces because is misconfigured, etc. As a consequence of that, the current validation is not really fulfilling the actual purpose of the RIPE-705, because "it is required to contain ... which is intended for receiving ...". If emails can't be received at least a % of the 92.5% is not being validated.
>> 
>> 2) Maybe I got it wrong, but I think it is important to see the progress of tickets that where needed to open in different passes of RIPE-705. It is expected that in each pass you have less and less failing abuse-c mailboxes, right? Otherwise, it will be an indication that some LIRs aren't really doing the job to comply with RIPE-705.
>> 
>> 3) Just to make it clear: Changing the validation period is let on-purpose, as an operational aspect to the RIPE NCC. I think it is a feature, not an issue. This also allows a slow-start, as RIPE NCC did with the implementation of RIPE-705, so it allows to avoid the extra overload indicated in the IA. May be a full year or even 1.5-2 years are needed in the first pass. Not an issue, you can accommodate the internal process to the available man power for manual follow up.
>> 
>> 4) The proposal doesn't specify that you need to run all the validations on the same day. I expect the system to be smart, and for example consider an even split of validations per day, which you can tune, depending on what happened every previous week, so not to overload the resources needed for manual follow up. This is also in line which 3 above, and I understand is also the way RIPE-705 was implemented (at least initially).
>> 
>> 5) I really feel that expecting that 32.000 tickets for each round will be created is very exaggerated. If that's the case, that will probe my point 1 above and indicate that we have a real problem. Even if that's the case, a smart slow-start process will not require 10 times the actual FTEs vs the current level. Again, it is important to insist that it should be done smartly and, in that sense,  it is a huge mistake, in my opinion, not considering it in the IA, because it provides a very biased view.
>> 
>> 6) Even if it is the case that in the first round we have 32.000 tickets, this is temporary, because following years will not be the same, otherwise, we have a different kind of problem with policy compliance.
>> 
>> One possible indication of if this really creates so much trouble, even if all the validations are sent on the same "day", will be to ask to APNIC, which already implemented a much stricter proposal a year ago, if I recall correctly. I understand that it is just an indication, different culture, NIR there/no here, etc., etc. LACNIC is on their way as well, but I don't know when it will be implemented yet.
>> 
>> Regards,
>> Jordi
>> @jordipalet
>> 
>> 
>> 
>> El 20/7/20 15:08, "anti-abuse-wg en nombre de Petrit Hasani" <anti-abuse-wg-bounces at ripe.net en nombre de phasani at ripe.net> escribió:
>> 
>>   Dear colleagues,
>> 
>>   Policy proposal 2019-04, "Validation of "abuse-mailbox"", is now in the Review Phase.
>> 
>>   This proposal aims to have the RIPE NCC validate "abuse-c:” information more often and introduces a new validation process.
>> 
>>   The RIPE NCC has prepared an impact analysis to support the community’s discussion.
>> 
>>   You can find the proposal and impact analysis at:
>>   https://www.ripe.net/participate/policies/proposals/2019-04
>>   https://www.ripe.net/participate/policies/proposals/2019-04#impact-analysis
>> 
>>   And the draft documents at:
>>   https://www.ripe.net/participate/policies/proposals/2019-04/draft
>> 
>>   As per the RIPE Policy Development Process (PDP), the purpose of this four week Review Phase is to continue discussion of the proposal, taking the impact analysis into consideration, and to review the full draft RIPE Policy Document.
>> 
>>   At the end of the Review Phase, the working group chairs will determine whether the WG has reached rough consensus. It is therefore important to provide your opinion, even if it is simply a restatement of your input from the previous phase.
>> 
>>   We encourage you to read the proposal, impact analysis and draft document and send any comments to <anti-abuse-wg at ripe.net> before 18 August 2020.
>> 
>>   Kind regards,
>> 
>>   --
>>   Petrit Hasani
>>   Policy Officer
>>   RIPE NCC
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> **********************************************
>> IPv4 is over
>> Are you ready for the new Internet ?
>> http://www.theipv6company.com
>> The IPv6 Company
>> 
>> This electronic message contains information which may be privileged or confidential. The information is intended to be for the exclusive use of the individual(s) named above and further non-explicilty authorized disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited and will be considered a criminal offense. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited, will be considered a criminal offense, so you must reply to the original sender to inform about this communication and delete it.
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 
> **********************************************
> IPv4 is over
> Are you ready for the new Internet ?
> http://www.theipv6company.com
> The IPv6 Company
> 
> This electronic message contains information which may be privileged or confidential. The information is intended to be for the exclusive use of the individual(s) named above and further non-explicilty authorized disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited and will be considered a criminal offense. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited, will be considered a criminal offense, so you must reply to the original sender to inform about this communication and delete it.
> 
> 
> 
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: </ripe/mail/archives/anti-abuse-wg/attachments/20200722/5bca8530/attachment.sig>