[db-wg] More-specific abuse-c

Hi Denis, WG,

> On 15 Nov 2016, at 07:36, denis <ripedenis at yahoo.co.uk> wrote:
> 
> Hi Tim
> 
> Looks like I have some competition now for writing long emails late at
> night :)

ah yes, that would be because I am in South Korea at the IETF at the moment.

> 
> You have some interesting ideas, but I would also like to point out some concerns.
> 
> On 13/11/2016 01:55, Tim Bruijnzeels wrote:
>> Hi,
>> 
>> Even though we don't have a formal problem definition yet, I felt it
>> would be useful to be more specific about possible solutions that we
>> see.
>> 
>> The short version of the below is that I believe we can have an
>> implementation that: - allows the use inheritance for "admin-c:",
>> "tech-c:" *and* "abuse-c:" - allows for a simple opt-in, but ensures
>> that nothing will break or change if you do not - simplifies
> 
> developing new ideas with an opt-in and being backwards compatible is always a good way to move forward.
> 
>> management of these contacts (in particular simple LIRs / end-users
>> with a few PI resources will be able to manage all on their
>> ORGANISATION object only) - but allows for more complicated
>> structures (inheritance can apply to just parts of the tree, and can
>> differ between branches) - simplifies finding the contacts
>> 
>> The long version, well, is a bit longer:
>> 
>> 
>> = Optional abuse-c on resource* objects
>> 
>> [*: for readability I will use the term "resource objects" to
>> indicate INET(6)NUM and AUTNUM objects]
>> 
>> Allowing an optional "abuse-c:" attribute on resource objects,
>> referencing a ROLE object with an "abuse-mailbox:" attribute would
>> already solve a lot of issues. I believe this would be better than
>> having "abuse-mailbox:" attributes directly on these objects, because
>> this way the model will be more consistent, and it will be possible
>> to reference the same ROLE object from various resources - avoiding
>> data duplication.
> 
> The reference to a ROLE object was to allow other options for reporting abuse, besides email. Also to allow for the possibility of separating automated reporting from one off, manual complaints. I still have issues about adding "abuse-c:" directly into resource objects but accept that is what many users want.
> 
>> 
>> In principle it's an option to only do this part.
>> 
>> Once done this would also allow us to migrate existing
>> "abuse-mailbox:" attributes to an "abuse-c:" ROLE object instead. We
>> can do this as a one time operation and generate ROLE objects using
>> the existing "abuse-mailbox:" - using the same maintainers as the
>> resource object has. We can of course avoid creating duplicate ROLE
>> objects - and we can avoid creating such objects altogether if the
>> value of the "abuse-mailbox:" is the same as the "abuse-c:" that
>> already occurs in the hierarchy.
> 
> I am not sure which "abuse-mailbox:" attribute you are thinking about here. This attribute has never existed in any resource object type. It was in PERSON, ROLE, MNTNER, ORGANISATION and IRT objects. It should now only be in the ROLE object. The others should all be deprecated. You cannot use any of these other object's data in a reliable way to auto generate any new data. That was one of the reasons why "abuse-c:" was introduced as there were so many places to put this data no one knew which was the reliable one.
> 
> I also don't think there should be any need to auto generate any data. AFAIK all the member's allocations were covered by an "abuse-c:" (subject to a few that were created during the re-write of the new LIR process). Many/most of the PI objects should also be covered now. It is a long time since the RIPE NCC produced any stats on "abuse-c:" coverage. So as an aside it would be a good idea to update the community on the present coverage before any changes are made.

Right, my bad. A quick grep shows 1257 occurrences of "abuse-mailbox:" in inetnum objects, in "descr:" or "remarks:" - so definitely some people have been trying to have an explicit mailbox this way - but indeed it's not an allowed attribute here.

In any case I think it would be best to discuss the deprecation of "abuse-mailbox:" as a bit of separate issue.


>> But, I believe that we can improve things much more.
>> 
>> = Finding relevant abuse-c information
>> 
>> As David pointed out there is quite a bit of hassle involved in
>> finding out what the appropriate "abuse-c:" is for a resource object.
>> This problem already exists today, but allowing "abuse-c:" on
>> resource objects add step #3 in the algorithm described by David.
>> 
>> I believe it would be better if we would just generate the proper
>> "abuse-c:" for resource objects on the server side instead, and we
>> save users (and tools) of the database the hassle of resolving this.
>> 
>> = Inheritance vs specified values
>> 
>> If a resource object is created or updated and the "abuse-c:"
>> attribute is left out, then this would indicate "inherit". In that
>> case the server can resolve and generate the appropriate value. We
>> can then "tag" the object so that we know that inheritance is desired
>> for this attribute.
>> 
>> If a user then queries and re-submits the object with minor changes
>> (a common practice), we can detect whether any change was made to the
>> "abuse-c:" attribute. If so, we can assume that the user intended to
>> stop using inheritance for this attribute. But, if the user made
>> other changes and kept the same "abuse-c:", then we can assume that
>> inheritance is still desired.
> 
> This is a bit confusing, but it may just be the wording. I presume you mean if a user makes a minor change to a 'resource' object. If they add an "abuse-c:" attribute directly to the resource object they wish to stop using inheritance from this point in the hierarchy. If they didn't add an "abuse-c:" attribute in the minor change to the resource object then it is still inherited from the ORGANISATION object.
> 
> With many object types involved in this process it is better to always spell out explicitly which object you mean at each step. It makes the spec longer, but there is no doubt about exactly what you mean.

I think I clarified this in my response to Sander's email.

>> Now if something changes higher up in the hierarchy, e.g. the
>> "abuse-c:" attribute of the ORGANISATION object is modified, then the
>> server can follow all the resource objects below that use inheritance
>> and generate the appropriate new values.
>> 
>> Note that this is not duplication of data, it's just explicit
>> duplication of references. And since it's handled automatically there
>> is no risk of human error.
> 
> It is also not clear exactly what you mean here. I am guessing you want to expand out the inherited reference to an abuse ROLE object that is specified in the ORGANISATION object by allowing the server to add the "abuse-c:" attribute to each resource object referencing the ORGANISATION object. And in some way tag these resource object to say they are using the inherited reference rather than a directly added user reference. Then if the "abuse-c:" value is changed in the ORGANISATION object this change is replicated throughout the hierarchy of resource objects that are tagged as using inheritance.

yes

> If this is what you man then there are issues and consequences of doing this. Firstly this IS duplication of data as you are adding an attribute to all the child objects. But this is an acceptable way these days of handling inheritance. But you have to balance out the cost of replicating this change vs looking up the inherited value from the parent object. In doing this calculation keep in mind that some of the large telcos have 200k or even by now 300k resource objects referencing an ORGANISATION object. So by changing the value of the "abuse-c:" in their ORGANISATION object they will trigger an update to maybe 300k objects in the database. That will put quite a load on the update server. Especially if they realise they made a mistake and change it again a few minutes later. Given that this update can take minutes and two such updates could be running in parallel and there is no versioning of objects, you could hit a race condition with these updates being done in the wrong order on some objects.

ok, good point. I think we can resolve this though.

I imagine that we can use an asynchronous process, and use a fairly low priority queue for this. This would ensure that these updates don't block other updates that may be more operationally important (e.g. related to route objects). Most organisations do not have such a large amount of objects though, and for them this approach should not result in (much) notable delays. If we go down this path and have a queue then we can also ensure that repeated updates happen in order.

> You also need to think about the history of the resource objects. What would be the intended or desired action here? Do you want a change to an ORGANISATION object to effect the "last-modified:" attribute in resource objects? Will this be a real update to the resource objects?

Another good point. I think this depends on what the WG would consider the least surprising. I am inclined to say that "last-modified:" should be updated because it's an actual user initiated change in content.

>> = Going further -> using inheritance on "admin-c:" and "tech-c:"
>> 
>> Currently both "admin-c:" and "tech-c" are mandatory on resource
>> objects, and they are optional on ORGANISATION objects. Contrary to
>> "abuse-c:" there is currently no inheritance for these attributes.
>> 
>> I propose that inheritance is added for these attributes and values
>> are always generated server side, exactly as I described above for
>> "abuse-c:". This means that in order to simplify the management of
>> "admin-c:" and "tech-c:" you would only have to re-submit your
>> resource object and leave these attributes out. The server would then
>> know to use inheritance and track it per attribute, e.g. you can use
>> inherit on "admin-c:", but not the other "tech-c:" if you prefer.
> 
> I am pleased to see you are considering using inheritance on these other attributes. That was always the plan when we first introduce "abuse-c:". I presume you will also include the fourth contact attribute "zone-c:" in the DOMAIN object as well for inheritance.

I have no objections to this, but it would mean that we also need to allow "zone-c:" in ORGANISATION objects.

So to summarise that would then make it two (ok 4) data model changes:
 - Add "zone-c:" as optional to ORGANISATION
 - Add "abuse-c:" as optional to INET(6)NUM and AUTNUM

I don't expect issues with this, but just mentioning explicitly.


>> Because these attributes are optional on ORGANISATION objects, the
>> server can generate an error in case inheritance was asked, but there
>> is no value in the hierarchy.
>> 
>> = Wrapping it up..
>> 
>> I believe that this approach for "abuse-c:" *and* "admin-c:" and
>> "tech-c:" will make the management of all this contact information
>> much easier, and consistent.
> 
> and "zone-c:"
> 
>> 
>> It will not require any changes in approach from users. If you are
>> currently managing your resource objects and maintaining "admin-c:"
>> and "tech-c:" there, you can continue to do so and nothing will break
>> or change. If you do not wish to use a different "abuse-c:" you can
>> just continue to leave it out and the server will generate it for
>> you.
> 
> OK I am thinking now of other scenarios. These may be covered in your plans but maybe not specified here.
> 
> Whatif I choose to opt-in to inheritance, add the values in the ORGANISATION object and remove them from resource objects. The inherited values are auto generated by the server in all the resource objects. So if it is the same value I started with, my resource objects just look the same as they did before I opted in. As a user, how do I know I am using inheritance? If I now query and re-submit an INETNUM object with a change to the "notify:" attribute, I am submitting a resource object 'with' an "abuse-c:", "admin-c:" and "tech-c". How will the server know I still want to use inheritance? It looks like I have just turned off inheritance from this point in the hierarchy by directly specifying these values that were shown in the query. But to me as a user it looks the same. If I now update the values in the ORGANISATION object, the auto generated, replicated data will not be changed in these resource objects I have just updated. My data is now out of step and as a user I don't have a clue this has gone wrong.

I believe my answer to Sander's email should clarify this.

In a nutshell I believe that the decision to inherit is done as opt-in per object, and per attribute for that object. The opt-out is also per object and attribute. If you wish to opt-out again, but keep the exact same contacts you need to use two updates. I don't see a way around this if we are to avoid surprises to automated updates using query-edit-submit strategy: i.e. we would treat either absence of the inherited attribute, or an exact match with what was there to mean that there was no intention to change the inheritance strategy for this attribute.

> (Incidentally this is why I proposed NWI 1)
> 
>> 
>> On the other hand it's very simple to opt-in for users who do want to
>> use this. Just set up "admin-c:", "tech-c:" and "abuse-c:" on your
>> ORGANISATION object, and resubmit your resource objects without these
>> attributes to start using inheritance.
> 
> Again large members may have hundreds of thousands of resource objects. To resubmit these objects without the attributes you want to inherit is a major undertaking.

Yes. Well, you can script it fairly easily, but you would have to use mailupdates to make sure that your updates come through asynchronously.

That said, we can investigate providing a UI and/or REST API call to facilitate this better. This is probably useful to have even for organisation that have 10s of resources rather than 1000s.

> 
>> Then when for example you want
>> to remove one "admin-c:" you only need to change your ORGANISATION
>> object and everything else will follow.
>> 
>> For more complicated topologies you can override values for specific
>> resource objects later, or simply not opt-in to inheritance to begin
>> with - since this is a per attribute and per object choice. You can
>> also override values at one point in the resource tree, and use
>> inheritance of those values again below.
> 
> Are you saying here that if a user manually sets an "admin-c:" at some point in their network, this manually added value will then be inherited by all the more specific resource objects? The server will then replicate this value in all the more specific objects.

Yes I am suggesting that in a case like:

 BIG INETNUM (inherits) - ORG (sets default)
 MEDIUM INETNUM (overrides)
 SMALL INETNUM (inherits)

The SMALL INETNUM inherits from the covering MEDIUM INETNUM, and any updates done there. If MEDIUM INETNUM would start to inherit, then the inherited attributes would also propagate to SMALL.

>> I hope the above makes sense and I would love to hear what you think.
>> Does this solve the issues? Does this introduce problems that I
>> overlooked?
> 
> Another issue you need to think about is documenting all the fine points about this process and see if it can be described in simple terms. The trainers also need to explain this to members on the RIPE Database courses.

Sure, documentation will be updated of course.

I do believe that the general concept will be easy to convey though: as a best practice set your defaults on your ORGANISATION object so that you can maintain all your contacts in one place, but override where necessary.

Cheers
Tim



> 
> Some food for thought :)
> cheers
> denis
> 
>> 
>> 
>> Kind regards,
>> 
>> Tim Bruijnzeels
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> On 11 Nov 2016, at 01:42, fransossen at yahoo.com wrote:
>>> 
>>> 
>>> 
>>> Hi,
>>> 
>>> Going down the path of allowing some form of abuse-email directly
>>> on the resources would indeed work for almost everyone and keep the
>>> centralised abuse-c system for the ones who do not need a more
>>> complicated setup.
>>> 
>>> The IPv4/6 PI holders would not be able to do the "granularity"
>>> with their single assignments where several abuse-mail might be
>>> needed in network segments from within a single inet(6)num. But
>>> that would be one more of the restrictions to add to the lists of
>>> differences between PI and PA since assignments, a large PI
>>> assignment already cannot be broken into smaller assignment unless
>>> converted to Allocated PA.
>>> 
>>> 
>>> It would still confuse any person inheriting the job of reviewing
>>> the Data once all the entries are already made. One would actually
>>> have to review the abuse contacts on:
>>> 
>>> 
>>> 1) Your own main org object and its abuse-c:.
>>> 
>>> 2) Any potential org objects that may contain an abuse-c:
>>> referenced on your more specific inet(6)nums
>>> 
>>> 3) Any abuse-mail on any inet(6)nums (parent and more specifics)
>>> and aut-nums.
>>> 
>>> 4) Figure out the hierarchy and order of which abuse mail-box is
>>> displayed based on the intricate DB rules that were decided in
>>> order to know if the displayed email address is the intended one or
>>> not for those DB entries.
>>> 
>>> 
>>> This reminds me of the RIPE DB update notifications system:
>>> 
>>> https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-database-documentation/notifications/9-2-notification-messages
>>> 
>>> 
>>> 
> "notify:"
>>> "mnt-nfy:" "upd-to:" "irt-nfy:" "ref-nfy:"
>>> 
>>> Each one of them makes sense individually, but really confuses
>>> people in general, notifiers require quite some DB knowledge once
>>> you want to know why you received a notification and even more when
>>> you want to know why you didn't receive one!
>>> 
>>> Complicated to setup, complicated to debug, but once understood, it
>>> does exactly what it is intended for and provides flexibility. All
>>> and all it might be the only way to achieve the intended results
>>> though.
>>> 
>>> Cheers, David Hilario
>>> 
>>> 
>>> 
>>> 
>>> On Tuesday, November 8, 2016 7:02 PM, Tim Bruijnzeels
>>> <tim at ripe.net> wrote: Dear WG,
>>> 
>>>> Problem statement: ==================
>>>> 
>>>> It is currently not possible to specify alternative abuse
>>>> contacts for different resources held by the same organisation in
>>>> the RIPE Database. This can be problematic.
>>>> 
>>>> For example for abuse contact management for big organisations
>>>> which want multiple abuse contacts for different parts of the
>>>> organisation. The org is using a lot of PI resources, some of
>>>> them legacy. Currently they all use the same ORGANISATION object
>>>> which is inextricably linked to the same abuse-c mailbox.
>>>> [example taken from Sebastian's first email]
>>>> 
>>>> Another example applies to different allocations held by a single
>>>> LIR. The reference to the LIR's ORGANISATION object is maintained
>>>> as part of the registry managed by RIPE NCC, and therefore all
>>>> allocations will have the same abuse-c mailbox. There may however
>>>> be good operational reasons for having different contacts. [same
>>>> issue, slightly different example also mentioned by David]
>>>> 
>>>> It should be noted however that it is considered useful that in
>>>> cases where there is no need to have a different abuse-c mailbox,
>>>> the abuse-c can be defined on an ORGANISATION object that is
>>>> referenced from an INET(6)NUM object, and have it apply to more
>>>> specific INET(6)NUM objects through inheritance. This avoids data
>>>> duplication, and makes it easier to manage this information.
>>>> 
>>>> But on the other hand it should also be noted that for a lot of
>>>> less experienced users of the RIPE database it has proven to be
>>>> difficult to find the applicable abuse contact for a resource,
>>>> following this hierarchy. For this reason the abuse contact is
>>>> now mentioned as a comment in whois output, and is highlighted
>>>> explicitly on the web query results.
>>>> 
>>>> As a somewhat related issue I would also mention the following:
>>>> 
>>>> Management of administrative and technical contacts in the RIPE
>>>> Database is done differently, and the challenges of maintaining
>>>> that data is somewhat different. While here it is possible, or
>>>> even mandatory, to have explicit references to admin-c or tech-c
>>>> contacts for each resource object that may differ from the
>>>> objects, it is *not* possible here to inherit these contacts from
>>>> a covering INET(6)NUM or ORGANISATION object. This results in an
>>>> increase in maintenance burden for this data and therefore makes
>>>> it more difficult to maintain accurate data.