[ncc-services-wg] Re: [db-wg] Clean up of unreferenced person/role objects

Denis Walker wrote:
> Introduction
> ------------
> 
> According to our database consistency statistics program (dbconstat [1]) we
> currently have 460,573 unreferenced person/role objects [2]. Some of these
> may be maintained, but are still unreferenced. Any personal data not
> referenced by Internet resources do not fit within the purpose of the RIPE
> Database. They should not be stored in the RIPE Database beyond a reasonable
> 'work in progress' period.
> 
> The RIPE NCC has had a mandate to delete these unreferenced person/role
> objects since RIPE 40:
> 
> http://www.ripe.net/ripe/wg/db/minutes/ripe-40.html (2001)
> 
> At RIPE 41, the Database Working Group agreed that "maintained objects will
> now be removed" and "gave a mandate to the RIPE NCC to continue with the
> cleanup process."
> 
> http://www.ripe.net/ripe/wg/db/minutes/ripe-41.html  (2002)
> 
> The cleanup process of 2003
> (http://www.ripe.net/db/news/unref-cleanup-200304.html) involved using a
> script to run periodic cleanups. This script was put in place. However, it
> failed about 18 months ago. Because of other priorities, we have not had
> the time to examine this issue again until now.
> 
> The graph showing the increase in these unreferenced person objects [3]
> indicates that the cleanup script appeared not to be performing correctly.
> At the start of 2006, there were about 300,000 unreferenced person/role
> objects. Since then there has been a steady increase with a large increase
> of almost 50,000 in February 2007. The graph also shows a slightly higher
> rate of increases in these objects since February 2007.
> 
> Because of the high number now of unreferenced person objects, we want to
> raise the issue with the community again and propose the below procedure
> to clean them up.
> 
> Because redundant personal data is a serious data protection issue, we want
> to take a new approach to this in the future. Once the initial cleanup is in
> progress, we will create a new proposal for a new, regular cleanup
> procedure. This will be sent to the Data Protection Task Force [4] and then
> to the rest of the community.
> 
> 
> One time bulk cleanup procedure
> -------------------------------
> 
> Month 1
> 
>     * Select 80,000 unreferenced person/role objects.
> 
> Month 2
> 
>     * Check selected person/role objects.
>     * Those still unreferenced:
>           o Delete using normal update process.
>           o 2000 objects per update message.
>           o Run updates overnight (Saturday/Sunday).
>           o One update every 15 minutes.
>           o This should avoid any unnecessary load on the servers.

Hello,

I suppose you have already foreseen this, but just in case:

What about RIPE DB mirroring? Is there a chance that this mass deletion 
will cause anomalies in the NRTM mirroring?

Regards,
Andreas

>     * Select next 80,000 unreferenced person/role objects.
> 
> Month 3
> 
>     * Repeat process until complete.
> 
> 
> Notes
> -----
> 
> This procedure will take about 6 months to clear the current backlog, not
> including the extra time that may be necessary due to the increases over
> that period.
> 
> Because of the high numbers involved, we prefer NOT to send out individual
> e-mail notifications, either before or after deletion. This is to prevent a
> high load on our mail servers, especially in the event of a high number of
> bounced e-mails.
> 
> This means that there will be no individual announcements to listed e-mail
> addresses before the deletion and none of the usual update notifications.
> This also means that even if the objects are maintained, the maintainer will
> not be notified directly about the deletion of their unreferenced
> person/role object.
> 
> We will, however,  announce the cleanup to the Working Group mailing lists
> and as a news item on our web site home page.
> 
> The worst problem that can occur is that someone will enter a reference to
> their person/role object just as we delete it. However, as we are only
> deleting unreferenced person/role objects, the time needed to re-create them
> is minimal. We suspect that a very large proportion of the unreferenced
> person/role objects that we will be deleting are abandoned objects that are
> no longer used.
> 
> 
> References
> ----------
> 
> [1] dbconstat
> http://www.ripe.net/projects/dbconstat/index.html
> 
> [2] current unreferenced person/role objects
> http://www.ripe.net/projects/dbconstat/html/cons-current.html
> 
> [3] graph of unreferenced person object increase
> http://www.ripe.net/projects/dbconstat/cons-unrefpero.html
> 
> [4] Data Protection Task Force
> http://www.ripe.net/ripe/tf/dp/index.html
> 

-- 
=========================================
Andreas Polyrakis
GRNET/NTUA NOC
apolyr at noc.ntua.gr
Office: +302107722409 Cell: +306972832445
=========================================