This archive is retained to ensure existing URLs remain functional. It will not contain any emails sent to this mailing list after July 1, 2024. For all messages, including those sent before and after this date, please visit the new location of the archive at https://mailman.ripe.net/archives/list/[email protected]/
[db-wg] Clean up of unreferenced person/role objects
- Previous message (by thread): [db-wg] Clean up of unreferenced person/role objects
- Next message (by thread): [db-wg] Clean up of unreferenced person/role objects
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Andreas Polyrakis
apolyr at noc.ntua.gr
Wed Apr 18 13:50:07 CEST 2007
Denis Walker wrote: > Introduction > ------------ > > According to our database consistency statistics program (dbconstat [1]) we > currently have 460,573 unreferenced person/role objects [2]. Some of these > may be maintained, but are still unreferenced. Any personal data not > referenced by Internet resources do not fit within the purpose of the RIPE > Database. They should not be stored in the RIPE Database beyond a reasonable > 'work in progress' period. > > The RIPE NCC has had a mandate to delete these unreferenced person/role > objects since RIPE 40: > > http://www.ripe.net/ripe/wg/db/minutes/ripe-40.html (2001) > > At RIPE 41, the Database Working Group agreed that "maintained objects will > now be removed" and "gave a mandate to the RIPE NCC to continue with the > cleanup process." > > http://www.ripe.net/ripe/wg/db/minutes/ripe-41.html (2002) > > The cleanup process of 2003 > (http://www.ripe.net/db/news/unref-cleanup-200304.html) involved using a > script to run periodic cleanups. This script was put in place. However, it > failed about 18 months ago. Because of other priorities, we have not had > the time to examine this issue again until now. > > The graph showing the increase in these unreferenced person objects [3] > indicates that the cleanup script appeared not to be performing correctly. > At the start of 2006, there were about 300,000 unreferenced person/role > objects. Since then there has been a steady increase with a large increase > of almost 50,000 in February 2007. The graph also shows a slightly higher > rate of increases in these objects since February 2007. > > Because of the high number now of unreferenced person objects, we want to > raise the issue with the community again and propose the below procedure > to clean them up. > > Because redundant personal data is a serious data protection issue, we want > to take a new approach to this in the future. Once the initial cleanup is in > progress, we will create a new proposal for a new, regular cleanup > procedure. This will be sent to the Data Protection Task Force [4] and then > to the rest of the community. > > > One time bulk cleanup procedure > ------------------------------- > > Month 1 > > * Select 80,000 unreferenced person/role objects. > > Month 2 > > * Check selected person/role objects. > * Those still unreferenced: > o Delete using normal update process. > o 2000 objects per update message. > o Run updates overnight (Saturday/Sunday). > o One update every 15 minutes. > o This should avoid any unnecessary load on the servers. Hello, I suppose you have already foreseen this, but just in case: What about RIPE DB mirroring? Is there a chance that this mass deletion will cause anomalies in the NRTM mirroring? Regards, Andreas > * Select next 80,000 unreferenced person/role objects. > > Month 3 > > * Repeat process until complete. > > > Notes > ----- > > This procedure will take about 6 months to clear the current backlog, not > including the extra time that may be necessary due to the increases over > that period. > > Because of the high numbers involved, we prefer NOT to send out individual > e-mail notifications, either before or after deletion. This is to prevent a > high load on our mail servers, especially in the event of a high number of > bounced e-mails. > > This means that there will be no individual announcements to listed e-mail > addresses before the deletion and none of the usual update notifications. > This also means that even if the objects are maintained, the maintainer will > not be notified directly about the deletion of their unreferenced > person/role object. > > We will, however, announce the cleanup to the Working Group mailing lists > and as a news item on our web site home page. > > The worst problem that can occur is that someone will enter a reference to > their person/role object just as we delete it. However, as we are only > deleting unreferenced person/role objects, the time needed to re-create them > is minimal. We suspect that a very large proportion of the unreferenced > person/role objects that we will be deleting are abandoned objects that are > no longer used. > > > References > ---------- > > [1] dbconstat > http://www.ripe.net/projects/dbconstat/index.html > > [2] current unreferenced person/role objects > http://www.ripe.net/projects/dbconstat/html/cons-current.html > > [3] graph of unreferenced person object increase > http://www.ripe.net/projects/dbconstat/cons-unrefpero.html > > [4] Data Protection Task Force > http://www.ripe.net/ripe/tf/dp/index.html > -- ========================================= Andreas Polyrakis GRNET/NTUA NOC apolyr at noc.ntua.gr Office: +302107722409 Cell: +306972832445 =========================================
- Previous message (by thread): [db-wg] Clean up of unreferenced person/role objects
- Next message (by thread): [db-wg] Clean up of unreferenced person/role objects
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
[ db-wg Archives ]