[db-wg] Decision on NWI-2 Historical queries

Hi Cynthia

I understand your concerns but feel I need to clarify the points you
made. Most people don't understand the internal workings of the
database. That is not surprising as it is quite complex. I spent 14
years with the RIPE NCC as a holistic engineer and analyst working
with the database. I know it inside out from all possible angles. I
wrote the spec for historical queries in 2013. We had no requirements,
it was just an idea that someone thought might be useful. To get
something up and running quickly I went for the low hanging fruit.
Because of the way objects are managed internally within the database,
stopping at the most recent deletion point for an object history was
simpler. It was a completely arbitrary, technical constraint. It had
no privacy reasoning behind it. At the time we released this service
in 2013, it was reviewed by the legal team and they had not asked for
this constraint for any reason. (The legal team is currently doing a
new review of the issue.)

So lets look at exactly what this constraint means. Currently we
provide some history of operational and corporate data objects. These
objects are all heavily redacted to remove anything that is considered
to be personal data. We do not provide any history of objects that are
considered to be mostly personal or security data. Any object can have
multiple versions and each version can have multiple instances. An
object may be created (v1), updated several times, deleted, re-created
(v2), updated, deleted, re-created (v3) and updated again. The
arbitrary constraint means only v3 data will be available with all
it's updates. None of the data for v1 and v2 is made available. By
dropping this arbitrary constraint all the history of this object, v1,
v2 and v3, will be available. All the data for v1 and v2 will be
redacted in the same way as v3 to remove any personal data. There is
no difference in the data for v1, v2 and v3. It is all operational and
corporate data with personal data removed. As we have generally
accepted that it is ok to release the historical, non personal, data
for v3, it should also be ok to release the historical, non personal,
data for v1 and v2. If there is any privacy concerns over the v1, v2
data, then we would have exactly the same privacy concerns over the v3
data that we currently provide.

This technical change does not change the object types that we allow
historical data to be queried for. Nor does it change any of the
attribute types within those objects that we redact. We simply provide
a complete set rather than a partial set of the operational and
corporate data we currently offer.

Lets look at some examples of what this means. Many objects have never
been deleted. Some of these have been around since this version of the
RIPE Database model was released in 2001. If you query for the history
of these objects you will get the full 21 years of history. Many
assignments are given out to end users, then deleted and re-assigned
to another end user. This can happen multiple times. This prefix will
have many versions v1...vn. Only the history of vn is currently
available, and only if this version still exists in the 'live'
database.  For many researchers of address space usage or routing
issues and abuse investigators and brokers, the history beyond vn is
useful data. Also when a resource is transferred or consolidated the
allocation object is deleted and re-created by the RIPE NCC. So for an
object which last week you could see 21 years of history, which has
just been transferred, you will only see 1 week of history now. There
is also occasionally a case of an operator who accidentally deletes
the wrong assignment and does not have an up-to-date copy of the
object as it was just before it was deleted. They currently have to
ask the RIPE NCC to supply them with the most recent version. Without
this arbitrary constraint they could just recreate it, then look up
the history themselves.

There is also the issue of NRTM. Some users have been operating an
NRTM stream for years, even decades. They have the un-redacted
versions of the history of all the resource objects since they started
streaming the updates. Redacting personal data only started a few
years ago. In many cases they have the full history, regardless of how
many times the object has been deleted and re-created. Anyone who
starts an NRTM stream now will start to build up their own history of
redacted, operational data objects that will remain in their own
database after objects are deleted and re-created. That is an accepted
practice.

This constraint really is totally arbitrary. It is rooted in technical
expedience in getting the service started. It has nothing to do with
privacy. There really is no good reason to keep this arbitrary
constraint. Either we provide historical operational data or we don't.
Offering a partial data set based on random events makes no sense.

cheers
denis
co-chair DB-WG

On Mon, 4 Apr 2022 at 23:33, Cynthia Revström <me at cynthia.re> wrote:
>
> Hi,
>
> I am sorry for the delayed response but I object to removing that constraint.
>
> It feels problematic to me from a privacy perspective, and it feels
> like the last deletion point is a fair balance between providing
> useful info and not providing too much info.
>
> I don't think this restriction should be removed, at least not without
> a very good reason to do so.
>
> -Cynthia
>
> On Thu, Mar 31, 2022 at 1:32 PM denis walker via db-wg <db-wg at ripe.net> wrote:
> >
> > Colleagues
> >
> > When I wrote the spec for historical queries, almost 10 years ago, I
> > included an arbitrary constraint to only show history back to the most
> > recent deletion point. This was to get something in production quickly
> > and see how useful it was. Over the years several people have asked
> > for this arbitrary constraint to be removed. No one has objected to
> > removing it. The co-chairs therefore recommend that this arbitrary
> > constraint be removed. The co-chairs now ask the RIPE NCC to produce
> > an impact analysis and implementation plan to remove it. We will then
> > seek a final approval from the community on the plan.
> >
> > cheers
> > denis
> > co-chair DB-WG
> >
> > --
> >
> > To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://mailman.ripe.net/