[db-wg] Two Proposals

Dear Ronald, Colleagues,

I've discussed Ronald's proposal internally, and we can generate a delegated stats file more frequently, keeping in mind:

(1) The current delegated stats file is generated once daily according to the "RIR Statistics Exchange Format": 

"The details of this format are to be considered a published RIR
 standard, on which other parties are expected to rely. RIRs must
 comply with this standard in every respect."

"This file is to be produced daily. The last time of any record
 for the file is 23:59:59 in the local time zone of the producing
 RIR.  (i.e. the last possible time on the specified calendar
 day in that time zone)"

Ref. https://ftp.ripe.net/ripe/stats/RIR-Statistics-Exchange-Format.txt

We could generate this file more frequently, but would need to update the format in consultation with the other RIRs first.

Alternatively, we could generate a separate file instead, whenever a change is made, or on a fixed schedule during the day.

(2) If changes in the delegated stats are made visible immediately, any mistakes made during the day will produce incorrect data in the file.

Currently we have a chance to correct any mistakes before the daily delegated stats file is generated.

(3) Inter-RIR transfers need updates in *both* RIRs delegated stats.

The delegated stats data is inconsistent until both RIRs delegated stats are updated, if one side is incrementally updated this inconsistency may be visible for longer.

(4) It's technically possible to provide updates via rsync, if the incrementally updated file is insufficient.

The RIPE NCC will produce an implementation plan to determine the time and effort needed, if the community finds this useful.

I ask the community for feedback whether publishing incremental changes to the delegated stats is useful for them.

Regards
Ed Shryane
RIPE NCC



> On 18 Sep 2021, at 03:48, Ronald F. Guilmette via db-wg <db-wg at ripe.net> wrote:
> 
> PROPOSED
> 
>    1) RIPE NCC shall be tasked to begin updating the following file on a near
>       real-time basis, as number resource assignments are transferred either
>       into or out of the authority of RIPE:
> 
>       ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest
> 
>    2) RIPE NCC shall be tasked to begin making the above file available for
>       incremental remote update via the rsync protrocol.
> 
> RATIONALE
> 
> 
> Background
> 
> At the present time the most reliable and up-to-the-minute information
> indicating which RIR any given number resource is being administered by
> is contained within the so-called "daily stats files" that are generated
> and published by the five RIRs:
> 
>  ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest
>  ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest
>  ftp://ftp.apnic.net/public/apnic/stats/apnic/delegated-apnic-extended-latest
>  ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest
>  ftp://ftp.afrinic.net/stats/afrinic/delegated-afrinic-extended-latest
> 
> Additionally, a single file representing the union of all of the above five
> files, along with additional data relating to (some but not all) IANA
> reserved number resources is also available:
> 
>  https://nro-extended-stats-from-rpki-team-prod.s3.eu-west-1.amazonaws.com/nro-extended-stats
> 
> Note however that the data in this (NRO stats) file lags behind the
> contents of the five RIR daily stats files, and the content of this
> file is therfore often less fresh and up-to-date than the information
> contained in the five RIR stats files.
> 
> 
> Problem Statement (Proposal 1)
> 
> Because the five RIR stats files are only updated once a day, and at
> different times for each RIR, and because number resources are, with
> increasing frequency, being transfered between regions, it is nowadays
> frequently the case that there will be, in effect, instances of "version
> skew" between the five RIR stats files.  Instance of such inter-RIR
> stats files version skew can render it impossible to know, based on the
> stats files, which RIR currently administers a given number resource.
> 
> Instances of internet-RIR stats file version skew can arise, and in practice
> they actually do arise as follows:
> 
>   *) RIR 'A' transfers a number resource `R' to RIR `B'.
> 
>   *) RIR `B' records its administration of `R' in its stats file which
>      is then published within 1 hour.
> 
>   *) RIR `A' records the fact that it no longer administers `R' by removing
>      the associated line from its stats file, and RIR `A's stats file is
>      then published 23 hours later.
> 
> Under the above scenario, the union of all five of the daily RIR stats files
> will indicate that *both* RIR `A" and also RIR `B' are the responsible RIRs
> for the resource `R' for a period of 22 hours.
> 
> An actual example:
> 
>    As of approximately 07:13 (-0700 / PDT) on 2021-09-17 the following blocks
>    were listed in *both* the stats file for ARIN and also the stats file for
>    RIPE:
> 
>    140.228.32.0/19
>    140.228.64.0/19
> 
>    (Evidence indicates that these 2 block have just been transferred from
>    administration by the ARIN region to administration by the RIPE region.)
> 
> The problem of version skew between the various daily stats files that
> are currently published by the various RIRs could be largely or entirely
> eliminated if all five RIRs henceforth began to update these files on a
> near real-time basis.  RIPE can and should begin doing this as an example
> to the other RIRs, which may then follow suit.
> 
> 
> Problem Statement (Proposal 2)
> 
> The RIR daily stats files, including the one that is generated and published
> by RIPE NCC, consist of plain text files which are themselves composed of
> a sequence of plain text lines.  As a general matter, very few of these
> lines change, even over time scales as long as a month.  Changes to these
> files, if any, are entirely minimal when viewed over even shorter time scales,
> e.g. a day or less.
> 
> The rsync protocol and rsync servers and clients are especially adept at 
> keeping remote copies of exactly such plain text files up-to-date and
> synchronized with a master copy stored elsewhere on the Internet.  They
> do so while minimizing bandwidth usage by transfering only those portions
> of a given file that have changed since the last time a local mirrored
> copy of the file was updated.  Publishing RIPE's stats file via rsync
> would thus be beneficial to all concerned by minimizing bandwidth usage
> on both the sending and receiving end.  Ideally, RIPE NCC could and should
> take the lead in making its stats file available via rsync.  Once it has
> done so, other RIRs can be encouraged to follow suit and do likewise with
> their own daily stats files.
> 
> 
> Regards,
> rfg
> 
> 
> P.S.  On a personal note I find it nearly unfathomable that these "daily"
> stats files are being updated and published only once a day.  This seems to
> me almost reminicent of technological solutions considered appropriate
> in the prior century.
>