[db-wg] NRTM replication inefficiencies

Dear Job, Working Group,

> On 7 Nov 2017, at 23:11, Job Snijders via db-wg <db-wg at ripe.net> wrote:
> 
> Dear all,
> 
> There may be some improvement opportunity for NRTMv3.
> 
> The problem with the RIPE NRTMv2/NRTMv3 format is that it has no
> 'end-of-stream' indicator, thus making the only way to know that the
> output has ended (vs. hung/stalled/server dead) is either by observing a
> time-out & reconnect & compare last serial, or by using single-command
> connections to the NRTM server - which means even worse performance.
> 
> When connecting to the RIPE NRTM service and issuing a command like:
> 
>    "-g RIPE:3:11012700-LAST -k"
> 
> It would be good that when the RIPE NRTM server is done sending its
> inital blurp of backlogged data, the end of that phase of the stream of
> objects is marked by the server sending something like: 
> 
>    "%CURRENT $TS You are now up to date" ($TS can be a unix timestamp)
> 
> after this server message the client knows that it can stay connected
> and that it has received all information so far.
> 
> I would also welcome an investigation into alternative approaches, (some
> not-via-WHOIS replication mechanisms), perhaps something over HTTPS can
> be done? Either way, something more robust would be useful.
> 
> Kind regards,
> 
> Job
> 
> ps. An analogy can be made between BGP route refresh as described in RFC
> 2918 and Enhanced Route Refresh as described in RFC 7313. One first one
> didn't have an "End of blurp" marker which negatively impacted its
> usefulness, the later RFCintroduced the End-of-RIB marker which is
> very useful.
> 


as part of the Whois 1.91 release, we've added an NRTM "end of stream" comment, and deployed to the Release Candidate environment.

Try connecting to whois-rc.ripe.net <http://whois-rc.ripe.net/> on TCP port 4444, and enter the command "-g RIPE:3:40956391-LAST -k". Each set of changes will be followed by an "end of stream" comment.

Changes to objects in RC will appear in the NRTM stream. Changes can be made in RC using the mntner-id (in uppercase) as the password.

Query for existing data: https://rc.db.ripe.net/db-web-ui/#/query <https://rc.db.ripe.net/db-web-ui/#/query>
Create a new object: https://rc.db.ripe.net/db-web-ui/#/webupdates/select <https://rc.db.ripe.net/db-web-ui/#/webupdates/select>
Multiple changes can be made at once using Syncupdates: https://rc.db.ripe.net/db-web-ui/#/syncupdates <https://rc.db.ripe.net/db-web-ui/#/syncupdates>

For more information on the Release Candidate environment, refer to:
https://www.ripe.net/manage-ips-and-asns/db/release-notes/rc-release-candidate-environment <https://www.ripe.net/manage-ips-and-asns/db/release-notes/rc-release-candidate-environment>

Please let us know if you see any issues with this change.

Regards
Ed Shryane
RIPE NCC

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ripe/mail/archives/db-wg/attachments/20180221/753c6151/attachment.html>