[db-wg] 2017 and RIPEdb only handles 700 updates per *MINUTE*

Hi Jeroen,

I'm probably the most relevant person to asnwer your question right
now, having been a core developer during the ripe database
reimplementation project in 2013. We did that by running side by side
with the old, C version of the ripe database software. As such, we
were in no position to change the underlying database or its format.
And guess what, the underlying database is coming from the previous
server's reimplementation project, all the way from 2000. Reminds me
of the story of the NASA booster rocket and the roman horse.

There have been minor updates done over the years (like switching to
innodb and sql transactions), the basics are still the same. Its data
model is 100% optimized for query performance (since queries outnumber
updates 100 to 1 or more). And most importantly, since updates (can
and often do) have effect on one another, they are strictly serialized
(as in, single-threaded). On top of that, all updates go through a
rigorous rule checks before being accepted and taking effect.

All this translates to that update performance. Note that the update
performance is much higher for simple object types (say, person; my
last knowledge was over 2000 objects/min), and very slow for complex
ones (namely aut-num, with all the extra DNS checks).

In practice, it was never an issue that a longer update ran for hours.
This performance could be improved greatly given enough priority
though. It was always a simple case of this requiring quite some dirty
hack or major refactors, while this issue was always considered a
minor nuisance, not enough to warrant either.

BTW, if you look at the update graph you included, you see that
updates are normally almost negligible compared to your batch. Thus
most users of ripe database updates see very quick response times.

I'd maintain that unless this really stops you from achieving your
goals, it's probably best to leave the engineers work on deeper / more
pressing issues. Like the fact that the RIPE DB actually needs a
massive documentation and a day long training course even for basic
usage. And even then, people are struggling with the weird object
hierarchy and rules. While could be considered an acceptable approach
in '99, this is way outdated by today's standards and user
expectations.

Cheers,
Agoston


On Tue, Jun 20, 2017 at 2:23 PM, Jeroen Massar via db-wg <db-wg at ripe.net> wrote:
>
>
> ---------- Forwarded message ----------
> From: Jeroen Massar <jeroen at massar.ch>
> To: db-wg at ripe.net
> Cc:
> Bcc:
> Date: Tue, 20 Jun 2017 14:22:49 +0200
> Subject: 2017 and RIPEdb only handles 700 updates per *MINUTE*
> https://www.ripe.net/analyse/statistics/ripe-database/ripedb-updates
>
> (see attached png for the current version as over time it will disappear)
>
> shows a nice peak with a max of 723.47 updates.
>
> Thus either the software is really bad, the updates are rate limited, or
> the queries are so high that no further performance is to be expected.
>
>
> As alluded to in the other thread, if there was less junk (those 400k
> inet6nums for instance) then likely this would all scale better....
>
> Are the RIPE DB-WG chairs still actively involved in the maintenance of
> the RIPE Database!?
>
> As two of the co-chairs are working for RIPE NCC, are they able to tell
> why such a process is slow and why these problems that are being raised
> over and over on the various lists are not being resolved?
>
> Greets,
>  Jeroen
>