[db-wg] Severe performance problems with RIPE whois server

So the new kid on the block for issuing IRR querys is bgpq3:

http://snar.spb.ru/prog/bgpq3/

(Thanks to Job Snijders for pointing it out).

Here are some comparative results for bgpq3 and irrtoolset using a random
large-ish as-set with ~5000 ASNs:

> ubuntu1:/home/nick/bgpq3-0.1.16> time bgpq3 AS-NEOT > /dev/null
> 0.144u 0.032s 0:04.26 3.9%	0+0k 0+0io 0pf+0w

> ubuntu1:/home/nick/trunk/src/peval> time ./peval -h whois.radb.net -protocol irrd AS-NEOT > /dev/null
> 8.600u 0.516s 10:14.14 1.4%	0+0k 0+0io 0pf+0w

> ubuntu1:/home/nick/trunk/src/peval> time ./peval -h whois.ripe.net -protocol ripe AS-NEOT > /dev/null
> 2.900u 0.304s 12:21.72 0.4%	0+0k 0+0io 0pf+0w

Due to its use of pipelining it can return IRRD query results ridiculously
quickly, and it's pretty clear that irrtoolset is being completely trashed
by comparison because it doesn't support pipelining.  The difference
between the RIPE and RADB whois server is related to the fact that the RADB
server supports recursive as-set expansion.

In fact, a more interesting question is not why bgpq3 is 175 times faster
than irrtoolset, but rather why it takes as long as 4.26 seconds to
complete an inverse query for the as-set in question.  If I did ran a SQL
query on a single term with indexed lookups and which returned 50k results,
I'd wonder why it was so slow. In this case, it turns out that the answer
mostly relates to the amount of data returned (~750k), and the rtt to the
whois server from my query box (~150ms), rather than as a result of the
query latency.

Anyway, it should be clear that for large query sets, the RIPE server is
extremely slow because of its lack of command pipelining and lack of
support for recursive expansion of certain types of objects.

This directly affects INEX because we run a full rebuild of our route
server configuration three times daily.  Our route server as-set
(AS-SET-INEX-RS) contains about 5400 ASNs and the consequent serialisation
delays are currently causing our rebuild scripts to take slightly more than
30 minutes to complete.  This varies depending on client policies: earlier
this year, rebuilds were taking about 2 hours.

In fact for performance reasons, we need to query whois.radb.net instead of
RIPE for the larger AS-SETs.  This is because of the recursive as-set
performance issue which affects queries to whois.ripe.net

In order to solve this problem, what we'd really love to see is:

1. support for command pipelining
2. support for recursive expansion of as-set to a list of ASNs.
3. support for recursive inverse key expansion of an as-set to a list of
prefixes

These would mean a massive performance gain for INEX's use case (and a
bunch of other IXPs around europe) and #3 would mean that the RIPE whois
server could leapfrog IRRD in terms of performance.

Would it be possible for the DB people to look into this?

Nick