[db-wg] Severe performance problems with RIPE whois server

On 27/12/2012 15:44, Joao Damas wrote:
> - client side resources are more plentiful than server-side (e.g. CPU)

this will always be true.

> - the client can apply smart filters rather than do full expansion

> - you can keep the connection open using the -k flag and issue commands
> in quick succession. A big part of the slowness was due to server
> process (or thread in the new one) initiation and the -l flag allowed
> persistent connections.

this sounds like a problem associated with fork() efficiency before the
widespread use of vfork() and copy-on-write memory handling techniques.

> - you can also narrow down the record types that are returned by the
> server to those who are relevant to policy so that you don't have to
> parse all the irrelevant info when using RPSL policy (like all the *-c
> info)

this is effectively available using the -K option.

> The server was designed to make combined use of these last two features
> very efficient.
> 
> Of course hardware limitations on servers were quite different than they
> are now (though a DoS is still very easy if you push more work onto the
> server).

I suspect things have moved on with more recent versions of the front /
back-end split in the whois server mechanism.  The back-end mechanism uses
SQL and it should be easy to implement recursive as-set expansion using
either direct sql joins or else a set of efficient

> I would however encourage people to try out the above before putting
> more crud into the server-side

irrtoolset / bgpq3 / rpsltool and everything else already use these
techniques.  The problem looks like it's related almost entirely to
serialisation delays, not client side delays or raw server performance delays.

Looking at the 10 minute / 4 second figures, the only way to get around
this with the ripe whois server is to open up multiple connections to the
whois server and issue parallel whois queries - this technique will reduce
the overall latency by a factor equal to the number of parallel open
sessions.  I don't think anyone in the NCC is going to thank me for opening
up 175 parallel connections in order to get my end-user performance similar
to merit irrd levels :-)

Nick