[atlas] Rate limiting on the APIs

Update: we applied this change a few minutes ago. We'll keep observing 
the system for ill effects.

Regards,
Robert


On 2023-01-30 14:57, Robert Kisteleki wrote:
> Dear all,
> 
> We have seen in the past that even with the best intentions, every now 
> and then some users are using excessive amounts of requests to the RIPE 
> Atlas APIs, in some cases resulting in degraded performance to all 
> users. An example for the curious: some AWS account asking the probe API 
> for live queries once a day at 8am like clockwork; over a million 
> queries for probe statuses (note: we don't have a million probes - every 
> query is repeated hundreds or thousands of times) [1]
> 
> We will apply rate limiting to protect from (some of) these. In particular:
> 
> * The API in general will impose a limit of no more than 300 
> queries/second (if done over multiple connections)
> 
> * The measurement API in particular will impose a limit of 150 
> queries/second (if done over multiple connections)
> 
> * When reaching the limit, an HTTP response code of 429 (Too Many 
> Requests; https://www.rfc-editor.org/rfc/rfc6585#section-4) will be given.
> 
> According to our logs these limits will in general not be visible to 
> users. Users going over these limits will see the 429 response code; 
> client libraries should be able to handle this. We will observe the 
> effect of these measures to determine if we need to tighten further - or 
> perhaps relax them.
> 
> We plan to apply these changes sometime next week; I'll let you know 
> once they are in place.
> 
> Regards,
> Robert
> 
> 
> [1] a much better way of doing this is to use the probe archive 
> (https://ftp.ripe.net/ripe/atlas/probes/archive/); this is one-time 
> fetch of 2MB or so and probe statuses are pretty stable in general
> 
>