[atlas] Incident report for 2019-10-02 (was: Error: No suitable probes and delayed results)

On 2019-10-03 08:16, Moritz Muller wrote:
> Hi,
> 
> In our experiment we’re trying to assign certain probes to a ping measurement but until now we always get the error message "NO SUITABLE PROBES”.
> According to the documentation, this is a sign that a probe might not have enough resources.
> However, when I check on the measurement a few hours later I do see results, but its state is still "NO SUITABLE PROBES”.
> See https://atlas.ripe.net/measurements/23016956/#!probes
> 
> Is that a common problem when selecting certain probes for measurements?
> 
> Moritz
> 

Hello,

Yesterday afternoon we had an operational problem within RIPE Atlas that
had consequences visible to users. I strongly suspect the above is a
side-effect of this.

Due to a combination of two configuration errors and a spike in requests
from users, the core infrastructure received an unreasonably high amount
of measurement requests from an internal process related to IPmap. The
measurements scheduler and participant management subsystems struggled
to keep up with this load and eventually things started piling up.

The issue started at approximately 12 UTC. We identified the root cause
about an hour later. Processes started to normalise late in the
afternoon, and processing the backlog finished sometime after midnight.

We're working on a post-mortem, and reviewing the code and configuration
in order to prevent this error from happening again.

Apologies for the inconvenience,
Robert

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: </ripe/mail/archives/ripe-atlas/attachments/20191003/b9b1778f/attachment.sig>