[atlas] When to consider a measurement finished

Hi Wouter,

The status of one-offs is not related to results collected. The stopped 
status is set by the timer after max 15 minutes. Since the measurement 
scheduling is based on the best effort approach there cannot be 100% 
guarantee that all the probes respond with the results so it becomes 
quite difficult to set the stopped status when all probes finish 
measuring. Moreover an additional lag can be introduced due to the 
cluster processing. We saw situations when a measurement had the stopped 
status but the results were coming.

So the workflow would be to start a measurement, allow it to be 
scheduled and first results collected, then after 5-20 seconds you can 
start polling it with the same or larger interval. The polling interval 
should be the longer the more probes you use in your measurement.

Additionally you can estimate the data processing delay using the 
https://atlas.ripe.net/api/v2/system/data-delay/ API call that shows the 
lag in seconds and the latest received measurement timestamp.

If you want it to be truly synchronous you can use the streaming API. In 
this case the results will be pushed to the client as soon as they are 
coming from the probes.

WBR
/vty



On 6/18/19 8:08 PM, Wouter de Vries wrote:
> Hi all,
>
> I am currently trying to do some measurements according to the following
> steps:
>
> 1. create a one-off measurement
> 2. wait for it to finish, by periodically polling the msmid and checking
> the status.
> 3. fetch the results
>
> However, the system takes a pretty long time to set the status to
> finished, even when all results are already in (maybe 10 minutes?).
>
> What heuristics/method do you recommend to determine whether to consider
> a measurement as finished?
>
> One option I consider is to periodically fetch all results for a given
> msmid and checking if the number of results matches the number of probes
> assigned to the measurement. However, I suspect that causes a much
> higher load on the infrastructure.
>
> Best regards,
>
> Wouter de Vries
>
>
>