[atlas] Friday's events on RIPE Atlas

On 2015-09-23 12:29, Colin Johnston wrote:
> maybe compress the data info on the probes before it is sent and save to backend storage with the data already compress’d ?
> 
> just a thought
> 
> Colin

It may be interesting to know, although not at all surprising: compressing
individual results achieves almost nothing (and has the drawback of having
to unpack/repack while the result moves in the pipeline). Compressing small
chunks, say dozens of related results (e.g. from the same measurement)
achieves 40-50% saving (a lot depends on what kind of result it is), while
compressing large blocks (multi-megabyte chunks) can achieve 90-95% in most
cases.

It makes sense to compress data once it's at rest, especially if it's not
often accessed. Therefore our processing pipeline is constructed such that
it applies compression once a large enough batch has been consumed. As Romeo
explained, this part was stretched a lot on Friday.

Cheers,
Robert