[atlas] Atlas probe offline

On 2016-01-06 15:58, Steve Hill wrote:
> On 06/01/16 13:17, Estelmann, Christian wrote:
> 
>> your probe is tagged as "readonly flash drive".
>>
>> Disconnect the probe, connect the usb stick to your computer and test it
>> (e.g. delete all partitions, create own partitions and write some data
>> to it). If data can not be written to the usb stick a) disable the write
>> lock, b) replace it by a new one one (4 GB capacity at minimum).
>> Then reconnect the probe (power and ethernet, but without the usb
>> stick), wait 5 minutes and then plug in the usb stick.
> 
> Thanks - looks like that was it.  The original drive has gone read-only,
> swapping it for another USB stick has fixed the issue.

A little bit of background information that could help:

The probes use USB storage to buffer results. This comes handy when the
probe is off-line, and especially handy if the error is outside the host's
network.

The unpredictable nature of power losses and other unfriendly events
regarding filesystems of course affect these sticks too. This is not
unexpected, so the probes do file system checks and repairs as needed to
overcome fs corruption resulting from this (and, as some people know this by
experience, they can even format and use a new USB stick if it's inserted
while the probe is powered up).

We learned the hard way that the particular type of USB stick (Sandisk nano)
has a particular "feature", namely that it switches to permanent read-only
mode if it detects possible corruption. This is most likely there to prevent
further escalation of the problem, while allowing recovery of the remaining
data from the stick. This is probably ok for generic use, as they are cheap
and replaceable. However, in the RIPE Atlas context, this is bad, as there's
nothing we can do to fix -- besides reporting to the host. We're unsure what
exactly triggers the behaviour.

Because of this behaviour we started using a different brand for storage a
while ago. We see no need to pro-actively swap out the sticks (and it's also
very difficult to do in practice) as the problem only occurs with a small
amount of probes.

Cheers,
Robert