[ncc-services-wg] DNS outage

[Apologies for duplicate emails]

Dear colleagues,

This morning, the RIPE NCC briefly experienced a problem on the
ripe.net DNS servers. Operation of K-root and other public DNS
services was not impacted at any time during this outage.

At 11:45 UTC this morning, an error in the deployment script used
to update the ripe.net zone caused a partial zone file to be activated.
This caused the ripe.net DNS servers to return NX-domain responses
for some valid hosts and/or subdomains in the ripe.net zone, which
effectively rendered some of the RIPE NCC services unavailable during
this period.

DNS resolution for ripe.net was fully restored at 12:15 UTC. Caching
of the erroneous responses may have caused reachability problems until
13:15 UTC.

The error in the script has been identified and is only triggered in
a specific corner case. This error will be corrected today.

We apologise for any inconvenience caused.

Incident start:  05-Mar-2013 11:45:01 UTC
Incident end:  05-Mar-2013 12:14:58 UTC

Regards,

Romeo Zwart
GII Services Manager
RIPE NCC

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ripe/mail/archives/ncc-services-wg/attachments/20130305/c1b47bfa/attachment.html>