Hardware problems at the RIPE NCC

Ticket Number: -                     Ticket Status:   open
Ticket Type:   unscheduled           Ticket Source:   RIPE NCC
Ticket Scope:  RIPE NCC              Site/Line:       RIPE NCC
Ticket Owner:  ops at ripe.net          Problem Fixer:   Geert Jan de Groot
                                                                
Ticket Opened: 950217 21:47 UTC      Problem Started: 950217 21:00 UTC
       Closed:                                 Ended:                         

Problem Description:
		Due to the failure of 2 (two) harddisks at the RIPE NCC
		on the same day, it seems we may have lost mail sent to
		the various role accounts on Friday.
		Also, whois.ripe.net has been offline on Friday for an
		hour (sorry).

Affected:
		RIPE NCC role accounts: mail which arrived between
		950217 00:00 UTC and 950217 22:00 UTC may be lost
		(we still have the sendmail logs)
		RIPE auto-dbm updates were deliberately delayed
		until machines are in full working order, but will
		be processed shortly. If you have not received an ack
		by Monday morning, please re-submit the update.
		We have been unable to send this ticker earlier, because
		the mail aliases were also on one of the disks involved..

Actions:
		Install replacement harddisk on whois.ripe.net on Monday
		(downtime is expected to be < 5 minutes).
		Restore disk contents of 950217 0:00 (last dump date)
		for NCC role accounts.
		Investigate sendmail logs for possible lost mail, and
		send individual messages, if possible.
		Restore role account transaction records for 950217 
		as far as possible.
		LOCAL-REGISTRIES: if possible, re-send your mail sent
		on Friday with RESEND in the subject line.
		Install some garlic in the engine room ;-)

		While we make dayly backups (of course!), the crash
		happened 21 hours after the last backup, and those 21 hours
		includes a full working day. I can't think of a better
		way to guard against these exceptional circumstances.

Time to fix:
		RIPE NCC operations should be back to normal by the time
		you read this, but us staff has to determine what mail
		has been lost. As a result, we might not respond immediately
		to messages on Monday.
		I expect to be able to do the harddisk swaps with minimal
		disruption.