[members-discuss] RPKI ROAs for sponsored resources got deleted?

Dear colleagues, 

Yesterday at 18:17 (UTC+2) we accidentally deleted 4,100 RPKI Route Origin Authorisations (ROAs). These ROAs were related both to members' and sponsored resources. This happened while we were performing maintenance on our internal software. 
We are currently investigating and we know which ROAs were deleted, which ones are (by now) recreated by users and which ROAs are still missing. We’re currently working on bringing back the missing ROAs and we will inform impacted users when we re-publish their ROAs. 
Apologies for the inconvenience. 

Kind regards

Nathalie Trenaman
Routing Security Programme Manager
RIPE NCC

> Op 2 apr. 2020, om 09:35 heeft Thomas Woidt <thomas at rootsecurity.be> het volgende geschreven:
> 
> We had the same thing happening on our sponsored objects, our own ranges (PA and PI) didnt break.
> I recreated the ROA's this morning and they are valid again.
> 
> Looks like some upgrade/change at RIPE going wrong?
> 
> 
> -- 
> Thomas Woidt
> Owner
> https://localitel.as
> (BelgiumIX.net/Fastic.be)
> 
> 
> On 02/04/2020 09:14, Johan Hedberg wrote:
>> Same thing has happened to me too, thanks for bringing it up, I wouldn’t have noticed otherwise.
>> There’s been no routing changes on my end, and those ROA’s were perfectly valid.
>> 2020-04-01 16:17:03systemUpdated ROA configuration. Additions: none. Deletions: [asn=AS8949, prefix=2001:67c:560::/48, maximumLength=48], [asn=AS8949, prefix=2001:67c:7bc::/48, maximumLength=48].
>> 2020-04-01 16:17:03systemUpdated suppressed routes for ROA alerts. Additions: none. Deletions: [asn=AS8949, prefix=2001:67c:560::/48], [asn=AS8949, prefix=2001:67c:7bc::/48].
>> —
>> Johan Hedberg
>> Senior Systems Engineer
>> Mobile: +46 708 474186
>> www.citynetwork.eu <http://www.citynetwork.eu> | www.citycloud.com <http://www.citycloud.com>
>> INNOVATION THROUGH OPEN IT INFRASTRUCTURE
>> ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED
>>> On 2 Apr 2020, at 08:02, Dominic Schallert <ds at schallert.com <mailto:ds at schallert.com>> wrote:
>>> 
>>> Hello Community,
>>> 
>>> what happened to RPKI ROAs for Sponsored Resources yesterday? All of a sudden, all ROAs for Sponsored Resources got deleted...
>>> 
>>> 2020-04-01 16:16:41 system Updated ROA configuration. Additions: none. Deletions: [asn=ASxxx, prefix=x.x.x.x/22, maximumLength=22].
>>> 2020-04-01 16:16:41 system Updated suppressed routes for ROA alerts. Additions: none. Deletions: [asn=ASxxx, prefix=x.x.x.x/22].
>>> 
>>> Not only our customers affected but also others.
>>> 
>>> Thanks
>>> _______________________________________________
>>> members-discuss mailing list
>>> members-discuss at ripe.net <mailto:members-discuss at ripe.net>
>>> https://mailman.ripe.net/
>>> Unsubscribe: https://lists.ripe.net/mailman/options/members-discuss/jh%40citynetwork.se
>> _______________________________________________
>> members-discuss mailing list
>> members-discuss at ripe.net
>> https://mailman.ripe.net/
>> Unsubscribe: https://lists.ripe.net/mailman/options/members-discuss/thomas%40rootsecurity.be
> 
> _______________________________________________
> members-discuss mailing list
> members-discuss at ripe.net
> https://mailman.ripe.net/
> Unsubscribe: https://lists.ripe.net/mailman/options/members-discuss/nathalie%40ripe.net