[routing-wg] RPKI Outage on 16.02.2023

Dear colleagues,

Yesterday, at 11:06 UTC we attempted to migrate the RPKI core on the production environment from JDK8 to JDK11 via an automated deployment. Unfortunately, the update to the RPKI core was not successful, which resulted in RPKI becoming partially unavailable. 

The RPKI dashboard and API were no longer accessible which meant that no changes could be made to the ROA configurations or alerts. The provisioning (up/down) protocol used by non-hosted Certificate Authorities was also unavailable during this time.

The RRDP and rsync public repositories were not impacted by this incident.

At approximately 12:30 UTC, we reverted our changes and the RPKI core was back up and running. The RPKI team has since then fixed the upgrade issue and the migration has been completed successfully.

The upgrade issue was caused by the production environment still using some file-based HSM keys. These file-based HSM keys are no longer used in the staging environment, so the issue was not detected before we attempted the production upgrade to JDK11.

Kind regards,
Stella Vouteva
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ripe/mail/archives/routing-wg/attachments/20230217/c36e584e/attachment.html>