[routing-wg] Improving operations at RIPE NCC TA (Was: Delay in publishing RPKI objects)

Dear RIPE NCC,

On Wed, Feb 17, 2021 at 11:28:32AM +0100, Nathalie Trenaman wrote:
> > The multitude of RPKI service impacting events as a result from
> > maloperation of the RIPE NCC trust anchor are starting to give me
> > cause for concern.
> 
> I’m sorry to hear this. Transparency is key for us, this means that we
> report any event. In this case, we were not compliant with our CPS and
> this non-publishing period had operational impact.

>From the previous email there might be a misunderstanding about what
rpki-client and Routinator do. Both utilities help Relying Parties
validate X.509 signed CMS objects and transform the validated content
into authorizations and attestations. Neither utility is a SLA or CPS
compliance monitor. RIPE NCC - as CA operator - needs different tools.

Neither utility has been designed to interpret the Certification
Practise Policy (written in a natural language) and subsequently
programmatically transform the described 'Service Level' into metrics
suitable for monitoring.

A relying party can never tell the difference between a publication
pipeline being empty because CAs didn't issue new objects, or a
publication pipeline being empty because of a malfunction in one of RIPE
NCC's RPKI subsystems.

More examples of 'out of scope' functionality for Relying Party
software: validators don't monitor whether lirportal.ripe.net is
functional, whether RIPE NCC's BPKI API endpoints are operational, or
whether LIRs paid their invoices, the list is quite long. The validators
by themselves are the wrong tool for RPKI CPS/SLA monitoring.

You state "transparency is key for us", but I fear ad-hoc low-quality
a-posteriori reports are not the appropriate mechanism to impress and
reassure this community regarding 'transparency'.

I have some tangible suggestions to RIPE NCC that will improve the
reliability of the Trust Anchor and potentially help rebuild trust:

A need for Certificate Transparency
-----------------------------------

RIPE NCC should set up a Certificate Transparency project which publicly
shows which certificates (fingerprints) were issued when, and store such
information in immutable logs, accessible to all.

How can anyone trust a Trust Anchor which does not offer transparency
about its issuance process?

Lack of transparency to signer software
---------------------------------------

The RIPE NCC WHOIS database software is open source, as is most of the
software for RIPE Atlas, K-ROOT, and other efforts RIPE NCC has
undertaken over the years.

Why has the signer source code still not open sourced? Why can't members
review the code related to scheduled changes? Why is an organisation
proclaiming 'transparency' being opaque about how the RPKI certificates
are issued?

Lack of Public status dashboard
-------------------------------

RIPE NCC should set up a website like https://rpki-status.ripe.net/
which shows dashboards with graphs and traffic lights related to each
(best effort) commitment listed in the CPS. RIPE NCC should continuously
publish & revoke & delete objects and verify whether those activities
are visible externally, and then automatically report whether any
potential delays observed are within the Service Levels outlined in the
CPS.

Metrics that come to mind:

* delta between last certificate issuance & successful publication
* Object count in the repository, repo size (and graphs)
* Time-To-Keyroll (and graphs on duration & frequency)
* Resource utilisation of various RPKI subsystems
* aggregate bandwidth consumption for RPKI endpoints (including rrdp, API, rsync)
* Graphs & logs of overlap between INRs listed on EE certificates under
  the RIPE TA and other commonly used TAs, matched against known
  transfers. This will help detect compromises as well as understand
  whether transfers are successful or not.
* Unique client IP count for RSYNC & RRDP for last hour/day/week
* Number of CS/hostmaster tickets mentioning RPKI

There is are plenty of aspects to monitor, perhaps some notes should be
copied from how the DNS root is monitored.

Lack of operational experience with BGP ROV at RIPE NCC
-------------------------------------------------------

I believe the number of potential future incidents related to the RIPE
NCC Trust Anchor can be prevented (or remediation time reduced) if RIPE
NCC themselves apply RPKI based BGP Origin Validation 'invalid ==
reject' policies on the AS 3333 EBGP border routers. RIPE NCC OPS
themselves having a dependency on the RPKI services will increase
organization-wide exposure to the (lack of) well-being of the Trust
Anchor services, and given the short communication channels between the
OPS team and the RPKI team my expectation is that we'll see problems
being solved faster and perhaps even problems being prevented.

An analogy: RIPE NCC is a kitchenchef refusing to eat their own food.
How can we trust RIPE NCC to operate RPKI services, when RIPE NCC
themselves refuses to apply the cryptographic products to their BGP
routing decisions? "RPKI for thee but not for me?"

Surely RIPE NCC staff has not disabled DNSSEC sig checking on their
resolvers, or disabled WebPKI TLS validation in their browsers? I'm not
joking, it makes zero sense to participate in a PKI and at the same time
not participate in the same way everyone outside RIPE NCC depend on the
service.

I am very aware of potential for circular dependencies between BGP and
RPKI, and I know exactly how catch-22s can be avoided. Unfortunately it
appears my feedback is ignored, problem reports remain unresolved.

Reporting issues has become a thankless effort, useless because no
remediation actions are taken, and obviously RIPE NCC staff are growing
tired of hearing about problems (but if one wishes to stop hearing about
problems... perhaps solve the issues, instead of a 'head in the sand'
approach?!)

Conclusion & Call to action
---------------------------

There is a fair chunk of work ahead for RIPE NCC, but RIPE NCC has a
multi-million budget and talented dedicated staff to achieve the above.
None of the above is impossible or unreasonable to ask from Trust
Anchors.

I recognize how the above information reflects negatively on the current
state of the RIPE NCC Trust Anchor. But the reality of the situation is
that we see an outage every few weeks, there is an apparent lack of
architectural oversight how to improve. I really hope this is a
temporarily state of being, on which we can look back a year from now as
"haha remember those RPKI teething pains?". I wish for RIPE NCC to
be successful in operating the Trust Anchor.

RIPE NCC would to well to allow themselves to be vulnerable to criticism
by exposing service level metrics and efforts like production of public
merkle tree logs - reflecting the certificate issuance process. RIPE NCC
should allow itself to be held accountable - which can only happen if
there is visibility into where friction exists.

Does RIPE NCC understand the precariousness of the current situation and
the negative impact on the long term viability of the RPKI if course is
not corrected?

This email outlines deliverables, will RIPE NCC commit to those? What
timelines can the community expect? What kind of help is needed to
achieve this?

Kind regards,

Job