[db-wg] [ncc-services-wg] Urgent release of RIPE Database software

Dear Wilfried,

In retrospect, we may have overestimated the severity of the situation. Based on Tim's message to the list, we made the initial assessment that this was impacting his operations but also that it was impacting anyone else wanting to use the new REST API. Without the proper documentation in place it would be very difficult to migrate to the new API.

At the point of the first announcement, we did not have a workaround available so we considered it necessary to update the Production Database as soon as possible. The documentation was part of the binary release so any changes to it required a new release in production.

We agreed with the approach suggested on the list to put the software in the TEST Database before going to production. The test period was set to 24 hours, as a result of further feedback from the list. This proved beneficial, as we immediately found new bugs. At that point, we decided to halt the release process and conduct further investigation.

The severity and urgency of this issue is now lower than initially estimated, partly because we now have a workaround in place: use the TEST Database documentation.

This was the first time we applied the new release procedure for this type of incident, and it has provided us with valuable feedback as to how to proceed in the future, should a similar situation arise.

I hope this clarifies the steps that were taken - and why.

Kind regards,

Johan Åhlén
Assistant Manager Database
RIPE NCC

On 21 Aug 2013, at 14:14, Wilfried Woeber <Woeber at CC.UniVie.ac.at> wrote:

> Hi Johan!
> 
> first of all, thanks for the openness and announcements on the list(s).
> 
> As my question is post-factum, I am not including the lists(s), but I'd still
> like to collect some feedback from those "involved":
> 
> Do I understand the stated facts correctly: that the software (the API) was
> working as intended, but the (auto-)generated documentation was in error?
> 
> If this is the case, than I (personally, not wearing my WG co-chair hat)
> would strongly lean towards Rüdiger's position. In that case I would rather
> issue an alert regarding the documentation error and a correction.
> 
> The approach as proposed to still go through the TEST cycle was the correct
> one, imho.
> 
> Btw, I do not easily agree to label the problem as "service impacting, if
> the software is correct, but the documentation is in error. If it were the
> other was 'round, then yes, the label would be correct in my opinion.
> 
> Johan Åhlén wrote:
> 
>> Dear Tim,
>> 
>> Instead of quickly fixing this issue and creating a new release today
>> we've decided to halt the deployment of any new release for now. 
>> 
>> This is very embarrassing and we're very sorry for this. We obviously
>> put too much trust in the framework that generates the documentation and
>> we don't have the proper mechanisms in place to validate the correctness
>> of the documentation generated. We will give this our fullest attention
>> the coming days and expect to have a working solution sometime next week.
>> 
>> In the meantime I hope the existing documentation in TEST Database
>> contains the information you need to proceed with your project. If you
>> have any questions about how the API works then please contact us
>> directly and we'll gladly help you out.
>> 
>> Kind regards,
>> 
>> Johan Åhlén
>> Assistant Manager Database
>> RIPE NCC
> 
> Regards,
> Wilfried.
>