[members-discuss] Registry Services Ticket Response Time

Dear Elvis,

Your email touches on multiple issues. I‘d like to address some of them 
now, with the understanding that we will share more detailed information 
soon. For now, I can provide a general overview on what the situation 
with wait times is, and how we plan to tackle this issue in the coming 
weeks.

First, a quick historical comparison for context. Back in 2013, the RIPE 
NCC had around 9,000 LIRs, and we processed a mere 150 policy transfers 
in the whole year, i.e. approximately 12-13 transfer requests a month. 
We currently have about 23,700 LIRs, which is 2.5x the LIRs in 2013. 
Last month, we processed over 400 policy transfers. This is 30x the 
number of transfer requests we processed when you were part of the 
Registry Services team. And remember that there are all of the other 
requests on top of this.

In 2020, we averaged around 1,600 resources-related requests per month. 
However, from November to December there was a threefold increase 
compared to the rest of the year. It is hard to avoid an increased wait 
time when this happens, and that’s why we prefer to be upfront when we 
expect delays, so members can plan around them.

While these statistics provide a glimpse into how things have changed 
over the past few years, I would like to reiterate that it is not just 
the number of tickets but also the complexity of requests that has 
increased. We report on this regularly at RIPE Meetings.

Despite these challenges, Registry Services strives to handle incoming 
requests efficiently. In order to improve our ticket handling times, we 
have:
- Hired new staff, who are still being trained, but will help address 
the workload once they are up to speed.
- Taken on temporary staff to enter registration details of resource 
holders into our system which will allow us to automate certain due 
diligence checks.
We will publish a RIPE Labs article about these changes in the coming weeks.

Some additional clarifications regarding internal targets:
- Every ticket that doesn't get a response by 18:00 (UTC +2) the 
following working day is counted as a ‘miss’ against our internal targets.
- Our notification emails do not count against these targets (we don’t 
see much point in gaming our internal KPIs).
- A gentle reminder that Saturdays and Sundays are not working days, so 
requests received on a Friday are likely to be addressed the next 
business day, i.e. the Monday after the weekend.

We can appreciate that it is frustrating to wait on tickets. However, 
while we’re still training additional staff and automating systems, we 
need to rely on your patience and ask that you allow us time to ensure 
requests are not just handled quickly, but also accurately.

Since we know from experience that the number of requests increases 
dramatically in the last quarter of the year, I would like to repeat our 
request for members to kindly not wait until the last minute to send in 
a request, and to expect longer handling times in the last quarter of 2021.

Once I have a more detailed overview of our ticket history, I’ll share 
it on this list, along with how we plan to further improve our services.

I am aware that I have not addressed all the concerns raised in your 
email, Elvis, but I plan to continue this conversation in the coming days.

Kind regards,
Marco Schmidt
Registry Services Assistant Manager
RIPE NCC

On 31/08/2021 06:06, Elvis Daniel Velea wrote:
> Dear Marco,
>
> thank you for the update below. I'd like to ask you to provide a few 
> more details and to detail your plans to fix the SLA violation by the 
> Registration Services department, please see my questions inline.
>
> On 8/4/21 3:43 AM, Marco Schmidt wrote:
>> Dear colleagues,
>>
>> We would like to give you an update on the ticket wait times in 
>> Registry Services, after discussions at RIPE 82 [1] and on the 
>> mailing list [2].
>>
>> We have been addressing the issues causing delays. As the number of 
>> incoming requests continue to exceed previous years, we have taken on 
>> additional staff to help with the workload. We are also automating 
>> some of our due diligence processes which are currently time consuming.
>
> You say that you receive more requests than in previous years, can you 
> please provide some statistics for the past 10 years showing the 
> number of employees vs number of requests/replies received/sent daily?
> - please try, if possible, to show us how many of these tickets were 
> related to transfers, how many to billing or closures, etc...
>
> Can you provide some stats showing how many replies are necessary for 
> a ticket to be resolved? Also, let me know what is being done so that 
> requests get evaluated and resolved in one reply.
>
> Also, please let us know how exactly have you addressed the constant 
> SLA violations, what part of which process has been automated and 
> whether you believe that with the additional staff you will resolve 
> the issues swiftly.
>
> As you probably remember, I used to work in the RS department between 
> 2007 and 2013. Back then, a day when the SLA was not respected was 
> almost a tragedy... it seems that now it's the norm and not the 
> exception any longer. What happened that your department ended in this 
> situation?
>
>>
>> You should start to see improvements in our response times towards 
>> the end of this year once our new colleagues are trained and our 
>> automation is ready. In the meantime, you may experience slower 
>> response times. In other words, we will be slower before we can get 
>> up to speed properly, but we think this will be worth it over the 
>> long term.
>
> Things are slower and slower. For example, last week I had a ticket 
> receive 1 reply after ~24 hours saying that because of time 
> constraints, the ticket will be evaluated later. Then Friday, Saturday 
> and Sunday passed before I got a reply on Monday.
>
> None of the tickets I or my customers opened last month (August) have 
> been replied to within the SLA. NONE! We're talking about more than 10 
> tickets in a month, none receiving a reply within the SLA.
>
> You are saying that waiting will be worth it over the long term while 
> right now all tickets are being delayed from a day to another and 
> sometimes a ticket that should take 10 minutes to evaluate is delayed 
> for 4 days or more... This is not acceptable and that is why I'd like 
> to ask you for more details to prove to us that the wait will be worth 
> it. As things are getting worse and not better, I hope you understand 
> why I can't take your word for it :)
>
>
> When you generate those stats, please let us know if the replies 
> saying 'Due to a very high workload, we were unable to process your 
> request at this time and will process it as soon as possible.' are 
> still counted as a violation of the SLA or not.
>
> Also, when you generate the stats I ask for, please also tell us how 
> many working days in a month (for the past year or two) the 
> Registration Services dept. has violated the SLA.
>
>
> I've been trying to help the RS department evaluate all the requests 
> received from V4Escrow and it's customers quicker. The help I offered 
> was refused while the SLA violations got worse and worse.. maybe it is 
> time you accept the help when it is offered to you.
>
> I've had several conversation with RS management and higher up, all 
> the way up to MD. I talked to several teams to explain the way I 
> wanted to help RS and our customers get all the tickets resolved 
> swiftly (at least the ones that are for transfers of resources). I 
> proposed the introduction of an API for tickets/requests so we can try 
> to automate a large part of the transfer ticket, this automation could 
> help RS evaluate requests faster, would help customers provide all the 
> required information in one go; everyone would be happy. I was told my 
> request will be looked at and maybe added to plan no earlier than 
> Q1-2022. Please tell me if that is still the plan or whether, in the 
> light of these constant SLA violation, you could enable the API faster 
> so we, brokers, can help you receive complete and easy to evaluate 
> transfer requests.
>
>> We also know from experience that we receive far more requests 
>> towards the end of the year which results in a bottleneck. We will 
>> shift resources to counter this, but we would also appreciate it if 
>> you could get some of these requests in earlier.
>
> Is this a warning that requests will take weeks to process towards the 
> end of the year? They already take days...
>
> Please clarify the bottlenecks you have and let's see if there's any 
> way we can help you with them. I am sure that automating a lot of 
> steps in your overly complicated processes will help but I am looking 
> forward to receiving more details from you about this.
>
>
> Elvis
>
> PS: Note that I wanted to also send an e-mail about the RIPE NCC's 
> ticketing system (zendesk) to the members-discuss mailing list and to 
> the board members. I will wait for an answer to the e-mail I sent to 
> HPH one week ago (which he promised he'll forward to the relevant team 
> for a reply).
>
>>
>> Kind regards,
>> Marco Schmidt
>> Registry Services Assistant Manager
>> RIPE NCC
>>
>> [1] RIPE NCC Operational Update (slides 4-10):
>> https://ripe82.ripe.net/archives/video/584/
>>
>> [2] Discussion on Members Discuss mailing list:
>> https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html 
>>
>>
>>
>> _______________________________________________
>> members-discuss mailing list
>> members-discuss at ripe.net
>> https://mailman.ripe.net/
>> Unsubscribe: 
>> https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net
>