[members-discuss] Registry Services Ticket Response Time

Dear Elvis,

Thank you for your feedback. You are correct in pointing out that it 
isn’t straightforward to compare the work of the Registry Services in 
2013 with the work we do in 2021. That’s why I’m preparing a deeper 
analysis that looks at recent changes and developments, as well as our 
approach to improving our services. A lot has changed since you left 
eight years ago, and you might be surprised to learn about the added 
complexity for even seemingly easy requests.

To share some more numbers on this situation, in the first eight months 
of 2021, the amount of resource-related requests has increased by almost 
25% compared to the same period last year. Looking at just August of 
this year, 62% more requests were created than in August 2020. At the 
same time, our due diligence activities have increased due to sanction 
checks. This is the main reason it has been challenging to provide the 
fast response time that our members are used to.

We are confident that with the steps taken, as outlined in my previous 
email, we will soon be handling this increased workload more quickly.

As soon as my overview is ready, I will share it on this list.

Kind regards,

Marco Schmidt
Registry Services Assistant Manager
RIPE NCC

On 03/09/2021 12:39, Elvis Daniel Velea wrote:
> Hi Marco,
>
> thank you for your reply. Please see further comments below. Maybe 
> some will help you with your next reply.
>
> On Fri, Sep 3, 2021 at 01:57 Marco Schmidt <mschmidt at ripe.net 
> <mailto:mschmidt at ripe.net>> wrote:
>
>     Dear Elvis,
>
>     Your email touches on multiple issues. I‘d like to address some of
>     them
>     now, with the understanding that we will share more detailed
>     information
>     soon. For now, I can provide a general overview on what the situation
>     with wait times is, and how we plan to tackle this issue in the
>     coming
>     weeks.
>
>
> ok
>
>
>
>     First, a quick historical comparison for context. Back in 2013,
>     the RIPE
>     NCC had around 9,000 LIRs, and we processed a mere 150 policy
>     transfers
>     in the whole year, i.e. approximately 12-13 transfer requests a
>     month. 
>
>
> Please do not compare apples to oranges. In 2013 the RIPE NCC was 
> processing hundreds of requests for initial and additional 
> allocations, requests that were way more complex than some of the 
> current resource requests (which no longer require any need based 
> justification or evaluation) or transfer requests that only require 
> the RIPE NCC to verify (i) if an organization is still active in their 
> country and (ii) the document you received is signed by the people 
> with PoA.
>
>
>     We currently have about 23,700 LIRs, which is 2.5x the LIRs in 2013.
>     Last month, we processed over 400 policy transfers. This is 30x the
>     number of transfer requests we processed when you were part of the
>     Registry Services team. And remember that there are all of the other
>     requests on top of this.
>
>
> That is why I was asking you to provide total number of tickets 
> handled, back then, now and every year in between. A ticket is a 
> ticket, some are simple, some are more complicated. Same was then as 
> it is now…
>
> You are right, you did not process hundreds of transfer requests in 
> 2013, but there were many other types of requests that were way more 
> complex than some of current transfer requests. So saying that you 
> process 30x transfer requests is a bit misleading as you are not 
> showing the complete picture.
>
> The NCC RS dept. even succeeded to respect the SLA when it was 
> evaluating requests in pairs, towards the end of the runout. At that 
> time you were receiving maaany requests, more than usually. If the 
> department was able to cope with that situation, why can’t it now? I 
> still have not seen the answer to this question.
>
>
>
>     In 2020, we averaged around 1,600 resources-related requests per
>     month.
>     However, from November to December there was a threefold increase
>     compared to the rest of the year. It is hard to avoid an increased
>     wait
>     time when this happens, and that’s why we prefer to be upfront
>     when we
>     expect delays, so members can plan around them.
>
>
> I await the numbers as requested.  The numbers that show the whole 
> picture, not just the parts you want to show, please.
>
> Remember when we had a ‘quota’ of 20-30 requests/replies per day. 
> Multiply that with 20-30 employees and at minimum you should be able 
> to process 2000 requests per week, not per month. There were employees 
> leaving the NCC and new ones (that needed training) joining all the 
> time. That did not stop the RS management to require the department to 
> respect the SLA daily.
> Again, and you remember this perfectly as we used to work in the same 
> department, a day missing the SLA was a tragedy, why is it now the norm?
>
>
>
>     While these statistics provide a glimpse into how things have changed
>     over the past few years, I would like to reiterate that it is not
>     just
>     the number of tickets but also the complexity of requests that has
>     increased. We report on this regularly at RIPE Meetings.
>
>
> Complexity of tickets before the RIPE NCC reached the last /8 
> (actually, until the requirements for additional allocations were 
> simplified in policy) was much higher than the complexity of current 
> transfer tickets. Indeed, back then the amount of attempts of fraud 
> were way lower.. but that doesn’t mean complexity has increased, au 
> contraire.
> Complexity has just changed from complex evaluations of networks, 
> deployment plans, equipment lists and network diagrams to anti-fraud. 
> The RIPE NCC’s RS dept. philosophy has also changed from ‘we trust our 
> members’ to ‘we need to make sure this is not fraud’. Maybe this 
> change has caused the increased complexity you are talking about?
>
>
>
>     Despite these challenges, Registry Services strives to handle
>     incoming
>     requests efficiently. In order to improve our ticket handling
>     times, we
>     have:
>     - Hired new staff, who are still being trained, but will help address
>     the workload once they are up to speed.
>     - Taken on temporary staff to enter registration details of resource
>     holders into our system which will allow us to automate certain due
>     diligence checks.
>     We will publish a RIPE Labs article about these changes in the
>     coming weeks.
>
>
> Not sure exactly what you mean by ‘enter registration details of 
> resource holders into our system’ but I look forward to reading more 
> details in the article.
>
>
>
>     Some additional clarifications regarding internal targets:
>     - Every ticket that doesn't get a response by 18:00 (UTC +2) the
>     following working day is counted as a ‘miss’ against our internal
>     targets.
>     - Our notification emails do not count against these targets (we
>     don’t
>     see much point in gaming our internal KPIs).
>     - A gentle reminder that Saturdays and Sundays are not working
>     days, so
>     requests received on a Friday are likely to be addressed the next
>     business day, i.e. the Monday after the weekend.
>
>
> See ticket #398137, for example. Ticket opened Aug 26 at 4AM, received 
> an automated message the same day saying that due to workload… That 
> was on Thursday. Do you agree with me that this ticket should have 
> been handled on Friday? If yes, why was it resolved only next Monday?
> Marco, this is one of many tickets that have been delayed for no 
> reason. For some I have called the NCC to see what’s the problem, we 
> even talked a few times about some of the tickets that were being 
> delayed for no reason.. You know about them.
>
> See the other e-mail sent by Elina. Their situation is even worse…
>
> So, Marco, please tell us how many working days in the past few 
> months/years has the RS dept. missed the SLA?
> - actually, what may be easier to calculate is how many working days 
> you haven’t missed the SLA
>
> Around 2015-2017 (I don’t remember when exactly) the RS dept. decided 
> to stop showing how many requests it received and handled daily. 
> Before that it used to have a page showing the workload and how long 
> it was taking for a ticket to receive a reply. Although at that time I 
> asked for that functionality to be restored, the NCC preferred to hide 
> it’s head in the sand and decided that too much transparency was not 
> so good.
> - any plans to become transparent again about your workload and SLA?
>
>
>
>
>     We can appreciate that it is frustrating to wait on tickets. However,
>     while we’re still training additional staff and automating
>     systems, we
>     need to rely on your patience and ask that you allow us time to
>     ensure
>     requests are not just handled quickly, but also accurately.
>
>
>
>     Since we know from experience that the number of requests increases
>     dramatically in the last quarter of the year, I would like to
>     repeat our
>     request for members to kindly not wait until the last minute to
>     send in
>     a request, and to expect longer handling times in the last quarter
>     of 2021.
>
>
> I think this warning was and is a bad idea. Instead of warning the 
> members that you will miss the SLA even more in Q4, you should work on 
> fixing the bottlenecks. This situation has been ongoing for 
> months/years now… you can’t come now and say ‘we know we are slow, we 
> will be even slower in the future’, this should have been solved 
> already so you don’t have a bigger problem next Q.
>
>
>
>     Once I have a more detailed overview of our ticket history, I’ll
>     share
>     it on this list, along with how we plan to further improve our
>     services.
>
>     I am aware that I have not addressed all the concerns raised in your
>     email, Elvis, but I plan to continue this conversation in the
>     coming days.
>
>
> I look forward to your next e-mails. So far this one has had almost no 
> new information, you just repeated the same info you sent last month, 
> just worded it differently. You also tried to say you are handling 
> more requests now than years ago, you haven’t convinced me, instead 
> you tried to trick us with some numbers that were misleading.
>
> Looking forward to have an actual conversation about this.
>
> Elvis
>
>
>
>     Kind regards,
>     Marco Schmidt
>     Registry Services Assistant Manager
>     RIPE NCC
>
>     On 31/08/2021 06:06, Elvis Daniel Velea wrote:
>     > Dear Marco,
>     >
>     > thank you for the update below. I'd like to ask you to provide a
>     few
>     > more details and to detail your plans to fix the SLA violation
>     by the
>     > Registration Services department, please see my questions inline.
>     >
>     > On 8/4/21 3:43 AM, Marco Schmidt wrote:
>     >> Dear colleagues,
>     >>
>     >> We would like to give you an update on the ticket wait times in
>     >> Registry Services, after discussions at RIPE 82 [1] and on the
>     >> mailing list [2].
>     >>
>     >> We have been addressing the issues causing delays. As the
>     number of
>     >> incoming requests continue to exceed previous years, we have
>     taken on
>     >> additional staff to help with the workload. We are also automating
>     >> some of our due diligence processes which are currently time
>     consuming.
>     >
>     > You say that you receive more requests than in previous years,
>     can you
>     > please provide some statistics for the past 10 years showing the
>     > number of employees vs number of requests/replies received/sent
>     daily?
>     > - please try, if possible, to show us how many of these tickets
>     were
>     > related to transfers, how many to billing or closures, etc...
>     >
>     > Can you provide some stats showing how many replies are
>     necessary for
>     > a ticket to be resolved? Also, let me know what is being done so
>     that
>     > requests get evaluated and resolved in one reply.
>     >
>     > Also, please let us know how exactly have you addressed the
>     constant
>     > SLA violations, what part of which process has been automated and
>     > whether you believe that with the additional staff you will resolve
>     > the issues swiftly.
>     >
>     > As you probably remember, I used to work in the RS department
>     between
>     > 2007 and 2013. Back then, a day when the SLA was not respected was
>     > almost a tragedy... it seems that now it's the norm and not the
>     > exception any longer. What happened that your department ended
>     in this
>     > situation?
>     >
>     >>
>     >> You should start to see improvements in our response times towards
>     >> the end of this year once our new colleagues are trained and our
>     >> automation is ready. In the meantime, you may experience slower
>     >> response times. In other words, we will be slower before we can
>     get
>     >> up to speed properly, but we think this will be worth it over the
>     >> long term.
>     >
>     > Things are slower and slower. For example, last week I had a ticket
>     > receive 1 reply after ~24 hours saying that because of time
>     > constraints, the ticket will be evaluated later. Then Friday,
>     Saturday
>     > and Sunday passed before I got a reply on Monday.
>     >
>     > None of the tickets I or my customers opened last month (August)
>     have
>     > been replied to within the SLA. NONE! We're talking about more
>     than 10
>     > tickets in a month, none receiving a reply within the SLA.
>     >
>     > You are saying that waiting will be worth it over the long term
>     while
>     > right now all tickets are being delayed from a day to another and
>     > sometimes a ticket that should take 10 minutes to evaluate is
>     delayed
>     > for 4 days or more... This is not acceptable and that is why I'd
>     like
>     > to ask you for more details to prove to us that the wait will be
>     worth
>     > it. As things are getting worse and not better, I hope you
>     understand
>     > why I can't take your word for it :)
>     >
>     >
>     > When you generate those stats, please let us know if the replies
>     > saying 'Due to a very high workload, we were unable to process your
>     > request at this time and will process it as soon as possible.' are
>     > still counted as a violation of the SLA or not.
>     >
>     > Also, when you generate the stats I ask for, please also tell us
>     how
>     > many working days in a month (for the past year or two) the
>     > Registration Services dept. has violated the SLA.
>     >
>     >
>     > I've been trying to help the RS department evaluate all the
>     requests
>     > received from V4Escrow and it's customers quicker. The help I
>     offered
>     > was refused while the SLA violations got worse and worse.. maybe
>     it is
>     > time you accept the help when it is offered to you.
>     >
>     > I've had several conversation with RS management and higher up, all
>     > the way up to MD. I talked to several teams to explain the way I
>     > wanted to help RS and our customers get all the tickets resolved
>     > swiftly (at least the ones that are for transfers of resources). I
>     > proposed the introduction of an API for tickets/requests so we
>     can try
>     > to automate a large part of the transfer ticket, this automation
>     could
>     > help RS evaluate requests faster, would help customers provide
>     all the
>     > required information in one go; everyone would be happy. I was
>     told my
>     > request will be looked at and maybe added to plan no earlier than
>     > Q1-2022. Please tell me if that is still the plan or whether, in
>     the
>     > light of these constant SLA violation, you could enable the API
>     faster
>     > so we, brokers, can help you receive complete and easy to evaluate
>     > transfer requests.
>     >
>     >> We also know from experience that we receive far more requests
>     >> towards the end of the year which results in a bottleneck. We will
>     >> shift resources to counter this, but we would also appreciate
>     it if
>     >> you could get some of these requests in earlier.
>     >
>     > Is this a warning that requests will take weeks to process
>     towards the
>     > end of the year? They already take days...
>     >
>     > Please clarify the bottlenecks you have and let's see if there's
>     any
>     > way we can help you with them. I am sure that automating a lot of
>     > steps in your overly complicated processes will help but I am
>     looking
>     > forward to receiving more details from you about this.
>     >
>     >
>     > Elvis
>     >
>     > PS: Note that I wanted to also send an e-mail about the RIPE NCC's
>     > ticketing system (zendesk) to the members-discuss mailing list
>     and to
>     > the board members. I will wait for an answer to the e-mail I
>     sent to
>     > HPH one week ago (which he promised he'll forward to the
>     relevant team
>     > for a reply).
>     >
>     >>
>     >> Kind regards,
>     >> Marco Schmidt
>     >> Registry Services Assistant Manager
>     >> RIPE NCC
>     >>
>     >> [1] RIPE NCC Operational Update (slides 4-10):
>     >> https://ripe82.ripe.net/archives/video/584/
>     <https://ripe82.ripe.net/archives/video/584/>
>     >>
>     >> [2] Discussion on Members Discuss mailing list:
>     >>
>     https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html
>     <https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html>
>
>     >>
>     >>
>     >>
>     >> _______________________________________________
>     >> members-discuss mailing list
>     >> members-discuss at ripe.net <mailto:members-discuss at ripe.net>
>     >> https://mailman.ripe.net/
>     <https://mailman.ripe.net/>
>     >> Unsubscribe:
>     >>
>     https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net
>     <https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net>
>     >
>
> -- 
> This message was sent from a mobile device. Some typos may be possible.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.ripe.net/ripe/mail/archives/members-discuss/attachments/20210906/d1057282/attachment-0001.html>