[members-discuss] Registry Services Ticket Response Time

Hi Marco,

thank you for your reply. Please see further comments below. Maybe some
will help you with your next reply.

On Fri, Sep 3, 2021 at 01:57 Marco Schmidt <mschmidt at ripe.net> wrote:

> Dear Elvis,
>
> Your email touches on multiple issues. I‘d like to address some of them
> now, with the understanding that we will share more detailed information
> soon. For now, I can provide a general overview on what the situation
> with wait times is, and how we plan to tackle this issue in the coming
> weeks.


ok


>
> First, a quick historical comparison for context. Back in 2013, the RIPE
> NCC had around 9,000 LIRs, and we processed a mere 150 policy transfers
> in the whole year, i.e. approximately 12-13 transfer requests a month.


Please do not compare apples to oranges. In 2013 the RIPE NCC was
processing hundreds of requests for initial and additional allocations,
requests that were way more complex than some of the current resource
requests (which no longer require any need based justification or
evaluation) or transfer requests that only require the RIPE NCC to verify
(i) if an organization is still active in their country and (ii) the
document you received is signed by the people with PoA.


> We currently have about 23,700 LIRs, which is 2.5x the LIRs in 2013.
> Last month, we processed over 400 policy transfers. This is 30x the
> number of transfer requests we processed when you were part of the
> Registry Services team. And remember that there are all of the other
> requests on top of this.


That is why I was asking you to provide total number of tickets handled,
back then, now and every year in between. A ticket is a ticket, some are
simple, some are more complicated. Same was then as it is now…

You are right, you did not process hundreds of transfer requests in 2013,
but there were many other types of requests that were way more complex than
some of current transfer requests. So saying that you process 30x transfer
requests is a bit misleading as you are not showing the complete picture.

The NCC RS dept. even succeeded to respect the SLA when it was evaluating
requests in pairs, towards the end of the runout. At that time you were
receiving maaany requests, more than usually. If the department was able to
cope with that situation, why can’t it now? I still have not seen the
answer to this question.

>
>
> In 2020, we averaged around 1,600 resources-related requests per month.
> However, from November to December there was a threefold increase
> compared to the rest of the year. It is hard to avoid an increased wait
> time when this happens, and that’s why we prefer to be upfront when we
> expect delays, so members can plan around them.


I await the numbers as requested.  The numbers that show the whole picture,
not just the parts you want to show, please.

Remember when we had a ‘quota’ of 20-30 requests/replies per day. Multiply
that with 20-30 employees and at minimum you should be able to process 2000
requests per week, not per month. There were employees leaving the NCC and
new ones (that needed training) joining all the time. That did not stop the
RS management to require the department to respect the SLA daily.
Again, and you remember this perfectly as we used to work in the same
department, a day missing the SLA was a tragedy, why is it now the norm?


>
> While these statistics provide a glimpse into how things have changed
> over the past few years, I would like to reiterate that it is not just
> the number of tickets but also the complexity of requests that has
> increased. We report on this regularly at RIPE Meetings.


Complexity of tickets before the RIPE NCC reached the last /8 (actually,
until the requirements for additional allocations were simplified in
policy) was much higher than the complexity of current transfer tickets.
Indeed, back then the amount of attempts of fraud were way lower.. but that
doesn’t mean complexity has increased, au contraire.
Complexity has just changed from complex evaluations of networks,
deployment plans, equipment lists and network diagrams to anti-fraud. The
RIPE NCC’s RS dept. philosophy has also changed from ‘we trust our members’
to ‘we need to make sure this is not fraud’. Maybe this change has caused
the increased complexity you are talking about?


>
> Despite these challenges, Registry Services strives to handle incoming
> requests efficiently. In order to improve our ticket handling times, we
> have:
> - Hired new staff, who are still being trained, but will help address
> the workload once they are up to speed.
> - Taken on temporary staff to enter registration details of resource
> holders into our system which will allow us to automate certain due
> diligence checks.
> We will publish a RIPE Labs article about these changes in the coming
> weeks.


Not sure exactly what you mean by ‘enter registration details of resource
holders into our system’ but I look forward to reading more details in the
article.


>
> Some additional clarifications regarding internal targets:
> - Every ticket that doesn't get a response by 18:00 (UTC +2) the
> following working day is counted as a ‘miss’ against our internal targets.
> - Our notification emails do not count against these targets (we don’t
> see much point in gaming our internal KPIs).
> - A gentle reminder that Saturdays and Sundays are not working days, so
> requests received on a Friday are likely to be addressed the next
> business day, i.e. the Monday after the weekend.


See ticket #398137, for example. Ticket opened Aug 26 at 4AM, received an
automated message the same day saying that due to workload… That was on
Thursday. Do you agree with me that this ticket should have been handled on
Friday? If yes, why was it resolved only next Monday?
Marco, this is one of many tickets that have been delayed for no reason.
For some I have called the NCC to see what’s the problem, we even talked a
few times about some of the tickets that were being delayed for no reason..
You know about them.

See the other e-mail sent by Elina. Their situation is even worse…

>
So, Marco, please tell us how many working days in the past few
months/years has the RS dept. missed the SLA?
- actually, what may be easier to calculate is how many working days you
haven’t missed the SLA

Around 2015-2017 (I don’t remember when exactly) the RS dept. decided to
stop showing how many requests it received and handled daily. Before that
it used to have a page showing the workload and how long it was taking for
a ticket to receive a reply. Although at that time I asked for that
functionality to be restored, the NCC preferred to hide it’s head in the
sand and decided that too much transparency was not so good.
- any plans to become transparent again about your workload and SLA?



>
> We can appreciate that it is frustrating to wait on tickets. However,
> while we’re still training additional staff and automating systems, we
> need to rely on your patience and ask that you allow us time to ensure
> requests are not just handled quickly, but also accurately.


>
> Since we know from experience that the number of requests increases
> dramatically in the last quarter of the year, I would like to repeat our
> request for members to kindly not wait until the last minute to send in
> a request, and to expect longer handling times in the last quarter of 2021.


I think this warning was and is a bad idea. Instead of warning the members
that you will miss the SLA even more in Q4, you should work on fixing the
bottlenecks. This situation has been ongoing for months/years now… you
can’t come now and say ‘we know we are slow, we will be even slower in the
future’, this should have been solved already so you don’t have a bigger
problem next Q.


>
> Once I have a more detailed overview of our ticket history, I’ll share
> it on this list, along with how we plan to further improve our services.
>
> I am aware that I have not addressed all the concerns raised in your
> email, Elvis, but I plan to continue this conversation in the coming days.


I look forward to your next e-mails. So far this one has had almost no new
information, you just repeated the same info you sent last month, just
worded it differently. You also tried to say you are handling more requests
now than years ago, you haven’t convinced me, instead you tried to trick us
with some numbers that were misleading.

Looking forward to have an actual conversation about this.

Elvis


>
> Kind regards,
> Marco Schmidt
> Registry Services Assistant Manager
> RIPE NCC
>
> On 31/08/2021 06:06, Elvis Daniel Velea wrote:
> > Dear Marco,
> >
> > thank you for the update below. I'd like to ask you to provide a few
> > more details and to detail your plans to fix the SLA violation by the
> > Registration Services department, please see my questions inline.
> >
> > On 8/4/21 3:43 AM, Marco Schmidt wrote:
> >> Dear colleagues,
> >>
> >> We would like to give you an update on the ticket wait times in
> >> Registry Services, after discussions at RIPE 82 [1] and on the
> >> mailing list [2].
> >>
> >> We have been addressing the issues causing delays. As the number of
> >> incoming requests continue to exceed previous years, we have taken on
> >> additional staff to help with the workload. We are also automating
> >> some of our due diligence processes which are currently time consuming.
> >
> > You say that you receive more requests than in previous years, can you
> > please provide some statistics for the past 10 years showing the
> > number of employees vs number of requests/replies received/sent daily?
> > - please try, if possible, to show us how many of these tickets were
> > related to transfers, how many to billing or closures, etc...
> >
> > Can you provide some stats showing how many replies are necessary for
> > a ticket to be resolved? Also, let me know what is being done so that
> > requests get evaluated and resolved in one reply.
> >
> > Also, please let us know how exactly have you addressed the constant
> > SLA violations, what part of which process has been automated and
> > whether you believe that with the additional staff you will resolve
> > the issues swiftly.
> >
> > As you probably remember, I used to work in the RS department between
> > 2007 and 2013. Back then, a day when the SLA was not respected was
> > almost a tragedy... it seems that now it's the norm and not the
> > exception any longer. What happened that your department ended in this
> > situation?
> >
> >>
> >> You should start to see improvements in our response times towards
> >> the end of this year once our new colleagues are trained and our
> >> automation is ready. In the meantime, you may experience slower
> >> response times. In other words, we will be slower before we can get
> >> up to speed properly, but we think this will be worth it over the
> >> long term.
> >
> > Things are slower and slower. For example, last week I had a ticket
> > receive 1 reply after ~24 hours saying that because of time
> > constraints, the ticket will be evaluated later. Then Friday, Saturday
> > and Sunday passed before I got a reply on Monday.
> >
> > None of the tickets I or my customers opened last month (August) have
> > been replied to within the SLA. NONE! We're talking about more than 10
> > tickets in a month, none receiving a reply within the SLA.
> >
> > You are saying that waiting will be worth it over the long term while
> > right now all tickets are being delayed from a day to another and
> > sometimes a ticket that should take 10 minutes to evaluate is delayed
> > for 4 days or more... This is not acceptable and that is why I'd like
> > to ask you for more details to prove to us that the wait will be worth
> > it. As things are getting worse and not better, I hope you understand
> > why I can't take your word for it :)
> >
> >
> > When you generate those stats, please let us know if the replies
> > saying 'Due to a very high workload, we were unable to process your
> > request at this time and will process it as soon as possible.' are
> > still counted as a violation of the SLA or not.
> >
> > Also, when you generate the stats I ask for, please also tell us how
> > many working days in a month (for the past year or two) the
> > Registration Services dept. has violated the SLA.
> >
> >
> > I've been trying to help the RS department evaluate all the requests
> > received from V4Escrow and it's customers quicker. The help I offered
> > was refused while the SLA violations got worse and worse.. maybe it is
> > time you accept the help when it is offered to you.
> >
> > I've had several conversation with RS management and higher up, all
> > the way up to MD. I talked to several teams to explain the way I
> > wanted to help RS and our customers get all the tickets resolved
> > swiftly (at least the ones that are for transfers of resources). I
> > proposed the introduction of an API for tickets/requests so we can try
> > to automate a large part of the transfer ticket, this automation could
> > help RS evaluate requests faster, would help customers provide all the
> > required information in one go; everyone would be happy. I was told my
> > request will be looked at and maybe added to plan no earlier than
> > Q1-2022. Please tell me if that is still the plan or whether, in the
> > light of these constant SLA violation, you could enable the API faster
> > so we, brokers, can help you receive complete and easy to evaluate
> > transfer requests.
> >
> >> We also know from experience that we receive far more requests
> >> towards the end of the year which results in a bottleneck. We will
> >> shift resources to counter this, but we would also appreciate it if
> >> you could get some of these requests in earlier.
> >
> > Is this a warning that requests will take weeks to process towards the
> > end of the year? They already take days...
> >
> > Please clarify the bottlenecks you have and let's see if there's any
> > way we can help you with them. I am sure that automating a lot of
> > steps in your overly complicated processes will help but I am looking
> > forward to receiving more details from you about this.
> >
> >
> > Elvis
> >
> > PS: Note that I wanted to also send an e-mail about the RIPE NCC's
> > ticketing system (zendesk) to the members-discuss mailing list and to
> > the board members. I will wait for an answer to the e-mail I sent to
> > HPH one week ago (which he promised he'll forward to the relevant team
> > for a reply).
> >
> >>
> >> Kind regards,
> >> Marco Schmidt
> >> Registry Services Assistant Manager
> >> RIPE NCC
> >>
> >> [1] RIPE NCC Operational Update (slides 4-10):
> >> https://ripe82.ripe.net/archives/video/584/
> >>
> >> [2] Discussion on Members Discuss mailing list:
> >>
> https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html
> >>
> >>
> >>
> >> _______________________________________________
> >> members-discuss mailing list
> >> members-discuss at ripe.net
> >> https://mailman.ripe.net/
> >> Unsubscribe:
> >>
> https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net
> >
>
-- 
This message was sent from a mobile device. Some typos may be possible.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.ripe.net/ripe/mail/archives/members-discuss/attachments/20210903/e94d4146/attachment-0001.html>