Test Traffic Measurements (TTM) Survey 2002

1. Introduction

Prompted by the RIPE NCC's 2002 Members and Stakeholders Survey, the New Projects group decided to initiate a survey focusing on aspects of Test Traffic Measurements (TTM) that, due to scope constraints, could not be a part of the general survey. The objective of the TTM survey was to shed some light on user experiences in addition to obtaining valuable input on improvements, direction and potential services.

2. Methodology

The survey encompassed two different methods for gathering information from respondents. Most respondents provided answers by filling out a questionnaire and returning it either via normal or electronic mail. However, in three cases, test-box hosts were interviewed on-site in order to attain a more comprehensive understanding of how users interact with TTM services in daily operations. These hosts were arbitrarily chosen out of a group of candidates within reasonable geographical proximity to the RIPE NCC in Amsterdam.

We approached all test-box hosts for whom we had complete address information in early December last year. This meant that a total of 45 users, some who have more than one test-box, were either contacted in person or sent a questionnaire via the postal system. Sixteen hosts, nearly 36 percent of the initial sample, responded to our survey by our deadline in mid-January, 2003.

3. Results

3.1 Usage

We learnt from the on-site interviews that the TTM services were mainly used internally at Network Operation Centres (NOCs) for investigating customer complaints and monitoring transit providers. This implied that other applications for TTM data seemed to be rare.

Our impression was also supported by results in the survey. For instance, 75 percent said that they had never used TTM as a sales tool to acquire new business, and furthermore, the same number of respondents had never shown plots to upstream providers or customers when there were connectivity problems. There appeared to be some interest in integrating TTM into Network Management Systems, however, the results were not conclusive. Forty percent of respondents answered that integration would definitely be a major benefit, while 53 percent said "no" and another seven percent did not know for sure.

Comments:

"...eases visibility and reaction time..."
"...prefer manual access, enough for now..."

The average test-box host accesses the TTM web site on a weekly or monthly basis. Delay plots/summaries and the traceroute database appear to be the most frequently accessed services, while jitter plots and plots-on-demand are used to a lesser extent. Moreover, with some notable exceptions, the test-box web interface seems only to be used sporadically or not at all. Seventy-nine percent, who answered the question: "How often do you directly access the test-box web interface?", had either used the web interface twelve times or less per year or never accessed it. The majority of respondents who had tried it indicated that they had only investigated either the "inbound delay" and/or "satellite conditions" features on the test-box. Another service that appears to be utilised to a lesser extent than expected are the e-mail alarms trigged when unexpected measurement conditions are detected by TTM. Sixty percent of respondents indicated that they either always or usually ignore the notifications, mainly because of the sheer volume of messages generated per day.

Comments:

"...filter out these emails and check the TTM site myself..."
"...too many alarms that didn't indicate real problem..."

According to the survey, half of test-box hosts first heard about RIPE NCC's Test Traffic Measurements at a RIPE Meeting; another 25 percent found out about TTM on the Internet; and the rest selected other options such as "RIPE NCC contact", "mailing list" or "training course". Many respondents, about 38 percent, also indicated that the RIPE Meetings were the main forum for staying informed of TTM news and developments. Mailing lists and the web site were other important information channels for users. Looking at survey results, it was concluded that the awareness of TTM services among users is very good. Eighty-one percent expressed that they knew at least four out of six stated services. When asked about new services 44 and 50 percent, respectively, indicated that QoS/DiffServ and native IPv6 networks would be deployed within the short-term future. This was encouraging to the New Projects group as TTM supports IPv6 measurements since 1 February 2003.

3.2 Ratings

Respondents were asked to rate different variables in regard to the TTM web site, test-box interface, support and data analysis on a scale from one to six. In general, the averages of ratings varied from 4.7 to 5.6, which would have to be interpreted as "very good" to "excellent".

Both support and data analysis were given significantly higher average scores than the TTM web site or test-box interface, especially support was rated very highly by users. However, in cases where respondents gave lower scores, usually around four, more improvements to services were noted.

We can conclude that "ease of use" and "presentation" with regard to the TTM web site's appearance as well as the layout of measurement data are issues that need attention. Other items such as improved "online help", "features" and "configuration" options could be added to the test-box interface at a later stage. Please note that only TTM test-boxes of series C/D have a web interface.

3.3 Statements

As the RIPE NCC has been evaluating the TTM business model for the 2004 Activity Plan, test-box hosts were asked to express their opinions regarding a number of statements. The statements were:

"The financial cost of having TTM installed at your site has been a worthwhile investment."

Practically all of the respondents, 88 percent, strongly agreed, agreed, or slightly agreed that installing a test-box at their site had been a worthwhile investment. This indicates that, although some elements of the service need improvements, it is a welcome addition to the set of tools available to network operators.

"TTM should not change its current business model."

The results were quite conclusive in this case. Seventy-seven percent of those who answered either strongly agreed, or agreed with the statement. A number of "non-respondents" said privately that the construction of a business model was an internal matter for the RIPE NCC to solve on its own and, therefore, refrained from selecting their preference.

"TTM may raise its service fees by 25% to 50% and my organisation will continue to use the service."

As one would expect, most test-box hosts indicated that they disagree with this statement. Two-thirds of those who gave an answer slightly disagreed, disagreed, or strongly disagreed with a 25-50 percent increase of the service fee. During our on-site interviews, we did sense a certain amount of flexibility regarding this issue if TTM would improve or extend its services.

"The RIPE NCC may outsource TTM to a third party not-for-profit organisation that continues to provide the service under the same conditions as today."

The responses from hosts with regard to this statement were mixed and, therefore, harder to interpret. Fifty-six percent chose an "agree" option, while the rest have indicated that they disagree to some extent with above. However, there is an emphasis in the distribution of answers on "slightly agree", which was the response of 31 percent. It is clear that TTM users show some flexibility on this point. Their main concerns regarding a not-for-profit third party are related to the neutrality of the organisation and its immunity against mergers and acquisitions.

"The RIPE NCC may outsource TTM to a third party commercial organisation that continues to provide the service under the same conditions as today."

It became very clear that the above statement was quite unacceptable to users. Nearly two-thirds of respondents, 62 percent, either disagreed or strongly disagreed with outsourcing TTM services to a commercial third party. It can be concluded that this is not a viable route for the future of TTM.

4. Conclusions

The main points from the survey have been summarised into three categories:

4.1 Strong Points

• Most of the respondents have only praise for the quality of TTM data and support. New Projects has apparently done well in this aspect as, in most cases, TTM is a RIPE NCC service that is highly regarded by users.
• During the on-site interviews, most have emphasised the importance of the neutrality and objectivity of the RIPE NCC. It is considered that few other organisations could be trusted to produce unbiased internetwork performance measurements. This is possibly the strongest argument for a TTM service provided by the RIPE NCC.

4.2 Improvements Required

• The usability of the TTM web site and presentation of data needs much attention. It is clear that the structure of our web site needs to be reconsidered and TTM data made more accessible to users who are not engineers. For instance, some survey participants suggested producing the TTM plots in two versions, one for network operators and another for people with a non-technical background. Other suggestions from respondents address the lack of customisation possibilities of TTM services such as controlling the amount of alarms sent to hosts, selecting relevant measurement relations on delay summaries, and being able to group them by region or upstream providers.
• Throughout the on-site interviews, we were surprised to learn that TTM users did not access the web interface features on the C/D series of the test-boxes as much as expected. However, most hosts did show an interest in these features once they became aware that the plots of inbound delays and GPS conditions were in near-real-time. There seem to be a couple of pitfalls here that should be avoided in the future. These include:
  a. Adding features that do not benefit most users instead of focusing resources on consolidating the existing product.
  b. Incorrectly assuming that hosts have digested TTM communications just because a message has been announced on a mailing list or at a RIPE Meeting.

These situations could be avoided by actively seeking more interaction with users, i.e. by conducting beta testing programmes, visiting customers on-site, and doing surveys more frequently. Improved interaction with customers is definitely needed.

4.3 Features Wanted by Users

As mentioned previously, it is prudent to focus on improving and consolidating current features before developing new ones. Nevertheless, survey respondents have provided many interesting suggestions that should be investigated further in parallel with consolidation. Here are a few examples:

• Bandwidth measurements
• SNMP/NMS integration of alarms
• Internal measurements between PoPs
• Measurements from multiple NICs
• Manipulating the ToS field in the IP header
• Protocol-based performance analysis

5. Lessons Learnt

The outcome of this survey has provided the TTM crew with much food for thought. The main lesson being learnt is the importance of actively approaching users for ideas on improving current features before moving on with new projects. It is imperative for TTM's appeal that the New Projects group approach its customer base more often in the future. Obtaining feedback by conducting a survey like this has been a very rewarding experience.

Credits to all who participated.
Many thanks!