Draft minutes RIPE50 TTM session #2

[Lorenzo Colitti <]

Apologies for getting Rickard and Niall's names wrong!

Here is a second draft with those corrections made.

Regards,
Lorenzo

====================8<-----------------------

RIPE 50 TTM WORKING GROUP MINUTES
=================================

Thursday, May 5, 14:00-15:30
Stockholm, Sweden

Lorenzo Colitti (RIPE NCC) volunteered to scribe.

The minutes of the previous meetings were approved.

Name abbreviations
==================

AS = Ad Spelt
AT = Alexander Tudor
BN = Brian Nisbet
CP = Christian Panigl
GN = Geert Nijpels
HU = Henk Uijterwaal
LC = Lorenzo Colitti
LD = Luca Deri
NR = Niall O'Reilly
RD = Rickard Dahlstrand
RR = Rainer Rudiger
TS = Thomas Schmid


Presentations
=============

Henk Uijterwaal: TTM status update
----------------------------------

- Internal restructuring done
- DNSMON v1 out, now a self-supporting service. Request list for
  v2 exists
- Technical improvements: taildaemon bug fixed, timekeeping
  improvements
- CDMA clocks: looking for a volunteer to help debug problems
- OWAMP: currently planning implementation
- Other work in progress
  - Bandwidth, VOIP measurements, v4/v6 comparison
  - Look at Neil O'Reilly's ISP quality measurements


Thomas Wana: Improving Time Measurement in TTM
----------------------------------------------

- Evaluated precision of current TTM setup

- Looked at NTP alternatives
  - ATOM is a little better than standard PPS_SYNC and will be used
  - ntpns only works on FreeBSD 5

- Improved sending timestamping
  - Implemented kernel-level timestamping for TTM sender
    - Lower offset, lower variation
    - Production ready

- Looked at receiver timestamping
  - This is the major source of noise in current TTM
  - Not easy to fix, depends on FreeBSD architecture
  - Proposal: equip testboxes with DAG cards
    - Accuracy is extremely good
    - Requires calculating delays centrally
    - Results on the testboxes themselves would continue using the
      old numbers
    - Would be optional
    - A DAG card costs ~800 euros


LD: You should use something like nCap to obtain direct access to
the card.


Discussion on Third Party Verified SLA's
========================================

AT: It's not clear to the membership what the value proposition of
TTM is. The original idea was that ISP would be interested in
comparing their performance to others, but it was not until very
recently that someone proposed to use TTM in operations.

There is a place for an organisation such as RIPE NCC that could
provide an unbiased metric. ISPs or IX operators could then point to
the data to show that their service is good.

Points for discussion:

- Does it make sense for RIPE NCC to play the role of a trusted
  third party?
- If so, what do we want it to measure? Loss, delay, etc. are
  probably not sufficient? What does it take to turn the service
  into something that's useful for members and their customers?

RD: We must distinguish between several markets: service providers,
    government, content providers, end users

TS: Let me add "the press"

AT: The press always turns any measurement into a competition or a
    race. This is meant to help ISPs. The measurements must be data
    that the members can point to, not something to pit one ISP
    against another.

RR: As soon as you measure something, you are comparing

RD: Compare with DNSMON: it can be used to see which of the three
    competitors for .se is best. They don't like the fact that DNSMON
    is publicly available. Similarly if TTM data is available that
    could be used to make ISPs look bad.

AT: We need to allow the ISP to provide info on SLAs

HU: 1. This is not about monitoring the quality of service of one ISP
    (meaningless since there are different packages, SLAs,
    overprovisioning, prices...). What is interesting for a customer
    is "does ISP X deliver what they promise?"
    2. The .se operators might not be happy, but they *are* committed
    to provide good service, and if there is an objective tool that
    shows that they are not then it is interesting both for users and
    for them themselves.

RD: This is still an obstacle. The test-boxes are placed in the network
    of the ISPs that they are going to monitor and the ISPs won't want
    to host them if they fear the results might make them look bad.

AT: Again this is not about ISP x vs ISP y but measuring ISP x against
    their claims. This is not contentious and is good for the service
    provider.

BN: It's a fine distinction. Users may say "well, I'll move to the
    ISP who *does* fulfil their promises"

AT: This is an important concern. But sooner or later users will
    become more savvy. Do ISPs want to be prepared or not?

BS: Well, for us the point is that we are not a commercial service...

AT: How many people here are ISPs?

Show of hands: 5

AT: So you ISPs: if you went to marketing and asked to do this, would
    it fly?

AS: In the Netherlands we already have a service that ranks ISPs

TS: You can cheat and apply qos to the addresses that measure quality

AS: We can't: they just buy our products and use them to go to
    random websites

AT: This is getting close to Neil's talk. Who would be happy with
    providing such information to their customers?

RR: We do publish information. Actually, we did the measurements
    to prove to the customer that our service was ok and that the
    problem was on their side.

AT: So the customer doesn't trust your numbers and wants a third party.

RD: ISP networks are complex. What are we testing? DSL? Distribution?
    Core?

AT: We don't know. What should we do? You tell us. Those ISPs who say
    this is not an idea, why would this not be a good idea?

Reaction from the participants: none

AT: So RIPE NCC is a good party to measure this? It would be
    trustworthy?

GN: we chose RIPE NCC precisely because it's is a trusted third party

RR: The residential customer does not know who RIPE NCC is.

AT: So you would need to build up the brand

HU: You just say "this is a third party"

RR: If we document exactly how the measurement is done, it doesn't
    matter who is doing it.

AT: So you could base your SLAs on standard numbers that TTM measures.

AT: Second point: we agree that RIPE NCC is a good third party to do
    this work, but what is this work? What is it that is interesting
    for ISPs and their customers to measure?

RD: In Sweden we have 3 types of services that are used to test
    bandwidth, etc. We could complement that not with a network of
    measurement points (too expensive to deploy many boxes in Sweden)
    but by providing a test suite or a tool which could be used by
    other people

AT: So we could put a test-box at a public exchange point and people
    can sign up for measurements by providing a pingable address.
    Would this be acceptable?

LC: How do you know it's not the user's fault if his connection is
    performing badly?
AT: You don't...

RD: I think we need something intermediate between the current ttm
    network and this solution. Maybe a 100 euro box that could be
    deployed where the customers want it? There is no single point in
    the network that you can use as reference for measurement

HU: You could provide a box at the ISP's edge and measure from there
    to the Internet and from there to the customers. Internet users
   installing hardware at home is a logistical nightmare.


AT: We need to walk before we run. First we need to show that there
    is community interest in a proposal. So is a public exchange
    point a reasonable place from which to use the TTM metrics as
    they stand today?

LC: An exchange point LAN doesn't make sense, most of the time they
    don't even use globally routed addresses.

GN: I talked to people at Euro-IX and there is interest from other
    IXs to use TBs for multi-location L2 exchange point monitoring.
    That's a new market.

CP: Operating a reference point at an exchange point is logical, but
    the exchange points in europe are very neutral with their
    customers (the ISPs). Providing data for ISP's would seem to be
    discrimination between ISPs. Also, if we were a reference point
    for measuring we would need to answer user requests too.

RD: In Sweden we have a bandwidth measurement project and there is a
    reference point for that at Netnod.

AT: The customer doesn't care where the box is. He knows that the box
    is operated by a third party and that his ISP has given him a URL
    to a page where he can check the measurements. But is an exchange
    point a good location?

BN: I don't think the ISP would think the IX was taking sides. But the
    exchange is not always the best egress point in the network from
    which to measure.

AT: Ok, so to wrap things up:
    - We agree that this is an idea that makes sense
    - We should continue discussion amongst people who are interested
    - Who would like to provide a proposal?

RD, RR, NR volunteer

AT: Ok, so we expect something before next meeting.