[OpenIPMap] Bringing CAIDA's geoloc efforts into the fold

Tue Jun 17 11:27:49 CEST 2014

Hi,

On 2014.06.10. 11:29, Emile Aben wrote:
> Hi group,
> 
> As I mentioned during my RIPE68 presentation there is active 
> development on infrastructure geolocation going on at CAIDA, and I'd 
> like to see if and how we can align both efforts. My personal take is 
> we can and we should.
> 
> CAIDA have been developing 3 things: - Automatically detecting naming
> schemes for networks. This is called DROP, paper is here: 
> http://www.caida.org/publications/papers/2014/drop/drop.pdf - A means
> of documenting structure in naming schemes (hostpat). - A repository
> for naming schemes (DDec), which includes DROP and the undns datasets.
> Beta version available here: http://ddec.caida.org/
> 
> Hostpat is documented as part of DDec: http://ddec.caida.org/help.pl , 
> and I think it is a good attempt at lowering the complexities
> 
> What I'm currently working on is an interface in OpenIPMap that runs 
> hostnames (that are already in the OpenIPMap system) through DDec, and 
> imports any resulting hostname->location mapping into OpenIPMap, with a
> lower confidence then the user inputs.
> 
> I'll first test this with all the hostnames that users put an empty 
> string for in, meaning they did override the guessing system in 
> OpenIPMap, but didn't know the correct geolocation either. I have 3k of
> these currently (out of a 16k of total crowdsourced entries for 
> hostnames (go team!)). This could easily be extended with batch runs of
> far more hostnames (all we can find?) and/or doing an 'online' lookup
> into DDec whenever information for an hostname is being called for, and
> storing the resulting hostname->location mapping local to OpenIPMap.
> 
> This would allows for people to document naming schemes in DDec and 
> have OpenIPMap use the mapping resulting from naming schemes. I think 
> this is the fastest path forward towards better infrastructure 
> geolocation.
> 
> I'd like us to evaluate if this works for the group. Specifically for 
> the people who have expressed interest in describing structure in 
> hostnames using regular expressions (Martin, Daniel, Robert): Does
> DDec/hostpat work for you. If not: what not?

One of our goals is to involve "the crowd" to supply input to this
service. I personally don't have a strong preference about how exactly
people enter this input, as long as:
* the data is freely available to our (future) service and to the
community in general
* it's possible to attribute contributions to contributors in a consistent
way (ie. NCC access or such)
* technically, and from the intended use point of view, it does not make a
difference where this data is entered
* going this way does not impede our plans and actions

I'm OK with CAIDA doing this part, as long as the above holds. It's even
better if along the way their systems can be improved with this data.

As for accepting regexps or such: I cannot tell you if this suggestion is
a good enough substitute or not, as I don't know the details yet :-) I do
however have a good understanding on how our service could directly be
enhanced with regexp style inputs.

Bottom line: we'd have to know more about how our community (and we
ourselves) could do this and what are the consequences of using this
method, in order to assess whether this is the preferred way.

Cheers,
Robert

> I've invited kc and Brad from CAIDA to join this list, so hopefully we 
> can get some conversation going on what works and what won't.
> 
> What I'd personally like to avoid is unnecessary complexity and 
> duplicate work, so doing things collaboratively would have my *strong* 
> preference.
> 
> cheers, Emile
> 
> 
> PS: Some more stats on user-contributed hostname->loc mappings: user_id
> | count ---------+------- 22 |     9 20 |    95 17 |   138 16 |   208 
> 11 |   208 10 |   265 15 |   381 12 |   409 9 |  1836 3 |  1910 18 |
> 4159 5 |  6758
> 
> 
> Top contributors: Please don't get addicted :) :)
> 
> _______________________________________________ OpenIPMap mailing list 
> OpenIPMap at ripe.net https://www.ripe.net/mailman/listinfo/openipmap
> 