[db-wg] [opensource-wg] Question for RPSL Parser

Shane is right, I don’t have the intention to shuffle the project with Bison and Flex because 1) I don’t have the knowledge of these technologies 2) I don’t have the time to learn them 3) I want to Pythonize as much as I can and be independent as much as possible (OK maybe a little bit dependent from other Python Libraries but this is the boundary). I am sure that you guys made a great job 14 years before but now or soon RPSL will be replaced with something else so I still want my toolset to be alive by easily and quickly replacing 1-2 Python libraries.

What my prototype does now is extract the attributes and values by using Regular Expressions, a solution that has already been proposed in this discussion and I think this is the way to go. In theory, it can be an easy parser based on Python’s RE but complexity starts when -for example- RPSL says “accept as-set BLABLA” instead of “accept 10.100.200.0/16" and that as-set needs to be resolved (another query to RIPE DB).

But anyway, thanks for your ideas. Crazy or not, I don’t mind everyone is welcome. If I still need to reply to Tomas last e-mail but I am quite busy today.

Kind Regards
Stavros

> On 20 Aug 2015, at 13:05, Shane Kerr <shane at time-travellers.org> wrote:
> 
> Denis,
> 
> There are two problems with this:
> 
> 1. Lex and YACC (actually flex and bison for the RIPE Database) are
>   tightly integrated into C (or C++ if you are feeling brave). 
> 
> 2. Stavros does state a goal of avoiding reuse of past or legacy
>   software, which surely means the RIPE Database code. (Okay, I think
>   it was only 12 or 14 years ago when we wrote these parts, but that
>   surely counts as "past"...)
> 
> :)
> 
> Cheers,
> 
> --
> Shane
> 
> On Thu, 20 Aug 2015 10:56:20 +0000 (UTC)
> denis walker <dw100uk at yahoo.co.uk> wrote:
> 
>> HI
>> The RIPE Database I believe still uses LEX and YACC to parse the routing policy information from the RPSL attribute/value pairs. Is it possible to just 'drop' this into a python script and extend it to do any extra parsing you want? (I know nothing about python so I may be way off the mark here :) but it is just a thought)
>> cheersdenis
>>      From: Shane Kerr <shane at time-travellers.org>
>> To: Stavros Konstantaras <stavros at nlnetlabs.nl> 
>> Cc: db-wg at ripe.net 
>> Sent: Thursday, 20 August 2015, 5:21
>> Subject: Re: [db-wg] [opensource-wg] Question for RPSL Parser
>> 
>> Stavros,
>> 
>> On Wed, 19 Aug 2015 16:00:59 +0200
>> Stavros Konstantaras <stavros at nlnetlabs.nl> wrote:
>> 
>>> To make things more clear, our project at NLnet Labs is the
>>> development of "a modern IRRtoolset" written in Python and targeting
>>> any operator, not just another homemade tool for our own needs. Or to
>>> say in more detail the creation of a tool that is able to configure
>>> BGP routers directly+automatically by extracting the policies from
>>> RIPE DB. This means that I am struggling to avoid the (re)use of any
>>> past or legacy software that exists, otherwise we loose independency
>>> and inherit restrictions from the past. 
>> 
>> I don't know of any Python RPSL parser you can use.
>> 
>> Your goals make a certain amount of sense.  Unfortunately RPSL is an
>> ugly language... it's actually more like several languages in one, all
>> bad. :P
>> 
>> * Start with simple attribute/value, one per line
>> 
>> * Oh, but then "RPSL-ize", which allows line continuation and
>>   end-of-line comments
>> 
>> * Oh, actually we want to make a separate grammar for each attribute
>> 
>> * But be sure to make all of this "extensible", to insure maximum
>>   confusion!
>> 
>> My approach in the past has been to start with a simple generic parser
>> that split text into objects, objects into attributes, extracts
>> attribute name & values (handling line continuation and end-of-line
>> comments). You can do all of this with Python regular expressions,
>> something like r"\n(?![ \t+])" to split objects into attributes;
>> cleaning end-of-line comments & line continuation characters is
>> straightforward. Then you can deal with the grammar for the attributes
>> you know, and ignore the rest. You can extend the parsing to be more
>> and more comprehensive until you are able to parse the data set you
>> worry about.
>> 
>> I'm not sure this is worth doing... but if there was a good,
>> easy-to-use Python library then maybe interesting things would result?
>> 
>> 
>> 
>> Cheers,
>> 
>> --
>> Shane
>> 
>> 
>> 
>> 
>