[atlas] Introducing: the new result parsing library (Sagan)

On 26/05/14 16:50, Stephane Bortzmeyer wrote:

    import sys

    from ripe.atlas.sagan import PingResult

    if len(sys.argv) <= 1:
    raise Exception("Usage: %s filename ..." % sys.argv[0])

    for filename in sys.argv[1:]:
    results = open(filename).read()
    result = PingResult(results)
    print filename
    print result.rtt_median
    print ""

The thing to remember is that this is a /result/ parser, not a
/result*s*/ parser. In other words, you have to pass each result
individually into Sagan for parsing and it will in turn return a Python
object for that single result.

Your code here takes /the entire file/, multiple results in a JSON list,
and dumps it into Sagan, which explodes because you're passing multiple
results. Generally, this is bad practise since it's entirely possible
that your one file could be bigger than a few GB, which would definitely
crash your system if you tried to load the entire file into memory.

Instead, I suggest the following:

|for filename in sys.argv[1:]:
    with open(filename) as my_results:
        for result in json.loads(my_results):
            result = Result.get(result)
            print(result.rtt_median)
|

or skip the manual JSON parsing by using the fragmented file format
(|?format=txt|):

| for filename in sys.argv[1:]:
     with open(filename) as my_results:
         for result in my_results.readlines():
             result = Result.get(result)
             print(result.rtt_median)
|

The key is to remember that you need to break up that Great Big File
into separate result blobs, either by looping over a fragmented file's
lines, or by parsing the whole JSON file into memory first and parsing
out each result. That step is up to you. Sagan only takes over once
you've got a single result to work with.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ripe/mail/archives/ripe-atlas/attachments/20140526/f7aac324/attachment.html>