Crawl rubenvarela.com¶

Get all results, even if okay (--verbose)
By default it will only return values that are errors or warnings.
You can omit the --verbose argument if you want.
Store the results in XML format, linkchecker-out.xml

In [141]:

!linkchecker -F xml http://rubenvarela.com --check-extern --verbose

30 threads active,     0 links queued,    4 links in   1 URL checked, runtime 1 seconds
30 threads active,    10 links queued,    7 links in   4 URLs checked, runtime 6 seconds
30 threads active,   124 links queued,  306 links in  39 URLs checked, runtime 11 seconds
30 threads active,    97 links queued,  333 links in  47 URLs checked, runtime 16 seconds
30 threads active,    59 links queued,  371 links in  60 URLs checked, runtime 21 seconds
30 threads active,    28 links queued,  402 links in  67 URLs checked, runtime 26 seconds
28 threads active,     0 links queued,  432 links in  75 URLs checked, runtime 31 seconds
13 threads active,     0 links queued,  447 links in  81 URLs checked, runtime 36 seconds
 0 threads active,     0 links queued,  460 links in  87 URLs checked, runtime 41 seconds

In [142]:

import xmltodict
from random import sample as randSample

In [143]:

with open('linkchecker-out.xml') as f: 
  obj = xmltodict.parse(f.read())

Explore the object.
- Get the main element and see it's content.
- Get 5 random elements and print them in order to see the objects.

In [144]:

for x in obj['linkchecker']:
    print x

# print type(obj['linkchecker'])
# print obj['linkchecker']['@created']
# print type(obj['linkchecker']['urldata'])

randset = randSample(obj['linkchecker']['urldata'], 5)
randset

# print obj['linkchecker']['urldata'][0]
# print obj['linkchecker']['urldata'][1]
# print obj['linkchecker']['urldata'][2]
# print obj['linkchecker']['urldata'][3]
# print obj['linkchecker']['urldata'][99]
# print obj['linkchecker']['urldata'][100]
# print obj['linkchecker']['urldata'][101]
# print ''
# print obj['linkchecker']['urldata'][100]['parent']['#text']

@created
urldata

Out[144]:

[OrderedDict([(u'url', u'http://getbootstrap.com'), (u'name', u'Bootstrap'), (u'parent', OrderedDict([(u'@column', u'13'), (u'@line', u'162'), ('#text', u'http://rubenvarela.com/')])), (u'realurl', u'http://getbootstrap.com'), (u'extern', u'1'), (u'dlsize', u'11937'), (u'checktime', u'7.068993'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'http://getbootstrap.com'), (u'name', u'Bootstrap'), (u'parent', OrderedDict([(u'@column', u'13'), (u'@line', u'261'), ('#text', u'http://rubenvarela.com/ana-g-mendez-university-system-agmus.html')])), (u'realurl', u'http://getbootstrap.com'), (u'extern', u'1'), (u'dlsize', u'11937'), (u'checktime', u'7.068993'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'http://rubenvarela.com/aaa-subastas-prasa-data-bids.html'), (u'name', u'AAA Subastas (PRASA Data Bids)'), (u'parent', OrderedDict([(u'@column', u'21'), (u'@line', u'83'), ('#text', u'http://rubenvarela.com/aaa-subastas-prasa-data-bids.html')])), (u'realurl', u'http://rubenvarela.com/aaa-subastas-prasa-data-bids.html'), (u'extern', u'0'), (u'dltime', u'0.000377'), (u'dlsize', u'9439'), (u'checktime', u'5.711945'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'http://rubenvarela.com/theme/css/pygments/native.css'), (u'parent', OrderedDict([(u'@column', u'5'), (u'@line', u'31'), ('#text', u'http://rubenvarela.com/museo-de-arte-de-puerto-rico-puerto-rico-art-museum.html')])), (u'realurl', u'http://rubenvarela.com/theme/css/pygments/native.css'), (u'extern', u'0'), (u'dlsize', u'4732'), (u'checktime', u'2.197381'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'/images/SUAGM/old/01_-_SUAGM_-_2015-05-25_21.55.28.png'), (u'name', u"'Old Home Page'"), (u'parent', OrderedDict([(u'@column', u'4'), (u'@line', u'147'), ('#text', u'http://rubenvarela.com/ana-g-mendez-university-system-agmus.html')])), (u'realurl', u'http://rubenvarela.com/images/SUAGM/old/01_-_SUAGM_-_2015-05-25_21.55.28.png'), (u'extern', u'0'), (u'dlsize', u'1165284'), (u'checktime', u'10.259280'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))])]

Now to extract links with errors:¶

For each URL,
If the result is not okay,
Check if the valid and parent keys exists.
If they do, then print the values.
If the necessary keys don't exist, print the object.

In [145]:

for x in obj['linkchecker']['urldata']:
#     print x['valid']['@result']
    if not x['valid']['@result'] == "200 OK" :
        if 'valid' and 'parent' in x: #if keys exists in x
            print "Result Code: " + x['valid']['@result']
            print "Link location: " + x['parent']['#text']
            print "Line: " + x['parent']['@line']
        else:
            print "Couldn't find all elements:"
            print x
        print "Links to: " + x['url']

In [ ]: