WIKIzualize it, WIKIzualize it!

Good evening, dear friends!

Recently, walking through the vast expanses of the Internet, I came across the amazing work of Chris Harrison , after sitting a little in shock, I thought, “Is it hard to visualize Wikipedia or not?” And decided to try it!


')
So let's get started!

Instruments and tools

First of all, it is necessary to decide what we will visualize and by what means. And having a little studied what is and how my choice fell on the following means:

• MediaWiki API
• Python language
• PyGraphviz module

The developers of the Wiki API described it just fine, so you should not translate the description literally to understand how to call one or another method.

On Habré already wrote about the package Graphviz, so I think you should not re-describe it. But I immediately read the description of the dot language and graphviz tools and I almost started writing my own .dot files.

I was rescued by a python module called PyGraphviz, which allows you to conveniently work with the structure of the graph, which is then written into a .dot file.

The basics

So we will visualize cross-articles from Wikipedia. To do this, we need to call the method by reference:
ru.wikipedia.org/w/api.php?action=query&format=xml&titles=_&prop=links
where action is the type of method, format is the output format of the response, in our case it is XML, prop - we ask for cross-references links

at the output we get the following answer:

1. < api >
2. < query >
3. < normalized >
4. < n from = "Habrahabr" to = "Habrahabr" />
5. </ normalized >
6. < pages >
7. < page pageid = "340809" ns = "0" title = "Habrahabr" >
8. < links >
9. < pl ns = "0" title = "2006" />
10. < pl ns = "0" title = "2006" />
11. < pl ns = "0" title = "2007" />
12. < pl ns = "0" title = "Digg.com" />
13. < pl ns = "0" title = "Linux.org.ru" />
14. < pl ns = "0" title = "News 2.0" />
15. < pl ns = "0" title = "Newsland" />
16. < pl ns = "0" title = "Pligg" />
17. < pl ns = "0" title = "Slashdot" />
18. < pl ns = "0" title = "URL" />
19. </ links >
20. </ page >
21. </ pages >
22. </ query >
23. < query-continue >
24. < links plcontinue = "340809 | 0 | Blog" />
25. </ query-continue >
26. </ api >

* This source code was highlighted with Source Code Highlighter .

Which is processed using any DOM or SAX method.

Programming

So for processing, I used SAX and inherited my class from xml.sax.handler.ContentHandler:
class LinksListHandler(xml.sax.handler.ContentHandler):
Further, the main challenges are redefined:

• startElement
• endElement

The procedure for working with a query is as follows:

1. def get_links (page):
2. #See wiki api documentation http: //en.wikipedia.org/w/api.php
3. query_val = { 'action' : 'query' ,
4. 'prop' : 'links' ,
5. 'titles' : page,
6. 'format' : 'xml' }
7. url = wiki_url () + '?' + urllib.urlencode (query_val)
8. request = urllib2.Request (url)
10. verbose_message ( "Wiki url:" + url)
11. try :
12. response = urllib2.urlopen (request)
13. except urllib2.HTTPError:
14. print "HTTP request error!"
15. sys.exit (1)
16. #verbose_message ( "Response xml: \ n" + response.read ())
17. lh = LinksListHandler ()
18. saxparser = xml.sax.make_parser ()
19. saxparser.setContentHandler (lh)
20. saxparser.parse (response)
22. return lh.links

* This source code was highlighted with Source Code Highlighter .

Graphing

Using the PyGraphviz module, the work is quite simple:

1. def make_wiki_graph (wiki_page, depth):
2. gv = AGraph ()
4. page_list = [wiki_page]
5. temp_list = []
7. verbose_message ( 'Create graph for' + wiki_page)
8. pageLinks = get_links (wiki_page)
9. gv.add_node (wiki_page)
10. for i in range (depth):
11. print '>>>> Get' + str (i) + 'level'
12. for page in page_list:
13. list = get_links (page)
14. node = gv.get_node (page)
15. node.attr [ 'fontsize' ] = "% i" % (MIN_FONT * 2 * (depth - i))
16. for link in list:
17. verbose_message (page + "=>" + link)
18. gv.add_edge (page, link)
19. temp_list.append (link)
20. page_list = temp_list
21. temp_list = []
22. return gv

* This source code was highlighted with Source Code Highlighter .

results

Article "Mathematics" with 4 levels of nesting

Article "Habrahabr" with 5 levels of nesting


Others:
Socrates
Habrahabr 3 levels

Other examples

Script itself