The Tech Stacks of Influential Sites

Posted on 2012/01/25 by shawn

When considering the development of our projects we naturally attempt to strike a balance
between working with our favorite, proven tools and making use of new technology that seems likely to improve processes or results. To this end I’ve been keeping a lookout for news on the technology stacks that innovative shops employ, hoping to hear about their experiences with sometimes bleeding edge techniques.

I’ll be keeping this dataset up to date, as I encounter good coverage of these matters.

Here are some resources on the subject.

Amazing animations of cell biology – 3D visualization done right

Posted on 2012/01/24 by wolf

3D rendering of human chromosome

One of the great tools enabled by technology is the creation of dynamic, 3D animations of complex data. It’s a technique that can be especially useful in the visualization of scientific information to illustrate and illuminate phenomena that are beyond human perceptions and/or even beyond the physical limits of our instruments.

Drew Berry, a biomedical animator, creates amazing and powerful motion visualizations of the chemical workings within our cells, that perfectly illustrate how useful computer graphics can be for education and scientific research. In a recent TED talk, Berry showed off some of his best creations in describing DNA and cell chemistry at nanometer dimensions, invisible to us because they are smaller than the dimensions of light waves. Continue reading →

Infographics and Story: The Balancing Act

Posted on 2012/01/24 by wolf

There is no question that an important part of creating compelling data visualizations is in being able to find and highlight the interesting stories within that data. When this is done right, data moves closer to becoming knowledge. However, the creative act of selecting and presenting graphical stories is a powerful one; it is also one in which it is possible to perpetrate the most grievous errors in misinformation (at best) or lies (at worst).

Infographics are definitely getting hot on the web these days. The amount of available data sloshing around the internet for free is terrific, and the tools to create professional graphics are everywhere, and so predictably graphic abuses are on the rise. Ever growing numbers of confusing mash-ups for entertainment, advertising, and edification are popping up, and this can’t help but be causing many to be led astray. And it is going to get worse: much, much worse. The temptation for bloggers, corporate websites, and so on to make ‘good’ content (interesting, sticky, popular) in SEO terms (after all everyone wants Google attention and social media interest in the hope of going viral), is awfully big. And the word out there is that infographics get links, get tweeted, get posted, get emailed… and for those interested in building their website, the motivation to get into the infographic business is irresistible. Continue reading →

World Energy Map Data Wrangling: relational to flatfile to topicmap

Posted on 2012/01/09 by shawn

We needed to be able to stop waiving our arms around while talking about “knowledge visualization” and be able to point to something real. It also seemed like a great opportunity to try out some software too, in particular the Tornado web server, the Mappa implementation of Topic Maps for Python, and interactive graphics using HTML5 open standards, in particular SVG, AJAX and Javascript rather than Flash, long a proprietary bêtes noire, now thankfully on its way out.

I remembered orinoco‘s lovely Find the Country game (which I had earlier made an OLPC Activity out of) and kunuk‘s further work making the SVG world map nicely zoomable, so it was clear what to start with on the tech front. Wolf found some great global energy data from the UN and had a vision for a punchy look for the site too.

There is a vast amount of legacy data in flat files and when most people have some data they’d like to put shape to they turn first to the venerable spreadsheet program, not relational databases or some frisky new knowledge management app that lets them collaborate semantically — hmm, where are those, anyway? :-) So, flat files are not going away anytime soon and besides, they are the lowest common denominator for structured information, so we’d best have some tools for upgrading them semantically, no? So it seemed appropriate to make a tool to renormalize this flattened relational data so it can be processed semantically.

What do I mean by flattened relational data? It is a pattern of flat file organization that is encountered when relational data is denormalized for export. Here’s an example. This is the very small sample from the original data:

Note the duplication of data in ‘Series Name’ and ‘Country Name’ columns. For example, every appearance of “Energy production (kt of oil equivalent)” — after the first one has established that as the human readable expansion of the code “EG.EGY.PROD.KT.OE” — is redundant, and a violation of the First Normal Form (1NF) of the relational model.

So the data format above is a ‘flattened relational’ version of a relational database which looked like this:

So using TDD (test driven development) I wrote a tool flatfile.py for reading CSV files which contain flattened relational data and creating an in-RAM representation in 1NF. Then wrote another tool called flatfile_to_ltm.py to convert such documents to Linear Topic Map (.ltm) format, to facilitate import into Mappa. Why .ltm? It is arguably the most human writeable topic map file format Mappa supports and since I’m all about full round trip knowledge management, (ie being able to edit knowledge either graphically or ~~in emacs,~~ sorry sorry… textually) it seemed to be the way to go. Here is a sample.

What are the advantages? Well, brevity; ease of data update; ease of querying; lowered cognitive load dealing with the data; and much greater flexibility in the use of the data. For example, in http://mapdemo.ponderate.com the pop menu is populated by the “Energy Statistic” table.

Human Writable Knowledge File Formats

Posted on 2011/11/25 by shawn

I would like to be able to quickly and efficiently take notes and express myself in machine understandable knowledge, without having to fire up some big knowledge management app. I’d like to take notes during meetings and capture: people, organizations, projects, prospects, products, technologies, needs, opinions, quotations and whatever else I realize at the time needs to be ontologized on the fly. Is there a file format which is easy and efficient for a (pretty nerdy yes, but still merely) human to read, write and alter flexibly connected knowledge? By knowledge I am meaning either Topic Maps or some sort of Semantic Web formalism, such as OWL or RDF. The candidates in TopicMap land appear to be: CTM, Linear Topic Map, JTM or AsTMa. XTM is excluded right off since (being XML) it does not satisfy the ‘efficiently’ requirement because of the bloatedness of tag-based markup.

Over on the SemanticWeb(tm) continent of the knowledge management world the N3 and Manchester OWL are the chief human writable syntaxes.