A recent study by IBM User Interface Researchers at San Jose analyzed the effectiveness of various automated graphs in understanding data. In the paper, “Understanding Users’ Comprehension and Preferences for Composing Information Visualizations,” researchers used crowdsourcing to study how well individuals can answer statistical questions, given a type of graph.
One of the most novel aspects of the study was the implementation of Amazon’s Mechanical Turk interface to crowdsource the visualization experiment. Mechanical Turk is a software interface provided by Amazon that distributes quick tasks, such as image recognition, to actual people around the globe. In addition to performing the experiment, researchers analyzed how effective Mechanical Turk was at generating results. By allowing researchers to pay the low price of $1.50 per survey completed, surveys could be done faster and cheaper than by traditional methods. The verification phase of the study found that the experimental results were in-line with leading visualization researcher expectations, and that the Mechanical Turk platform is a viable solution for future research. The main downside of using Mechanical Turk was that the Turks were understandably more time-focused than traditional research subjects, and often made short or curt responses. In addition, by focusing on a wide, unskilled, general-purpose audience such as the Mechanical Turk employees, the study might not be applicable to specialized industries were certain types of graphs are universally understood and preferred.
In addition to testing the Mechanical Turk platform, the goal of the study was to analyze the effectiveness of automated multivariate graphs (graphs consisting of multiple variables and distributions). The long-term dream of the researchers is automatic generation of graphs by statistical software, so that users may one day be able to build their own charts based on natural language queries. The study found that the ideal graph for the job often depended on the type of data analyzed.
The best all-around graph was the “Crossed Bar” graph. Users found it to be the easiest for analysis of almost all data questions. The “Stacked Bar” graph was found to be the most difficult to understand, except when identifying extrema, or the maximum and minimum values. Since the stacked bar chart lays one bar on top of another, the comparison of maximum and minimum across entire distributions was easiest on this graph.
Colors were found not to matter much in graph readability, while the geometry to was found to be the most important factor in whether or not a graph was easy to read. One of the problems with the study was that all graphs were sized to the same width and height. Thus the Nested Bar and Nested Line graphs were more difficult to read than the Crossed Bar and Stacked Bar charts. If the Nested Bar graph was made a little bigger, perhaps it would have improved performance in a few of the experimental queries.
Although the results of the study may seem common-sense after completion, they do provide a strong statistical basis for future work. Once real data sets become more prevalent and indexed by search engines, it will truly be exciting times when web users will be able to search not just by simple word queries, but by data – dynamically generating charts and graphs to find answers to the questions of life, the universe, and everything.
Written by Andrew Palczewski
About the Author
Andrew Palczewski is CEO of apHarmony, a Chicago software development company. He holds a Master's degree in Computer Engineering from the University of Illinois at Urbana-Champaign and has over ten years' experience in managing development of software projects.
Google+
