DESIGN AND VALUE OF "BIG DATA"

Last Friday I had the opportunity to attend Web 2.0 Expo in San Francisco.  San Francisco transit connections being what they are, I missed some of the opening keynotes, but luckily I didn't miss Jeff Veen's Designing for Big Data. Jeff's presentation echoed some of my thoughts -- we have access to much more data than ever in the past -- how can we gain value from it?  As he noted, we don't want to just "decorate the data."  We want to gain value from it.  His new start up, Small Batch (apparently stealthy given this website), created Wikirank for visualizing and comparing what people are reading on Wikipedia.  The tool provides interesting data in a clean interface with some user control in that they allow you to decide what data to compare:
Compare Wikipedia topics. At the top of the front page, you’ll see a comparison of two or more topics that we’ve chosen. From that section, you can view more detail on the featured topics, share the chart with others, or choose your own terms to compare.
They also ask that you share particularly interesting comparisons with them (and provide an email link). I think this last, user control of data comparisons, is going to be key in how we "design for big data," and eventually in how we gain value from big data.  We have the data.  We are gaining the tools and skills.  Next we need to understand how to extract value from this combination. Quoting from a prior post: Bellamy et al. draw from Card, Mackinlay, & Shneiderman’s book, “Readings in Information Visualization: Using Vision to Think” to suggest that visualizations should increase people’s visual capabilities and amplify cognition. They note that Card et al. outline six visualization benefits:
  1. Increasing the users’ memory and processing resources
  2. Reducing the search for information
  3. Enhancing pattern detection
  4. Enabling perceptual inference
  5. Using perceptual attention mechanisms for monitoring
  6. Encoding information in a malleable medium
I followed that post with one thinking about how teams might gain value from visualization of team data.  Veen's presentation, and his wonderful links back to data mavens like Edward Tufte, push me to think about how big data can provide more general value.  Tom Davenport's ideas around Competing on Analytics are a great background, but we have even better access to data inside and outside of our organizations since his 2007 book. Here are some views into big data.  I'm not saying they all have value, but provide them as a start to some brainstorming. Wikirank (described above) Yahoo Pipes allows normal people to "aggregate, manipulate, and mashup content from around the web" Twitter visualizations ProgrammableWeb.  If you want to go deep, go here. What else should we add?