Technology and Organizations

Measuring Systems Savvy

I've been talking about systems savvy, but the rubber needs to hit the road. How do you measure it? To help people develop systems savvy you need to know how to assess their starting and progressing competency. If we can't measure it, we can't effectively develop training, hire people who have it, or track its organizational impacts. [photo

Systems Savvy is the "ability to grasp the capabilities of a technology and how that technology might be meshed with organizational practice. People with systems savvy understand that technologies and practices are intertwined — and they know how to make adjustments to both the technology and the practice to effectively weave them together."

Systems savvy is a bit more than systems thinking (For example, Senge's The 5th Discipline). Systems thinking is ability to see the whole and thereby use the leverage of small changes to make improvements. Systems savvy includes understanding how to intertwine the technology, organizational, and people components for better performance -- not just focusing on one small change, but rather more on overall design. Complicated ideas. Complicated measurement. But we do have some foundations to build on. M. Frank has developed measures of capacity for engineering systems thinking. The topics measured include:

  • Desire to work with systems and to ‘love’ working on the systems level
  • Understanding the synergy of the system
  • Understanding the system from multiple perspectives
  • Not getting stuck on details
  • Interdisciplinary knowledge
  • Learn or analyze the customer’s or market’s needs
  • Perform engineering & economic optimization

Other measurement for systems thinking focuses more on the dynamics

Sweeney & Sterman present simple problems such as graphing the contents of a bathtub over time given how much is flowing in and flowing out. This gets at whether people understand system concepts like feedback, delays, and stocks and flows.

Measures of intelligence also can provide background for measuring systems savvy. In a 2006 paper, Hedlund, Sternberg (an expert on measuring tricky things like tacit knowledge), and their colleagues describe their creation of measures of "practical intelligence." They note that:

.. individuals who effectively solve practical problems are able to recognize that a problem exists, to define the problem clearly, to allocate appropriate resources to the problem, to formulate strategies for solving the problem, to monitor their solutions, and to evaluate the outcomes of those solutions. Furthermore, in order to understand the problem in the first place, individuals need to be able to filter relevant information from irrelevant information, relate new information to existing knowledge, and compile information into a meaningful picture. The effective use of these skills to solve practical, everyday problems can be viewed as an indicator of one’s practical intelligence.

Their measures of problem solving skills were based on the solutions provided to a variety of business scenarios (though they were designed to be answered without business background) and then the open-ended solutions were rated by business school alumni and current students on: (a) time requirements, (b) realism, (c) accuracy and sufficiency of information, (d) prerequisite knowledge or experience, and (e) types of skills/abilities addressed.

Creating Measures of Systems Savvy

Measuring systems savvy would include a similar set of steps (following Sternberg et al.): Approach organizational leaders with the request to identify people with clear systems savvy (using the definition given above). Ask the identified "savvy" people to describe a situation that required them to use systems savvy. Have them describe what they did and why it involved systems savvy. Have them describe what a novice or person without system savvy might have done instead. This first portion provides the basic scenarios and some better and worse responses to the scenario.

The next step is to have other experts help you create additional possible responses to the situation. The experts are asked to create responses that indicate high and low levels of systems savvy. Each of the responses is then weighted (again using experts) to create the score for choosing the particular response. In the case of systems savvy, we will need to be sure that the responses include the possibility of only focusing on technical or organizational solutions (lower scores), as well as responses that intertwine technical and organizational possibilities in sophisticated ways (higher scores).

The scenarios themselves should focus on initial analysis tasks (how to get a clear picture of the organizational and technical context), problem solutions, and evaluation of results. The validity of the measurement tool is initially tested by approaching still more experts -- and now also novices -- and having them select (via multiple choice) responses to the scenarios. The results should find the identified savvy experts scoring significantly higher than the novices. If so, your measurement tool is ready for the open road.

Where is Work? More on TOP Management

To what extent do we need to adjust our thinking around "going to work"? A recent article by Peggy Stritch and Clark Sept says that Cisco (the Fortune 100 technology company) "is at the beginning of a shift: How working together is now a global activity. How work gets done -- collaboratively.

Our Personal Security Role in a Web-Enabled World: Vigilante and/or Ostrich?

Location aware tools and ubiquitous networks are enabling new activities... or are they just opening up old ones to a broader community?

Elegant Can Beat High Tech

Yesterday was Reid-Hillview Airport Community Day. One of the activities was a tour of the Control Tower. Great experience. Thank you to Vincent and Spencer for taking the time to explain the process that keeps hundreds of flights going in and out safely. Thank you to the rest of the team for letting us observe you at work.

I was surprised by how physical the process is, versus my high tech expectations. Yes, they have access to radar and a huge portion of the work involves radio communication with the pilots going in and out of the airport. But they also make heavy use of those big windows and a unique physical tracking system. They track planes by type, tail number, and request for inbound or outbound route -- by writing the information on plastic "pucks" with a grease pencil, and then physically sorting that puck onto the taxi and runway slots.

We weren't allowed to take pictures, so I'm showing a similar process below using wooden blocks. ATC desk

When I asked about the process, using the plastic pucks versus keeping track on a computer, I was told that sometimes "elegant is best." Great point! The solution is elegant in that the physical blocks trigger sensemaking (in my words) more than a screen version might. They can push a puck slightly out of its track to highlight that more action is necessary. All the members of the team can immediately step in to provide relief given their common understanding of the system. Elegant, green (no need for power or paper), easily visible to all in the room -- good for team visualization. Beautiful approach to a complex problem. Sometimes systems savvy means using elegant, but less high tech systems. Comments appreciated describing other examples.

Sharing Knowledge with Your Network

... and those you want to join your network. My elder niece came home from pre-school one day talking very seriously about making "good choices."  Apparently a little boy in her class had not been making good choices and was sent to a different room. Making good choices becomes more difficult the more choices you have.  I'm struggling with making good choices about my own knowledge sharing. 

I generally share my knowledge via (in order of formality and timeliness): Twitter, Business Exchange, this blog, class, and my academic publishing (pdf of my vita).  Each of these channels ties me to a different network.  The networks overlap to a small degree. Below I present five dimensions to help my clients (and me) make good choices about sharing knowledge with their networks. 

This is another topic related to how we all are becoming systems designers -- we all need to make good choices about how we share our knowledge so that we get the knowledge to the people who need it, when they need it, in a form they can use, and in a way that doesn't overburden them. 

These are decisions based on technology, organizational practice, and people: part of TOP Management. Formality and timeliness are two dimensions I've already mentioned.  For example, I don't tend to post to Twitter about past research unless some new and hopefully interesting thought strikes me.  On the other hand, I don't necessarily expect academic readers to be interested in this week's current event given that the article won't be out for over a year.  To do otherwise would be to ignore my audience's perspective. Content interest is another dimension to consider as you share your knowledge.  I assume that my networks are interested in gaining benefit from managing technology and organizational practice, and innovation more broadly, or they wouldn't be following me on my Twitter/Business Exchange/blog/class/academic networks. 

That said, I do toss in a sailing or golf reference from time to time because it provides context about who I am.  I like knowing a few details about my own knowledge providers gives me background for interpreting their content.  It also gives us more of a social connection if we have the chance to meet face to face.  Kind of like the beer effect without the beer. I'm told :) that detail and depth are issues I should consider as I share knowledge.  References (e.g., Barley, 1986; Weick 1979), as part of the sentence are not as interesting to most people as they are to academics.  (Really, they are interesting to us!) 

Each of my channels provides the opportunity for more and less detail and depth -- either by technological limitation (e.g., Twitter and 140 characters), human preference (e.g., my blog audience's interest), or organizational requirement (e.g., the APA Style Manual). Signaling is my final dimension, so far.  Signaling is how you let your networks know that there has been a new contribution.  Some of the technologies have their own signaling capabilities.  A few examples:

  • Twitter: Network members can decide whether they want email or SMS notification of new posts from specific network members.
  • Business Exchange: Network members can follow specific topics or specific people.  Business Exchange then summarizes new activity on the user's homepage.
  • Blogs: Network members can opt in to to automatically receive new contributions via RSS reader or email.

Ideally, some contributions on one network will be of interest to members of other networks.  Often I post announcements of new blog posts to Twitter, my LinkedIn account (the short message area on the bottom left), and my Facebook account. I give enough information so the people on that network will know if it's worth it to click through or not.  My blog is also tied to LinkedIn's Blog Link and NetworkedBlogs on Facebook.  As long as these networks have limited joint membership, this crossover signaling is ok. The more interconnected your networks become, the more careful you have to be about duplication. 

Duplication is akin to spam. We all have multiple opportunities to provide original knowledge contributions (a blog post, and comment to a blog) and/or to share valuable links. I've provided these dimensions:

  • Formality
  • Timeliness
  • Content
  • Detail/depth
  • Signaling

Do you have other dimensions that will help us all make good choices?

Subscribe to Quick Insights You Can Use

Syndicate content