MEASURING SYSTEMS SAVVY
/
I've been talking about systems savvy, but the rubber needs to hit the road. How do you measure it? To help people develop systems savvy you need to know how to assess their starting and progressing competency. If we can't measure it, we can't effectively develop training, hire people who have it, or track its organizational impacts.
Definition
Systems savvy is the "ability to grasp the capabilities of a technology and how that technology might be meshed with organizational practice. People with systems savvy understand that technologies and practices are intertwined — and they know how to make adjustments to both the technology and the practice to effectively weave them together."
Systems savvy is a bit more than systems thinking (For example, Senge's The 5th Discipline). Systems thinking is ability to see the whole and thereby use the leverage of small changes to make improvements. Systems savvy includes understanding how to intertwine the technology, organizational, and people components for better performance -- not just focusing on one small change, but rather more on overall design. Complicated ideas. Complicated measurement. But we do have some foundations to build on. M. Frank has developed measures of capacity for engineering systems thinking. The topics measured include:
Desire to work with systems and to ‘love’ working on the systems level
Understanding the synergy of the system
Understanding the system from multiple perspectives
Not getting stuck on details</li><li>Interdisciplinary knowledge
Learn or analyze the customer’s or market’s needs
Perform engineering & economic optimization
Other measurement for systems thinking focuses more on the dynamics
Sweeney & Sterman present simple problems such as graphing the contents of a bathtub over time given how much is flowing in and flowing out. This gets at whether people understand system concepts like feedback, delays, and stocks and flows.
Measures of intelligence also can provide background for measuring systems savvy. In a 2006 paper, Hedlund, Sternberg (an expert on measuring tricky things like tacit knowledge), and their colleagues describe their creation of measures of "practical intelligence." They note that:
... individuals who effectively solve practical problems are able to recognize that a problem exists, to define the problem clearly, to allocate appropriate resources to the problem, to formulate strategies for solving the problem, to monitor their solutions, and to evaluate the outcomes of those solutions. Furthermore, in order to understand the problem in the first place, individuals need to be able to filter relevant information from irrelevant information, relate new information to existing knowledge, and compile information into a meaningful picture. The effective use of these skills to solve practical, everyday problems can be viewed as an indicator of one’s practical intelligence.
Their measures of problem solving skills were based on the solutions provided to a variety of business scenarios (though they were designed to be answered without business background) and then the open-ended solutions were rated by business school alumni and current students on: (a) time requirements, (b) realism, (c) accuracy and sufficiency of information, (d) prerequisite knowledge or experience, and (e) types of skills/abilities addressed.
Creating Measures of Systems Savvy
Measuring systems savvy would include a similar set of steps following Sternberg et al.: Approach organizational leaders with the request to identify people with clear systems savvy (using the definition given above). Ask the identified "savvy" people to describe a situation that required them to use systems savvy. Have them describe what they did and why it involved systems savvy. Have them describe what a novice or person without system savvy might have done instead. This first portion provides the basic scenarios and some better and worse responses to the scenario.
The next step is to have other experts help you create additional possible responses to the situation. The experts are asked to create responses that indicate high and low levels of systems savvy. Each of the responses is then weighted (again using experts) to create the score for choosing the particular response. In the case of systems savvy, we will need to be sure that the responses include the possibility of only focusing on technical or organizational solutions (lower scores), as well as responses that intertwine technical and organizational possibilities in sophisticated ways (higher scores).
The scenarios themselves should focus on initial analysis tasks (how to get a clear picture of the organizational and technical context), problem solutions, and evaluation of results. The validity of the measurement tool is initially tested by approaching still more experts -- and now also novices -- and having them select (via multiple choice) responses to the scenarios. The results should find the identified savvy experts scoring significantly higher than the novices. If so, your measurement tool is ready for the open road.
Measuring Systems Savvy
I've been talking about systems savvy, but the rubber needs to hit the road. How do you measure it? To help people develop systems savvy you need to know how to assess their starting and progressing competency. If we can't measure it, we can't effectively develop training, hire people who have it, or track its organizational impacts. [photo flickr.com/photos/ideonexus
Systems Savvy is the "ability to grasp the capabilities of a technology and how that technology might be meshed with organizational practice. People with systems savvy understand that technologies and practices are intertwined — and they know how to make adjustments to both the technology and the practice to effectively weave them together."
Systems savvy is a bit more than systems thinking (For example, Senge's The 5th Discipline). Systems thinking is ability to see the whole and thereby use the leverage of small changes to make improvements. Systems savvy includes understanding how to intertwine the technology, organizational, and people components for better performance -- not just focusing on one small change, but rather more on overall design. Complicated ideas. Complicated measurement. But we do have some foundations to build on. M. Frank has developed measures of capacity for engineering systems thinking. The topics measured include:
- Desire to work with systems and to ‘love’ working on the systems level
- Understanding the synergy of the system
- Understanding the system from multiple perspectives
- Not getting stuck on details
- Interdisciplinary knowledge
- Learn or analyze the customer’s or market’s needs
- Perform engineering & economic optimization
Other measurement for systems thinking focuses more on the dynamics
Sweeney & Sterman present simple problems such as graphing the contents of a bathtub over time given how much is flowing in and flowing out. This gets at whether people understand system concepts like feedback, delays, and stocks and flows.
Measures of intelligence also can provide background for measuring systems savvy. In a 2006 paper, Hedlund, Sternberg (an expert on measuring tricky things like tacit knowledge), and their colleagues describe their creation of measures of "practical intelligence." They note that:
.. individuals who effectively solve practical problems are able to recognize that a problem exists, to define the problem clearly, to allocate appropriate resources to the problem, to formulate strategies for solving the problem, to monitor their solutions, and to evaluate the outcomes of those solutions. Furthermore, in order to understand the problem in the first place, individuals need to be able to filter relevant information from irrelevant information, relate new information to existing knowledge, and compile information into a meaningful picture. The effective use of these skills to solve practical, everyday problems can be viewed as an indicator of one’s practical intelligence.
Their measures of problem solving skills were based on the solutions provided to a variety of business scenarios (though they were designed to be answered without business background) and then the open-ended solutions were rated by business school alumni and current students on: (a) time requirements, (b) realism, (c) accuracy and sufficiency of information, (d) prerequisite knowledge or experience, and (e) types of skills/abilities addressed.
Creating Measures of Systems Savvy
Measuring systems savvy would include a similar set of steps (following Sternberg et al.): Approach organizational leaders with the request to identify people with clear systems savvy (using the definition given above). Ask the identified "savvy" people to describe a situation that required them to use systems savvy. Have them describe what they did and why it involved systems savvy. Have them describe what a novice or person without system savvy might have done instead. This first portion provides the basic scenarios and some better and worse responses to the scenario.
The next step is to have other experts help you create additional possible responses to the situation. The experts are asked to create responses that indicate high and low levels of systems savvy. Each of the responses is then weighted (again using experts) to create the score for choosing the particular response. In the case of systems savvy, we will need to be sure that the responses include the possibility of only focusing on technical or organizational solutions (lower scores), as well as responses that intertwine technical and organizational possibilities in sophisticated ways (higher scores).
The scenarios themselves should focus on initial analysis tasks (how to get a clear picture of the organizational and technical context), problem solutions, and evaluation of results. The validity of the measurement tool is initially tested by approaching still more experts -- and now also novices -- and having them select (via multiple choice) responses to the scenarios. The results should find the identified savvy experts scoring significantly higher than the novices. If so, your measurement tool is ready for the open road.
