A Framework for Evaluating Performance Measurement Systems IN BUSINESS

A Framework for Evaluating Performance Measurement Systems

Over the last 20 years or so, established frameworks and reference models, along with specific, lesser- known measurement approaches, APL, IPMS, OPTIMAS (Jensen & Sage, 2000) and others, have populated the business world. Companies have choices when building a BPM. Despite the large amount written about how to measure, much less has been written on what are the key attributes that are desirable in a BPM system. De Haas & Kleingeld (1999) mention seven pre-existing measurement system criteria from other studies and add their concept of coherence (discussed below) to the list to make eight:

1. Controllability
2. Validity
3. Completeness
4. Cost-effective measurability
5. Specificity
6. Relevance
7. Comprehensibility (Van Tuijl)
8. Coherence

In another study, Jensen & Sage (2000) enumerate metric design attributes (goals) and metric set goals and measurement system infrastructure goals.� The metric goals include:

1.       Cost-effectiveness

2.       Strategic alignment

3. Acceptability (buy-in)
4. Usefulness
5. Acquirability and implementability
6. Consistency
7. Accuracy
8. Reliability
9. Repeatability
10. Believability
11. Timeliness
12. Responsiveness
13. Known responsibilities
14. Security

The Jensen & Sage (2000) metric set goals include:

1. Balance across types of metrics
2. Organizational coverage
3. Completeness, minimum overlap
4. Cost-effectiveness
5. Total number, number per measurement area
6. Standardization
7. Documentation
8. Coverage of strategic thrusts
9. Current status and trend measures
10. Communications to staff

The Jensen & Sage (2000) measurement system infrastructure goals include:

1. Automation
2. Repository, communications and other security (access to archival information)
3. Labor hour reduction
4. Information dissemination

As can be seen, design attributes vary from author to author. This paper attempts to enumerate a minimum set of BPM system design attributes (comprised of criteria and factors) and foregoes discussion of specific metric design attributes, metric set attributes or their linkages to a defined strategy. Metric and metric set design attributes can be derived from the BPM system design attributes. In addition, some useful and successful BPM systems are operational in nature and may not be designed to clearly link to and communicate a firm’s strategy. In the interest of minimalism, the criteria and factors offered in this paper are silent on this matter of strategic linkage. Underlying the following design attributes is the notion that BPM systems provide a key component to a firm’s ability to sense and respond to its internal and external environments. Data in them is often tied to key motivational aspects for both the firm and its employees. In addition, the term BPM system refers to the information technology and the human process that interact with the technology. The two are conceived as joined in a symbiotic relationship with each other and hence design attributes must take into account both aspects. Using this biological organism metaphor, this paper recasts the prior design criteria described elsewhere into the following four key measurement criteria (Table 2). In addition, 12 factors that link to these four criteria are discussed (Table 3).

1.       The BPM system should help the firm accurately perceive relevant internal and external phenomenon. These include threats and opportunities, shortcomings in its ability to perceive phenomenon as well as shortcomings in its ability to control its actions (breadth, depth, coherence and predictability).

2.       Measurement information needs to be delivered, processed and acted upon within the time frame needed for market survival (latency: propagation and response).

3.       The BPM system must aid the decision-making process (provability, explainability, believability, communicability).

4.       The BPM system needs to operate self-reflexively and largely below the threshold of the firm's awareness (adaptability, measurability, autonomic).

Table 2. Measurement system design criteria

Breadth	Refers to how much of the total set of activities needed to be measured are actually measured. Breadth needs to be balanced between internal state and activities inside of the firm and activities and items external to the firm such as customers, suppliers, competitors, market conditions, environmental conditions, etc.
Depth	Refers to the unit of analysis. Levels of analysis, or granularity, can include the employee, the workgroup or team, the functional unit, the business unit, the product, the customer, the firm as a whole, the marketplace or the economy at large. BPM systems can and typically do cover multiple levels of analysis.
Coherency	Refers to the how much breadth and depth factors combine together to improve performance. How do lower levels of measurement contribute to higher levels? How do units of measurement at the same level coordinate together to contribute higher levels?
Predictability	Refers to how accurately and far into the future a BPM system can project.
Provability	Refers to how the BPM system can show the relationship between causes and effects. Identifying causes and effects helps managers better understand where (which object) to apply attention.
Explainability	Refers to how easily people in the firm can explain relationships between measurements and how the BPM system functions.
Believability	Refers to how much people in the firm trust the BPM system. Do people in the firm believe what the BPM system is expressing? Data quality and overall measurement trust (reliability, consistency, accuracy) are key components.
Communicability	Refers to how well can people in the firm communicate measures and discuss them amongst themselves?
Adaptability	Refers how easily and completely the BPM system can be altered. Is the BPM system automatically self-changing? How much intervention is required to change it? Is the human component capable of changing?
Measurability	Refers to how the BPM system itself is measured (meta-measurement). Is the BPM system working within normal parameters? What is the quality of service? How effective is the BPM system? Where is improvement in the BPM system warranted? Is it measuring the right things?
Autonomic	How much does the BPM system help the firm self-correct? How much management attention and effort does operating the BPM system require?

Table 3. Measurement system design factors

The measurement criteria are non-gradated; that is the BPM system either meets the criteria or it does not. If anyone of the four BPM system design criteria is not met, the BPM system may not be successful in contributing to the success of the firm or may fall into disuse. The 12 factors are gradated. Individually they vary depending on the constraints inside or outside the firm but collectively they meet the criteria threshold.

Austin & Gittell’s (2002) discussion of the three conventional attributes (performance should be clearly defined; performance should be accurately measured; rewards should be contingent upon measured performance) is unnecessary to explicitely include here. Nor is a discussion of intrinsic, ambiguous or extrinsic and unambiguous metrics/motivations.� The factors in Table 3 relevant to the topic of ambiguous-unambiguous metrics and intrinsic-extrinsic motivations can be scaled to either direction to satisfy the criteria. In addition, causality is folded into the model as a factor, not a criteria, under the notion that it might be possible (albeit remotely) for a BPM system to satisfy all the four criteria without the need for strict causal proof or even causal reasoning. While managers generally intend to do things with a causal framework in mind, the BPM system may not be able to capture (or need to capture) the causal linkages. These design attributes (criteria and factors) make a clear distinction between what managers intend with regard to causality and what the system is capable of detecting.