Product quality metrics
     
Date   24 February 1997
Leader   Shawn Butler
     
Papers   T. McCabe and C.W. Butler.
Design Complexity Measurement and Testing.
Communications of the ACM 32(12): 1415-1425.

J.A. McCall.
Factors in Software Quality.
General Electric, No. 77C1502, June 1977. Chapters 2 and 3.

M. Sheppard.
An Evaluation of Software Product Metrics.
Information and Software Technology 30(3): 177-88. UK, April 1988.
     
Supplemental Readings   T. McCabe.
A Complexity Measure.
IEEE Transactions on Software Engineering SE-2(4): 308-320. December 1976.
     
Meeting Material   Shawn produced a set of slides on quality metrics, available in Acrobat format.
     
Other Material   McCabe & Associates offer a suite of tools for calculating metrics from your code, like cyclomatic complexity.

Verilog's LOGISCOPE tool suite provides code-based quality metrics that can be tailored to your development group.
     
Notable Remarks   Garlan suggested that these metrics seem to be looking at a macro-level phenomenon through a microscope. Shaw added that it's like trying to derive the gas equations from the three-body problem, the four-body problem, ..., the Avogadro-number-body problem, etc.
   Shawn Butler told how she used these quality metrics in practice at her pervious job. In one instance, she used the metrics to hunt out "rogue" programmers in the group she managed, programmers whose code was too complex and poorly expressed to maintain. The metrics provided a standardized way to judge programmer output. In a second instance, she used the metrics to sift through a large number of unfamiliar legacy systems to determine which were most likely to cause problems. In both cases the metrics were not used as absolute measures but as relative measures to help establish priorities -- which programmers and which legacy systems needed the most attention.
   Several participants noted that these metrics, when applied to generated code, yield very poor ratings. This suggests that improved metrics should be applied to the true "source code" of the system, namely whatever artifacts the developer actually deals with.
   Garlan related an idea from Hamlet on testing: the modules on which to spend the most testing resources are those created by programmers whose modules have been buggy in the past. Is there an implication here for complexity metrics, namely that the people involved -- and not just the artifacts they create -- need to be be part of the measure?
   Gleichauf warned that teaching bad tools (like this session's metrics), especially to the uncritical, is dangerous. Shaw retorted that it's often important for an instructor to give the lay of the land, even if we don't like the way the land lies.
     
Post Mortem   [Butler:] The second and third chapter of McCall's study provided the context for discussing software quality. It was an easy introduction into what is meant by software quality, at least as defined by one source. Another discussion session could be devoted to whether it is complete or concise. Sheppard did an excellent job of summarizing the salient points of several software product metrics. However, his critique of the metrics did left the reader wondering if code metrics, particularly Halstead's and McCabe's metrics, were of any value. Most notable was his comment that Source Lines of Code (SLOC) were more indicative of problems than the more complicated metrics. Most of the discussion centered on highlighting the value that metrics play in detecting potential problems in software products, rather than critiquing a particular metric.
   Follow on topics could include: Quality Frameworks, Metrics for Object Oriented Progamming and Design, and SLOC versus other product metrics. Some participants felt that the McCabe and Butler paper was difficult to follow and would not recommend it, and that the Sheppard paper was a very good selection.

[Butler/DeLine 03/24/97]