Computers currently are being introduced into the control systems of dangerous processes (such as nuclear power, public transportation, and weapons) without any way to determine whether the associated risk is reduced, the same, or increased. Because analog and mechanical control systems with measurable risk are being replaced by computers, we need to develop procedures that provide the same level of assurance of acceptable risk.
Numerical risk assessments of physical systems usually are derived from (1) historical information about the reliability of individual components and models that define the connections between these components or (2) historical accident data about similar systems. Neither of these assessment approaches apply to software: Historical information is not available, software is usually specially constructed for each use, and random wearout failures are not the problem. Devising probabilistic models of software reliability is an important research topic; they are potentially very useful in software development. But their usefulness in certifying safety is less clear.
The very low failure probabilities and high confidence in these assessments that is required in safety-critical systems require more experience with the software than could possibly be obtained in a realistic amount of time. More important, these models are measuring the wrong thing. Software reliability is defined as compliance with the requirements specification, but accidents most often occur as a result of flawed specifications, i.e., faulty assumptions about the behavior of the environment or the required behavior of the software. Software reliability prediction models assume that it is possible to predict accurately the usage environment of the software and to anticipate and specify correctly the appropriate behavior of the software under all possible circumstances. Both of these goals are impossible to achieve.
Just as probabilistic evaluation may not be the most appropriate way to provide confidence in the proof of a theorem, it may also not be the best way to achieve confidence that software will always do the correct or safest thing under all circumstances. An emphasis on formal and informal verification, analysis, and review may be more appropriate in evaluating a software and system design. We need more research on procedures to identify software-related hazards, to eliminate and control these hazards through design, to apply safety-analysis techniques during software development to provide confidence in the safety of software and to aid in the design of hazard protection, and to evaluate the effectiveness of the analysis and design procedures to assess the level of confidence they merit.
Qualitative risk assessment and assurance techniques need to be developed if government and society are going to continue to allow the use of computers to control processes that potentially affect public safety.