SwE 2642 Professional Practices and Ethics

1
SwE 2642Professional Practices and Ethics

• The Therac 25 Accidents
• September 17

2
Safety-Critical Systems

• In safety-critical systems, the goal is to
  prevent the system from entering an unsafe state,
  a state where harm could occur.
• Some systems have an immediate safe state.
• In Therac 25, removing beam power brings the
  machine to an immediate safe state --no longer
  performing its designed function, but not in an
  unsafe condition
• The problem was it did not call attention to the
  failure

3
Safety-Critical Systems and Safe States

• Some classes of systems do not have an immediate
  safe state.
• EX An aircraft If a flight safety system
  fails while the aircraft is in flight, there is
  no immediate way to transition from flying to
  being parked safely on the ground.
• Stopping the engines of a flying aircraft because
  a problem was detected is not an option.
• In systems without an immediate safe state, the
  entire system must be able to continue in
  operation, safely, until a safe state can be
  reached.
• This may require redundancy in the operational
  portions of the system.

4
Medical Linear Accelerators
Deliver radiation in measured doses to precise
locations for treatment of cancer. The Therac 25
could deliver two kinds of radiation, electron
beam radiation or x-radiation. The Therac 25 was
significantly more compact than competing
machines.
5
The Accidents

• Kennestone Regional Oncology Center, June, 1985.
  (mastectomy, paralysis of shoulder, constant
  pain)
• Ontario Cancer Foundation, July, 1985 (patient
  died of cancer, but would have needed a complete
  hip replacement because of radiation damage)
• East Texas Cancer Center, March, 1986 (death from
  radiation burns after five months)

6
The Accidents

• Yakima Valley Memorial Hospital, December, 1985
  (radiation burns on right hip, chronic skin
  ulcers, constant pain)
• After careful consideration we are of the
  opinion that this could not have been produced by
  any malfunction of Therac-25 or any operator
  erroroverdose by Therac-25 was impossiblethere
  have been no other instances of similar damages
  to other patients

7
The Accidents

• East Texas Cancer Center, April, 1986 (death from
  radiation burns to the brain after three weeks)
• Yakima Valley Memorial Hospital, January, 1987
  (death from complications of radiation overdose)

8
The Accidents
Six accidents, three deaths and three serious
injuries over a period of almost two years. First
lawsuit filed in November, 1985, before four of
the accidents had occurred.
9
The Therac 25
10
How the Therac 25 Worked
For electron beam radiation, a low-intensity
electron beam is shaped and focused by scanning
magnets an ionizing chamber measures the
dose. For x-radiation, a high-intensity electron
beam bombards a beam-flattener that attenuates
the beam, and an x-ray target that generates
x-rays from the electron beam an ionizing
chamber measures the dose.
11
Choice of Beam Type
http//neptune.netcomp.monash.edu.au/cpe9001/asset
s/readings/www_uguelph_ca_tgallagh_tgallagh.html
12
How the Therac 25 Worked

• For positioning the patient, a light could be
  shined from the beam-emitter showing where the
  beam would appear.
• These three modes were accomplished by a
  turntable.

13
Therac 25 Turntable
From Leveson, Nancy, Safeware System Safety and
Computers, Addison-Wesley, 1995.
14
Proximate Causes
Software failures (at least two) Timing
dependencies
15
Causal Factors

• Overconfidence in the software
• Confusing reliability with safety
• Lack of defensive design
• Failure to eliminate root causes
• Complacency
• Unrealistic risk assessments
• Inadequate investigation and follow-up
• Inadequate software engineering practices

16
Lessons Learned

• Safety is a quality of a system, not of the
  software
• Safe vs. friendly user interfaces
• User and government oversight and standards

17
Therac 6 and Therac 20

• Therac 6 and Therac 20 were similar machines and
  shared software with Therac 25
• There were no reported accidents with the earlier
  machines.
• How were they different?

18
Redundancy

• Systems are redundant when there is more than
  one component, any of which can accomplish the
  desired purpose.
• Therac 6 and Therac 20 had mechanical interlocks
  as well as software safety features.

19
Redundancy

• Redundancy requires two or more components, any
  one of which can accomplish the purpose.
• Redundancy requires that the components have
  independent probabilities of failure.
• Implementing redundant safety features is the
  single most effective way to assure safe
  operation of safety-critical systems.

20
The Math of Reliability
Consider a system with two components, A and B,
each with a reliability of 90. The reliability
of the system as a whole will be 0.9 ? 0.9
0.81 81
21
The Math of Redundancy
Consider two redundant components, C and C ',
each with a reliability of 90. The system is
reliable as long as both C or C ' do not fail
simultaneously.
22
The Math of Redundancy
The probability that either system will fail is
1.0 0.9 0.1The probability that both
systems fail is 0.1 ? 0.1 0.01 1This
system is 1.0 0.01 0.99 99 reliable
23
Questions?

• In what way(s) are considerations of quality of
  life and property rights encountered in the
  Therac 25 case?
• In what way(s) are considerations of safety and
  honesty encountered in the Therac 25 case?
• What are the other risks, issues, problems and
  considerations presented by the case?

24
Questions?

• What were the critical points in time in the
  development of therac 25, and what were some
  possible actions, other than those actually
  taken, at each point in time?
• What were the possible impacts of these actions?
• Who were the decision-makers involved with the
  development, production and operation of Therac
  25, and what were their responsibilities?