Failure of the Therac-25 Medical Linear Accelerator

I assume that most people (like myself) have at least a healthy fear of the doctor, dentist, surgeon, etc. But it's unlikely that our biggest concern is the reliability of the tools they use. More often we fear the mistakes that could result from what we perceive to be "human error," (or we just fear the pain - no surprise there).

But the Therac-25 accidents show us that this "human error" is more or less ingrained in all kinds of systems and processes, not just those directly controlled by an operator.

Background

Medical linear accelerators are particle accelerators designed to kill tumors and cancerous cells. The concentrated beams of energy are intended to reduce the amount of harm done to surrounding (healthy) cells. The electron beam is converted to X-ray photons using a filter to reach deeper tissues, but requires a much higher intensity beam due to the processes inefficiencies.

In 1975, the Atomic Energy of Canada Limited (AECL) developed a new "double pass" concept for electron acceleration. Accelerators using this concept required much less space and cost less to build than previous models. The Therac-25 (shown right) was the third generation of these accelerators, boasting the most compact design and ease-of-use to date at that time.

The Incident

The Therac-25 was involved in at least six accidents between 1985 and 1987. The patients received extreme doses of radiation (15000-25000 rads - 200 rads was a normal dose) during their treatments, resulting in either severe injury or death.

The most well-reported of these incidents occurred at the East Texas Cancer Center (ETCC) in March 1986. During treatment, the operator made a quick correction of an input mistake which had selected X-ray mode instead of electron mode. The quick correction had caused the X-ray mode to remain configured but without the required filter (or "target") to produce the X-rays. After its initial activation, a "Malfunction 54" error appeared. The documentation on this error indicated either underdose or overdose. Since such quirks in the machine were typical and documentation on the error was minimal or missing, the operator chose to override the error, causing the high intensity beam to activate a second time.

A simple picture of the two modes of operation.

The patient, Ray Cox, said he felt like he had received two distinct electric shocks, one on his back and the other on his arm after sitting up from the operating table. The physician who examined Cox after the incident suspected it to be nothing more than electric shock, and he was sent home. Later, he was admitted to an emergency room after coughing up blood. His injuries led to paralysis and death five months later.

The Cause

Several software coding bugs were found to be the direct source of the "Malfunction 54" error and others (such as incorrect or unreported dosing) generated during the accidents. However, the primary cause of the six reported accidents was contributed to bad software design and development practices. Proper review, testing, risk management, and reliability monitoring could have caught the errors associated with the buggy code, and adequate documentation could have prevented error on the part of the operator.

In the case of the ETCC, we can see that there was negligence on the part of the machine, the operator, AECL, and the clinic. The machine failed when the operator made a quick fix to an input mistake, which caused race conditions in the system leading to the X-ray beam being in place instead of the lower-intensity electron beam. The operator was at fault in assuming the malfunction error was just another machine quirk and overriding it without further investigation. AECL did not provide documentation that informed the operator on the details regarding the malfunction, leading to this poor decision. In addition, the clinic allowed the procedure in a room with no functioning audio or video for communication between the patient and operator.

Lessons Learned?

Accidents are seldom simple, especially those related to software/computer engineering problems. Most involve multiple levels of failure. In the case of the Therac-25, there are a number of engineering screw-ups that could be partly to blame for the injuries that occurred. These include deficiencies in documentation, quality assurance procedures, development practices, coding, design complexity, testing, safety procedures, and associated fail-safes.

But this doesn't even include the institutional errors and overconfidence that were also to blame.

Case-in-point: the day after the first ETCC overdose, the Therac-25 was shut down for testing, but the error could not be reproduced. The AECL personnel and machine operators did not believe the complaints, confident an overdose from the machine was impossible. Less than three weeks later, after checking calibration and the potential for electrical discharge, the machine was put back into service. They were convinced that nothing was wrong…

Decades later, it is one of the leading case studies for software engineers, and a horror story for everyone else.

For more reading, check out these studies on the subject:

http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/THERAC25.html

http://courses.cs.vt.edu/cs3604/lib/Therac_25/Therac_1.html

http://cda.morris.umn.edu/~mcphee/Courses/Readings/Therac_25_accidents.html

http://staff.washington.edu/jon/pubs/safety-critical.html

Image Credit:

Southern Illinois University Edwardsville