The actual problem turned out to be the oxygen supply. Hospitals weren't designed for that many patients on hi-flow oxygen. The Veteran's Administration contracted for trailer mounted oxygen generators that could supplement fixed generation systems. Only around 20% of vent patients are ever successfully weaned off, but you have to try something.

James

Just chiming in to relate: one of the first computer controlled x-ray machines had a buffer overflow—triggered by a system clock event or something that wasn’t caught in testing—that caused the emitter to remain active instead of flashing on and off. Patient died consequently in the misery one would expect.

Therac-25

One of the many problems was that it reused the (bad) software of the previous model, which had hardware interlocks. The 25 had no hardware interlocks.

And that's why at work, we (the software designers and developers) routinely discuss with our hardware designers what damage an intentionally malicious software could do on the given hardware. Of course we don't write malicious software, but the discussion shows where hardware interlocks are needed in case of an error generating the wrong signals.

An example from a real project using a 32-bit microcontroller:

A light barrier should limit where software could switch motors on (a little bit simplified). Hardware should ensure that the motors could be switched only if the photo transistor "sees" light. And the microcontroller should be able to read that signal. The signal from the photo transistor was fed into the motor interlock logic hardware, and in parallel to an input pin on the microcontroller. Looks harmless, right?

Until you realise that the pin could be switched to output mode, that would overwrite the signal from the dark photo transistor. This way, software could bypass the hardware interlock and drive the motors during unsafe conditions, damaging hardware and - in the worst case - could harm people.

Switching the pin to output mode with the right level needs one, at most two write accesses to memory-mapped I/O, each with only one important bit. Half of all possible values would bypass the interlock, when written to two of a few GPIO control registers. A chance of a few addresses in 2^32. Also, damaging the stack in the right way would end in a jump into one of a few routines that write those control registers. And that is much more likely than 1 in 2^32.

The solution is quite obvious: Don't feed the raw signal into the microcontroller, insert a buffer. It could be as simple as an extra transistor, or almost any logic gate. In our case, we used a previously unused output from the programmable logic device implementing the interlock, delivering just the unchanged signal from the photo transistor to the microcontroller, via a resistor that prevents excessive current in case the microcontroller should accidentally switch the input pin to output mode.

Now the worst thing that even intentionally malicious software can do is to heat up that resistor a tiny little bit. The resistor can withstand that for at least decades. There is no way to bypass the hardware interlock any more.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

The 25 had no hardware interlocks.

I imagine the whole thing started something like this: "Hardware interlocks are expensive. And if something goes wrong, the machine shuts down for days until we can get an onsite visit from a certified tech."

Certain Ricoh laser printers also have a fail-deadly "feature". There are sensors in there that detect certain dangerous conditions like "paper jam" and "output tray full" that could result in a fire. Thing is, if you disconnect the switches, the printer thinks "everything is fine", instead of shutting down with an error message...

perl -e 'use Crypt::Digest::SHA256 qw[sha256_hex]; print substr(sha256_hex("the Answer To Life, The Universe And Everything"), 6, 2), "\n";'

On a tangent... I recently learned that the real problem with self driving cars is not to for the AI to safely drive the car.

It's the phase when the human driver takes control over during the ride, and the AI has to anticipate the DWIM of the human while releasing control.

and DWIM brings us back on topic with Perl again ;-)

Cheers Rolf
_{(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery}