The Grand Illusion
Yesterday, my wife the professor set out for work… and was back in less than half an hour. Why? Because the power was out at the university, and the reason was that a student who was texting while driving lost control of his car and ran into the electric power distribution transformer. The immediate result was that the entire university lost power for more than half a day, but because the transformer is actually connected to the music building, the university public works crews could shunt power to the rest of the campus, but repairs to the music building’s electric distribution system may take several days, perhaps a week. In the meantime, even without power, the building is effectively shut except for emergency access, because all the locks are electronic.
Texting while driving strikes again, and while this comparatively small accident injured only one driver and shut down a small-to-mid-sized university for half a day, the fact that all the doors are electronically actuated underscores the growing reliance of everything in society on electronics and electric power.
A far larger problem occurred early Monday morning when apparently the failure of a piece of electronic equipment at Delta’s headquarters triggered a power failure and crashed the entire computer system and grounded the airline’s flights for hours, causing more than 600 flights to be canceled and some 3,000 or more to be delayed. Delta said it didn’t know why the power outage hadn’t brought all the back-up systems on line.
And that wasn’t the first airline computer problem. A month ago Southwest airlines had to delay and/or cancel some 2,000 flights due to a computer system glitch.
These kinds of events point out the vulnerability of the United States to power outages and computer system crashes, and, frankly, the fact that it appears that we just might need to be doing more as a society to safeguard and back up both our power sources and our electronic information networks and security systems.
Perhaps some businesses are, but it appears that the cost to Delta from the latest outage/crash is going to cost quite a bit more than up-grading and improving back-up systems would have… and now back-up costs will still be incurred. From what I’ve observed of the local power company here in Utah, they’re still playing roulette with their placement of spare transformers and the like, suggesting that the efforts of electric power system providers leave quite a bit to be desired.
But then, maximizing current profits takes precedence over everything, even long-term profitability – or even survival – just as the immediate pleasure of texting appears to take precedence over safety… and even survival.
NOTE: The latest word from the university is that the damage was so extensive that it will likely cost at least several hundred thousand dollars to replace the equipment, and take six months to a year to obtain a permanent replacement transformer/distributor. In the meantime, the university is operating on a mix of a back-up generator and “temporary” re-wiring.
Texting while driving; idiotic doesn’t begin to reach the level of incompetency that this represents. I understand the reflex action of hearing your phone go off and reaching to see what it is, but, this is without a 2,000 to 3,000 lbs. machine being under your personal control. Nothing more to say.
The issue of dependency on electronic devices, and therefore electricity, is somewhat different. It is not so foolish, as the genesis is often simply the attempt to improve efficiency, which leads to the ability of interdependent systems all being upgraded to a level of speed and efficiency that people alone cannot attain. This sets the stage for a cascade failure when one or more of the systems goes down, and, albeit where there are no systems in place in a parallel set up, there just to take over if a failure occurs. So the question becomes how much to spend to avoid a “single point of failure” by setting up such systems? This is not an easy question, and while I would opine that the shutting down of a school may have been losing a “good bet,” the failures of Delta and Southwest were losing a “bad bet.” I’m just thankful there was not life/safety implications.
On vulnerability, it is of course down to cost and the business model. I doubt Delta could compete with its rivals if their computer systems had high resiliency, as every extra 9 in the 99.xxx availability will increase the cost significantly, if not logarithmically as the 9s increase.
It is a balance, and in this world we humans are increasingly demanding high quality, low cost and safe solutions. This does not work and something has to give.
It’s worse than that. Estimates are that it would take between $100 million and $2 billion to harden the electrical grid against EMP (whether weapons or massive solar flares that WILL occasionally occur). But the legislation has repeatedly died in Congress.
I’m not a fan of activist government, but when a relatively small expenditure can can MASSIVELY improve infrastructure resilience to both natural and man-made threats, it’s a bit odd that it can’t be accomplished.
I’ve always found it fascinating that, generally speaking, the more technologically advanced a method of data storage/processing, the more vulnerable it is.
Additionally, I suspect that many vulnerabilities in technical systems occur because they are not built from the ground up, and with expansion in mind. They are essentially assembled piece-meal with disparate parts often brute-forced in integration. This applies to information systems as well as power systems, utilities, construction and roadways.
The brute-forced integration usually only works when there are humans employed who know the piece-meal system in total. They manage and finesse and subrogate the pieces of the system as needed.
Institutional knowledge should be more acknowledged than it is.
Other issues with technical systems include sub-optimal programming, sub-optimal design, and sub-optimal implementation as a result of non-technical people in charge of technical projects or the (often non-sensical) demands of higher ups that have to be integrated, but are integrated poorly because they often make decisions on a whim and outside the scope of the project.
From what I can tell, people in general don’t look at the larger picture, even those who are supposed to such a politicians or CEOs.
Technology can also improve resilience, but that has to be a priority; and to some degree, it has to be a priority from the beginning to be comprehensive, although single points of failure can usually be retrofitted, if often at greater cost than for having addressed them from the beginning.
Yet technology is a moving target, and a realistic plan also has to include a cutoff at the point that the initial essentials AND hooks for future growth are in it, and leave the rest to future revision, or nothing will ever be deployed.
The saying “Fast, good, or cheap: pick any two” applies, although “lean” project management, with early development of a minimal satisfactory version and subsequent updates, claims to offer all three; but the complexity of actually delivering on that seems seldom enough met; and updates make the most sense with software, and increasingly less sense with hardware as the price of it (and cost of responsible disposal) rises.
Effective planning and troubleshooting are two areas where few are really skilled; many can learn to follow a cookbook, but few can invent excellent new recipes, let alone document them that others might follow.
Color me skeptical regarding most grandiose plans, ESPECIALLY as implemented by government, where lack of accountability and useful collection and application of “lessons learned” provide little more consequence for incompetence or even negligence, than a lateral move that relocates the ineffective individual. Although other hierarchies (and hierarchs) with some similarities to government are sadly notorious for similar behavior. Some have done better, but the attempt to institutionalize success is not a one-shot, but requires continuing effort and leadership to cope with both change and turnover; and few are the hierarchies that retain for generations priorities other than continued existence.