Last Thursday, the stock market took its single biggest one day drop in its history, somewhere slightly over a thousand points, as measured by the Dow Jones Industrial Index. While the market recovered sixty to seventy percent of that drop before the close Thursday, the financial damage across the world was not inconsiderable. Did this happen because Greece is still close to a financial meltdown, or because economic indicators were weak? No… while the leading cause or precipitating factor may have been a typographical error – a trader entered a sell order for $16 BILLION of exchange futures, instead of a mere $16 million, there are a number of other possibilities, but the bottom line [literally] was that, whatever the cause, all the automated and computerized trading engines immediately reacted – and the market plummeted. Later, the NASDEQ canceled a number of trades, but that was long after the damage had been done.
From the Terminator movies onward, there have been horror stories about computers unleashing doomsday, but the vast majority of these have concerned nuclear and military scenarios – not world economic collapse. While I don’t fall into the “watch out for those evil computers” camp, I have always been and remained greatly concerned about the growth and uses of so-called “expert systems” – in all areas of society, largely because computers are the perfect servants – they do exactly what their programming tells them to do, even if the result will be disastrous.
For example, Toyota is now having all sorts of problems with runaway acceleration. When this first occurred, my question was simple enough: Why didn’t the drivers either shift into neutral or turn off the ignition. Apparently, it turns out, at least some of them may not have been able to, not quickly, because they had keyless ignition systems. Yet the automakers are talking about cars that will be not only keyless but also totally electronic, that is, even the shifting will be electronic and not physical/manual. And if the electronics malfunction, exactly how will a driver be able to quickly “kill” the system? Let’s think that one over for a bit.
President Obama and the health care reformers want all medical records to be electronically available, both for cost-saving purposes and for ease of access. The problem with that kind of ease of access is that it also offers greater ease of hacking and tampering, and, I’m sorry, no system that offers the kind of ease the “reformers” are proposing can be made hacker-proof. The access and security requirements are mutually antithetical. Years ago, Sandra Bullock starred in a movie called “The Net,” and while many of the computer references are outdated and almost laughable, one aspect of the movie was not and remains all too plausibly real. At least two characters die because their medical records are hacked, and changed. In addition, national databases are manipulated and identities switched. Now… the computer experts will say that these sorts of things can be guarded against… and they can be, but will they? Security costs money, and good security costs a lot of money, and people use computers to cut costs, not to increase them.
As far as economics go, now that an “accident” has shown just how vulnerable securities markets are to inadvertent manipulation, how long before some terrorist or other extremist group figures out how to duplicate the effect? And then all the programmed trading computers will blindly execute their trades… and we’ll get an even bigger disaster.
Why?
Because we’ve become an instant-reaction society, and electronic systems magnify the effect of either system glitches or human error. Those programmed securities trading computers were designed to take advantage of market fluctuations on a micro if not a nano-second basis. For better or worse, they make decisions faster than any human trader could possibly make them – and they do so based on data that may or may not be accurate.
We’re seeing the same thing across society. Today’s young people are being trained to react, rather than to think. Instead of letters or even email, they use Twitter. Instead of bridge or old fashioned board games like Risk or Diplomacy, they prefer fast-acting, instant reaction videogames with a premium on speed. More and more of the younger generation cannot form or express complex concepts, even as technology is taking us into an ever more complex world. Business has a greater and greater emphasis on short-term gain and profits. People want instant satisfaction.
The societal response to the increase in speed across society is to use computers and electronic systems to a greater and greater extent – but, as happened last Thursday, what happens when one’s faithful and obedient electronic servants do exactly what their inputs dictate that they’re supposed to do – and the result is disaster?
Do we really want – and can our society survive – a world where a few high-speed mistakes can destroy more than a trillion dollars worth of assets in seconds… or do even worse damage than that? Not to mention one where thinking is passé… or for the old fogies of an earlier generation… and where all that matters is instant [and shallow] communications and short-term results that may well result in long-term disaster.
Daniel Suarez has recently released a couple “techno-thriller” books about some of the problems new to the information age that most people don’t think about. They books are fictionalized or course and push the bounds of realism, but there’s enough truth there to scare anyone with a little insight into the state of most computer networks.
According to Suarez the real dangers aren’t from hackers breaking into government or public networks, but the amount of damage to an information-age society that could be done from a coordinated attack on private networks. Public systems at least have an apparent mandate for security, but there is no government oversight over the quality or security of private networks – no matter how much financial, personal, or private information passes through them.
You know I had been reading the Eternity Artifact around the time when this economic disaster happened and I couldn’t help but remember the dialogue you wrote between the professor and his students discussing the dangers of high-speed, instant reaction communication, and how it limited understanding of the particular situation. It seems like this situation closely resembles the one you described in your book. It’s too bad that these financial institutions didn’t see or ignored the risks that this type of instant reactionary program would have, or how the programs could play off each other into an artificial disaster. At least, that’s how I understand it happened.
All information security is ultimately only as good as the people who use it. I work in healthcare and can state for a fact that we can slap all the software measures we want on our National Health Record, but all it will take is some federal numbskull employee — they are legion — to botch things. Give away codes and passwords to the wrong people. Erase, switch up, or confuse records. Etc. No patient, especially those going into surgery, should allow themselves to be treated based solely on an electronic record. Hospitals already make too many mistakes. I see it only getting worse.
John Barnes explores — in his Million Open Doors universe — the ramifications of human society become totally dependent on artificially intelligent computers. They rely so utterly on what the computers are telling them that, at least in Barnes’ future history, humanity must forever be on the lookout for AI conspiracies which seek to subtly or not so subtly destroy humanity because the AIs are smarter, faster, and tired of being the “servant,” thus they attempt to manipulate the human factor in all kinds of clever, nasty ways.
Even if we never achieve a sentient computer, human error or deliberate sabotage, replicated at computer speed, can do all sorts of damage. The movie SNEAKERS is old, and took certain liberties with technology, but it made a great point in that many of our vital systems and infrastructures are computer-controlled and computer-monitored. What happens when those computers are tampered with, especially subtly? When will we know to ignore what the screen tells us, and use our own brains and eyes?
“And if the electronics malfunction, exactly how will a driver be able to quickly “kill” the system? Let’s think that one over for a bit.”
Ctrl+Alt+Delete but that doesn’t work all of the time.