The Grand Illusion
A week or so ago, I was making airlines reservations online – rather I was attempting to do so, but found I couldn’t because my computer wouldn’t let me get beyond the first screen or so at the Delta website, claiming that the Delta website’s security certificate had expired or was not valid. This had happened to me once before, because the date on my computer was wrong. So I checked my computer. No problems that I could find. Then I tried the other computer. Same results. I called my wife at her office. She tried on her work network. The same results. I called Delta. The first representative insisted it was my computer, and then I got disconnected. I tried Delta’s technical support line, waited, and got disconnected.
I waited an hour and tried Delta again. This time the representative actually knew about the problem and informed me that the tech team was working on it – and agreed to ticket me at the online price.
But my question is: How on earth could the IT staff at one of the world’s largest airline systems, a system that depends heavily on website bookings, EVER let their website security certificate get close to expiring? Or was this just the result of hacking? I don’t know that I’ll ever know, but when I talked to one of my daughters, who used to run the IT division of a major chemical company, she informed me that all too many companies have IT divisions that often tend to ignore or postpone the routine “necessities” – until they become a crisis. Of course, one of the reasons she was successful was because she didn’t allow that sort of thing to happen.
I’m certain that tracking security certificates is not the most exciting of IT tasks. Nor is the business of methodically checking to see what holes may have developed in a website’s security, but both are vital. Just last month, the state of Utah discovered that its Medicaid/Health database had been hacked, and the hackers had access to the addresses of 800,000 people and the Social Secuirty numbers of more than 150,000… and the initial investigation concluded that “laxity” and failure to follow procedures for handling data were the principal causes.
I also find it interesting that my readers often get upset over a handful of typos in a 400-500 page book, which is annoying, and which I wish didn’t happen, but does, despite my best efforts and those of editors and proofreaders. But those errors don’t have anywhere near the potentially disastrous impact of software glitches in an economy that has become increasingly dependent upon computers.
In the end, it boils down to one thing. Failure to do what is required, whether what is required is routine, dull, or boring, amounts to incompetence, no matter how skilled the technicians and engineers may theoretically be, and such incompetence leads to huge problems, if not disasters.
Boredom and uninterestedness aren’t a valid excuse. Neither is management failure to recognize the problem, regardless of the “costs.” In the case of books, costs are a valid concern, but when lives and livelihoods are at stake, costs shouldn’t be the primary focus.
The trouble is that managing public certificates is an exceptionally exceptional activity.
It’s partly technical, as the bits of it that you see are all about cryptographic signatures and other such “digital bits” that are likely to make your eyes cross unless you’re a “crypto geek.”
But worse, it’s also partly an exercise of “legal activity,” as the agreements and contracts surrounding digital signatures, which involve somewhat publicly-facing contracts involving the set of signators, also smell “legal.”
The set of people that are into both things (e.g. – managing public legalities for companies, and geeking around with operations of servers) tend to be part of the “null set.”
So while you’re totally right about the importance, finding someone prepared and able to actually be responsible about the digital signatures is a surprisingly difficult task.
I don’t think it’s about cost; it’s about the notion of there being someone suited to have attention to the matter at hand.
And that’s a more difficult matter than people seem able to recognize.
One of the other news stories of the last week was where an Air Canada pilot crew wound up injuring some passengers due to fatigue. The copilot misperceived Venus as being an oncoming plane, disoriented from a nap that went TOO long, and dove towards the Atlantic, dislodging passengers and crew. “Suck it up, be strong!” is NOT a solution to the problem. The solution to pilot fatigue requires analyzing and acting on causes of fatigue.
Managing pilots properly evidently involves setting policies surrounding their sleep patterns, which is pretty unintuitive.
I’m not sure there are “intuitive” answers to the digital certificate problem, either.
Christopher Browne is quite correct in that updating digital certificates is an exceptional activity. I will also mention that it’s exceptional in a chronological sense. Unlike well-practiced activities, like creating new user accounts or resetting forgotten passwords, the certificates expire after many years, necessitating a very rare series of activities.
It’s quite possible that the original IT manager, who was involved in purchasing and installing the certificate four years ago had set a reminder in his calendar; but chances are he is now working in a different position, if not a new company.
Certificates are a relatively low-cost, low-impact resource, and it’s certainly possible that nobody pays attention to them while they are working. To make it worse, many monitoring tools that would alert people on website failure do not pay attention to an expired certificate — it’s actually your browser, Mr. Modesitt, designed to be “suspicious” for very good reasons, that is preventing you from casually accessing a site with an expired certificate. The site itself is working fine — not that I would ever recommend giving your credit card info in such a situation.
No excuse, really, for lack of organization and monitoring, but perhaps at least a partial explanation.
Hi Lee,
I just wanted to say that even though I don’t comment much here, I’ve really been enjoying your blog. I’m a long time fan, having been raised on your Recluse novels, and it’s great to see that in addition to crafting great fiction, you also write such a fine blog.
Thank you.
Suffice to say it’s sadly common. I work for a major creditor in Fraud Investigations. We see pretty much every breach that happens as we try to play damage control and keep our cardholders free from fraud charges.
We’ve found about 8-9 out of 10 breaches happen because a merchant hasn’t patched their software, or haven’t changed the default password on their security software.
So I wish I could say I was shocked that Delta would have such a thing happen, it’s no different from what helped cause the TJX hack a few years back.
Unfortunately, most places are still in the “the magic boxes work well enough so we’ll trust the weird guys who make ’em work are doing their job” mindset. Until they get a major failure at a bad time. My employer discovered they needed a senior level IT manager when a few VIPs were unable to connect to the network despite the presence of an IT department rep to facilitate exactly that… My only hope is that they don’t promote from within, because the likely candidates lack the actual management ability.
And that’s one of the big challenges, as you alluded to in the Time God books. Finding someone who can handle the tech side (at least well enough to wrangle the ones who do the work, which is way too much like herding cats crossbred with sharks, with the attention span of a bluejay) and the management side of things.
Responding to Jim S… In my experience of IT, technically-aware IT managers rarely get promoted to senior rank. Often the “senior level IT manager” who is recruited is often just very good at self-promotion and has long left his technical roots behind, but may be a good manager if he can inspire his technical subordinates.
It is a bit like the MBA seminars I attended : not everyone is good at everything but the best teams are formed of people who do not get on as they stretch each other. So a good IT manager needs techical people on his team even if they irritate him.
“Never ascribe to malice that which is adequately explained by incompetence.”
– Napoleon Bonaparte
(note, some say Mr. Hanlon said it first… no, he just made it web-popular and now calls it Hanlon’s Razor).
I would add ‘laziness’ and a culture of chronic procrastination to this. And then it is a crisis when it comes to a senior person’s attention.
This was due to incompetence, not hacking. But they still got your business.
Security holes are however a different matter. They’re not due to forgetfulness, but are the consequence of many bad design decisions heaped on top of each other. Because security is not highly visible, customers do not make decisions based on it, and most companies invest little effort into it. Your boss wants to ship something now, “of course it should be secure”, but have you added this feature yet?
Doing the job properly would require better foundations, better tools, and better programmers. It is quite ridiculous that the government creates new laws (CISPA) and grows new departments to ensure “cybersecurity”, when the problem can only be solved by writing secure code. Instead corporations save maximize the “productivity” of their programmers by outsourcing to India, or employing kids with little experience.
They got my business because unless I want to drive over 200 miles one way, there’s no other airline I can fly.
Precisely. As long as the cost to you of putting up with their bad service is less than the cost of switching (be it not travelling, or travelling in another way), they have no direct incentive to improve their service. And that fits into what Kathryn said below: as long as a business believes that the cost of losing business to bad IT is lower than the cost of funding IT adequately, IT will be underfunded.
LOVE All YOUR WORK. “handfull of typos in a 400-500 page book” . This is off topic, but letting you know that I recently purchased ebook (combined forever hero,dawn of a distant earth,the silent warrior) “handful of typos per PAGE” SOMETIMES 20!!! kobo books(chapters ebooks in canada) Loved books anyway but was angry for you. That many errors was like dragging a corvette through the mud. Ebooks usually have more typos but this was truly unbelievable on your publishers part. Not looking for refund. Wouldn’t part with it even with errors. Your work is that good. Have read all your work I can find and find it not just entertaining but thought provoking about real life (politics,society etc.)Really looking forward to “Princeps”. Love how you create strong females in your work. Good Luck,
I suspect that the answer may be neither incompetence nor boredom. It may well be overwork. I left IT for the field of environmental remediation several years ago, but in both fields I see the same thing – we want our people doing something that brings in money 100% of the time. If you can plan your work perfectly, that works. Unfortunately in real life unexpected things occur, and suddenly there’s enough work that needs to happen now to require 150% of your people’s time. And yes, that means they will work overtime, but often it also means that less critical things get done later, and the certificate update was probably viewed as less critical. A significant portion of the customer base will just click past the warning and buy anyway.
Whether they’ll click past the warning depends on the systems. My wife’s work system won’t allow it unless you’re a system administrator.
Responding to Joe on outsourcing to India. I worked for a large multinational and this became the modus operandii. All software went that route but we soon discovered that there was a small economic problem : the daily rates were far less, but the quality management processes were not available for inspection and it turned out that the suppliers employed far more people than we expected. The overall quality and cost turned out to be about the same or even less. Over time the daily rates crept up so that employing offshore designers as contract cost more than local guys. I think the ride is turning though.
As a person who did upgrading SSL certificate for a website in a subsidiary of a large S&P 500 company, it is indeed an exceptional activity, which occurs so infrequently, that usually there is no established process for it.
In my case I went as far as having the very last screen of “Getting an real SSL certificate” ready, with only details missing being the credit card details, but then apparently purchasing SSL certificates was not on a list of approved expenses for the corporate credit cards. After assuring my boss that there is no way I can make the transaction to say it was a business dinner, there been a multi-month process of going through the proper channels, creating a purchase order, and such to complete the $250 dollar transaction.