18. common damn sense - oh mcafee, you poor bastards

mcafee. holy damn. can i ask, was pcp involved? was that it? was your damn dev team high on the dust?

this outburst was real - actually said out loud while reading the morning news in my office concerning mcafee security and their recent blunder.  there's dropping the ball, then there's what you're about to read - tantamount to dropping a wrecking ball through the ground from a hundred feet up.

most of the time when you get updated virus files, it goes on in the background while your going about your business and you're none the wiser.  95% of the time, this kind of update goes on without any issues.  the other 5% maybe something doesn't quite go properly and the worst that happens is that the update installs itself again.  the remaining (i'll round down to) 0% of the time, an update triggers a false positive, quarantines a system file, renders a machine fully inoperable, forcing a user to operate with pen, paper, and an indelible ink milled from tears and pure rage.

that's what happened wednesday when mcafee - yes, the mcafee, pushed updates for their corporate versions of their security suite around 9am.  the end effect - thousands, maybe millions of computers worldwide on corporate networks were down, cursed to reboot over and over and over, doomed to see any attempt at running applications ending in failure.  this update, which only affected windows xp machines running service pack 3, was so crippling that even huge companies like intel were basically on down time.

i'm going to repeat that - intel, the world's leading PC microprocessor manufacturer, was affected by this.  so if you were, don't feel to bad.

others that were affected were dish networks, the university of new hampshire, and the university of illinois at urbana-champaign to name a few.  commerce halted, labs closed, classes cancelled.  so what the hell happened?

well for a lot of you, even those of you who aren't the greatest of PC pundits, the filename "svchost.exe" may ring a bell.  it's a file that's needed for windows operation, and is sometimes targeted and exploited for certain viruses to hide.  mcafee's update registered a false positive for all affected machines and shoved this crucial file into the quarantine, again, effectively downing a machine.

now comes damage control, with an apology on their "security insights" blog.  but in the immortal words of george costanza, "stuff your sorrys in a sack."  it's way too late for apologies to mean anything now, especially given what their reasoning is.  they admittedly say they were working with sub-par quality assurance procedures, and flat out failed to test the changes in a windows xp sp3 environment.  why this bothers me on a very fundamental level - windows xp service pack 3 is the most common PC operating environment currently being run in the united states, if not the world.  it's like testing a car out on everything but paved road.  personally, being in charge of a relatively small-scale software project and IT department, every change i make is thoroughly tested by my team in a test environment before even thinking about putting it into production.  i'm this careful even knowing that my actions only affect one company and would result in a minor impact - mcafee's changes affect thousands of servers and workstations, and thousands of companies and small businesses.  at this level, especially given what mcafee's software focus is, a hotfix release and a "mea culpa" isn't going to help their case.  they're adding some QA procedures and expanding their "safe" file whitelists.  whoo.

what still surprises me is that they have positive feedback on the issued apology, praising them for quick turnaround times and "hey keep your chin up, microsoft and apple have had issues in the past too."  with all due respect to these posters, politely shut your damn traps and return to watching 90210 before you hurt yourself.  you clearly have the technical IQ of a cashew nut.  the man hours required to repair this in large enterprise organizations and the amount of money in lost time is going to be staggering.  i'm waiting to see the numbers that i'm sure someone is currently tabulating.

now given this near-rant on my part, i'm not heartless.  mcafee - i can definitely appreciate your apology to the public.  i get it, it's a tough industry.  coders that choose to use their powers for evil really do force antivirus / anitmalware software manufacturers to publish multiple updates per day.  newly bought antivirus software is obsolete out of the box.  i feel bad for mcafee, i really do, but when inadequate testing is the issue, they really did bring it upon themselves.

i'm not a mcafee customer, and i'm still upset over this, and feel for my fellow geeks that are on the job around the clock on this one.  i can only imagine how their paying customers feel about a ball dropping of this sort of epic proportions.  and all it took to prevent it was some decent QA processes and some common damn sense.