McAfee ePO VirusScan DAT 5958 and W32/Wecorl.a

[Michael McNamara]

Here's a story for everyone out there... as you might or might not know by now there was a significant issue today with a virus signature update put out by NAI McAfee for their VirusScan product. DAT 5958 falsely identified svchost.exe on Windows XP machines as being infected with W32/Wecorl.a In some cases VirusScan tried to delete and/or quarantine svchost.exe and then initiated a reboot. On the machines where VirusScan was able to quarantine (move) svchost.exe the machine was left almost unusable after the reboot, no network connectivity, no NetLogon, etc. The machines where VirusScan was unable to quarantine svchost.exe ended up in a infinite reboot loop as VirusScan would continually try to quarantine the file, fail and then reboot only to start all over again after the reboot.

We have ~ 6,000 desktops in our organization but thankfully the problem hit my personal desktop right as I sat down at my desk during the lunch hour. We were quickly able to jump on the problem before it become a really large headache by shutting down down our repository servers and ultimately deleting the 5958 DAT. We then redeployed DAT 5957 through the ePO console. Unfortunately we were left with about 30 machines that needed manual intervention. For those that still had svchost.exe present in C:\Windows\System32 we simply aborted the shutdown (shutdown -a), loaded Super DAT 5957 and then rebooted the machine. For those machines that had quarantined svchost.exe we had to restore the file from the McAfee VirusScan console, load the Super DAT 5957 and then reboot the machine.

I don't think I ever got a chance to eat lunch... oh well!

[Michael McNamara]

Here's the acknowledgement from McAfee;

http://siblog.mcafee.com/support/mcafee-response-on-current-false-positive-issue/

I'm really taking exception to some of the text they used;

"Our initial investigation indicates that the error can result in moderate to significant issues on systems running Windows XP Service Pack 3."

It's a fact within my organization that any Windows XP SP3 machine and received DAT 5958 was essentially rendered useless. There was no automated way to recover from the problem, system administrators had to manually fix each and ever device in-person. When you only have a dozen field engineers and over 30 facilities with over 6,000 devices the possibility of physically reaching all those devices in any reasonable time is non-existent. Thankfully we caught this very early, perhaps only through luck because I was one of the first users.

"The faulty update was removed from all McAfee download servers within hours, preventing any further impact on customers. We believe that this incident has impacted less than one half of one percent of our enterprise accounts globally and a fraction of that within the consumer base."

Where is McAfee getting those numbers from... their website couldn't even handle the crush of users desperately trying to find a solution flooded the site. Forget about actually talking to someone and opening a support ticket, numerous people documented waiting on hold over 90+ minutes.

There's not enough humble pie in that announcement for me... I guess we'll need to test DAT updates now internally, great another task to throw under change control with yet more unless paperwork and bureaucracy.