More than a year after Microsoft changed its decades-long security updating practices, enterprises running Windows 7 continue to struggle with the new system, patch experts said today.
"I still see people asking for individual updates, even on the [Windows] 10 operating system," said Susan Bradley in an email reply to questions. Bradley is known in Windows circles for her expertise on Microsoft's patching processes: She writes on the topic for the Windows Secrets newsletter and moderates the PatchMangement.org mailing list, where IT administrators discuss update tradecraft.
Bradley was referring to Microsoft's debut last year of radically-different cumulative updates for Windows 7 and 8.1, a change to the long-established practice of letting customers choose which patches they applied. From October 2016 on, updates for the two older Windows versions were comprehensive wholes, not collections of separate patches that could be selectively applied.
The model, which Microsoft first used with Windows 10, requires that updates not only be cumulative in the sense that they include everything from all past updates, but in that they cannot be broken into parts, as had been the case prior.
When Microsoft announced the changes, and before the new-style updates were issued, experts in Windows patching warned that commercial customers might be forced to decide between patching - and thus keeping their PC environments secure - and possibly breaking a business-critical application or workflow. "There is a real concern that there will be an issue that because we have to keep the business operational, we will not be able to install the update rollup," Bradley said at the time. "As a result, we [will] leave ourselves exposed to risk of attack."
Under the earlier patch scheme, users were able to set questionable updates aside - perhaps for further testing, maybe to give Microsoft more time to quash a just-found bug - even as they deployed all other updates. That isn't't possible under the all-or-nothing cumulative regime.
Nearly 14 months later, enterprises labor to adapt.
"It's elongated the patch cycle," said Chris Goettl, a product manager with client security and management vendor Ivanti, in an interview. He explained that many businesses were forced to postpone all patches, at least on some systems, because a code change included in the cumulative Windows 7 update had broken a critical application.
"We're seeing a lot of customers who have not been able to roll out any updates until a problem was resolved, either by Microsoft or a third-party vendor," Goettl said. "Before, [admins] would say, 'We're going to roll out only the critical updates, and ignore those [labeled] "Important,"' but now they don't have that option. So they ignore all updates on a sensitive system."
And by all, he meant just that: all updates, including succeeding cumulative updates. Because each Windows 7 update now bundles this month's fixes with all those issued in the past, a bug must be fixed before any update - including those from next month, or the month after that, or one six months down the line - can be applied.
Not all systems may be affected by such bugs. But there could be enough in use that they couldn't be dismissed as outliers. "I guarantee that there have been pockets where patches weren't put into place," said Goettl.
As an example of what Goettl meant, Bradley pointed out an issue with the November cumulative update for Windows 7 and 8.1 (as well as Windows 10) that broke dot-matrix printing for customers wedded to that 20th-century technology.
"Everyone had to roll off the November updates and hang tight if they needed dot-matrix printing," Bradley noted. "Initially you think, 'Oh, that's ancient technology, who uses that anymore?' [But] then you think of every single airport you've ever been through that still used dot-matrix printers for their passenger manifests at the gate and you go, 'Uh maybe that's more widespread than you think.'
"Ergo, [that's] why that fix came out pretty fast," she said.
Other problems with the new model stemmed from mixed messages and the resulting confusion, Bradley said. "The biggest bumps I've personally seen are in the detection and supersedence areas," she said. "[Microsoft] originally did it one way, realized it had issues, and then changed."
Bradley was right to highlight the supersedence problem - which updates took priority, and thus replaced others - because Microsoft's decision to offer two monthly updates (actually three when counting a preview) wasn't working for some customers. "Starting in December 2016, monthly rollups will not supersede security-only updates," Microsoft announced that month in additions to a long blog post of two months prior.
What Microsoft dubbed security-only quality updates contained just that month's patches, while the security monthly quality rollup included that month's fixes as well as the patches from all previous months (hence the use of "rollup," a term Microsoft has long deployed to signal past-and-present updates). Some were bewildered about what showed up where, when and in what order.
"Many admins are deploying the security-only updates, only to find that any fixes for the security-only updates are in the quality rollups," Bradley said.
Comments from the puzzled were appended to the supersedence post Microsoft wrote. "If a security-only update has a bug of a non-security nature, and this bug is fixed in a monthly rollup, can we expect the bug to also be fixed a) in an updated version of the problematic security-only update, b) in a separate update, or c) none of the above?" asked someone identified as Brian.
Brian didn't get an answer.
In January 2017, Microsoft made another change to the update process, this time separating the security update for Internet Explorer 11 (IE11) from the rest of the fixes; the move was a minor repudiation of the cumulative model, as it allowed customers to not install the IE11 update while still rolling out the rest of the Windows patches.
"One thing that softened the [cumulative] blow was they offered the security-only bundle, and then the IE update each month," said Goettl, "so users could get just the security pieces."
But don't expect Microsoft to bend much more than that. With just over two years of support remaining to Windows 7 - Microsoft plans to retire the OS in mid-January 2020 - and Windows 8.1 accounting for a small piece of Windows (8% in November by Net Applications' estimate), the cumulative update standard won't be upturned.
"Microsoft is not going to change this model, other than maybe offer some more flexibility in user experience," said Goettl.