The July tech outage that knocked out businesses worldwide renews scrutiny over why some companies have access to the very heart of a computer and, therefore, the ability to suddenly crash it.
Access to the “kernel,” considered the core of a computer’s operating system, is critical to protecting against viruses. But last month’s tech outage showed the risk of such openness on Microsoft’s Windows computers. Developers and software makers say there are safer ways to tap in to the kernel without full access, but Microsoft hasn’t made the leap.
An errant software update from the cybersecurity firm CrowdStrike in July took out 8.5 million Microsoft Windows computers, disrupting operations for banks, emergency services, schools and hospitals while forcing airlines to ground flights.
Allowing software makers to run “kernel drivers,” which are programs built to access a computer’s core and its hardware, isn’t the only option for their products to work. Apple, for instance, cuts off access to its MacOS operating system kernel to all third parties, forcing cyber vendors to operate in the more restricted “user mode.”
In user mode, where most applications run today, buggy software can’t crash a computer. In kernel mode, where CrowdStrike and most other antivirus makers run on Windows, a faulty update can bring on the “blue screen of death.” But there are also benefits to running in kernel mode, such as giving antivirus makers access to the low-level system data critical for detecting cyberattacks and threats, and letting their tools activate before malware can.
Another open-source alternative is already available on the Linux operating system, but it’s up to Microsoft to make it available for Windows, said Alexei Starovoitov, a Meta Platforms engineer and a creator of the technology.
Called the Extended Berkeley Packet Filter, or eBPF, the decade-old technology pioneered by Starovoitov and others could have helped prevent CrowdStrike’s global outage, its proponents and cyber vendors say. EBPF puts programs in a walled-off environment in the kernel, preventing a bad or malicious update from reaching it and crashing a computer.
CrowdStrike agrees. It’s a “super revolutionary technology,” the company’s president, Michael Sentonas, said. “If something happens where you have a crash, you don’t take out the entire kernel.”
While headlines rolled out on the impact of the outage, Brendan Gregg, an eBPF pioneer and fellow at Intel, said he and other leaders of the open-source technology were talking: “We’re like, ‘We have worked on the solution to this for so many years.’”
Using the kernel isn’t new. “There’s a lot of software that we don’t even realize has access in kernel mode,” said Allie Mellen, a cybersecurity analyst at Forrester Research. Many functions on a computer simply work better with kernel access, Microsoft said.
Gaming and photo-editing software and programs for printing and enabling Bluetooth all tap in to the kernel to interact with a computer’s hardware and improve its performance. Any of these can knock out a computer—and have.
“Kernel modules crashing an operating system is nothing new,” said Craig Connors, chief technology officer of Cisco’s security business group. “It’s only new that it happened eight million times, from the same thing.”
For Microsoft, the decision to give developers kernel-level access dates back about 15 years. A company spokesman said it can’t legally wall off its operating system because of an understanding it reached with the European Commission at the time.
The case—brought by the commission over concerns that Microsoft may have abused its dominant market position by tying its web browser and other software to Windows—was resolved in December 2009 when Microsoft agreed to give users more browser choices. The company also committed to giving third-party vendors the same level of operating system access that Microsoft gets.
In other words, Microsoft’s Defender security product can’t hold a competitive advantage over competing products by preventing them from accessing the kernel or running a kernel driver.
With access to the Windows kernel, CrowdStrike and other antivirus makers must determine when it’s best to tap user mode instead of kernel mode in their products, and customers, in turn, have had to figure out the trade-offs in kernel programming decisions—no easy task for even experienced software developers.
One of eBPF’s primary benefits is that it enables software like CrowdStrike’s to access the kernel, but lets only safe, verified programs actually run. That means the bad CrowdStrike update would have been walled off.
Plus, a member of eBPF’s open-source community could have spotted bugs much sooner, Starovoitov said, “before it’s deployed to millions of servers around the world.”
Microsoft has been working with the eBPF Foundation, which is part of the Linux Foundation, for several years to make it available on Windows, said Gregg of eBPF. CrowdStrike said it uses eBPF for Linux, and is willing to use it on Windows when Microsoft makes it available.
But, Microsoft said eBPF isn’t ready for Windows yet, citing new security risks it could introduce. Its task is to secure and manage Windows holistically, not just the slice that cyber vendors use, it said.
Gregg said that while it will take time for the technology to improve on Windows, its additional security measures still make it safer than using the kernel.
Without providing a specific timeline, Microsoft said the most likely near-term solution is to give developers other options for direct and indirect kernel access. In the longer term, it will move toward an Apple-like approach where developers get pushed deeper into user mode and out of the kernel. That takes custom engineering work, it said, much of which is already done.
Microsoft said it already has worked with some vendors, including CrowdStrike, to put more functionality in user mode. Regardless of how any developer accesses the kernel, they can still crash a computer without proper safeguards, testing and software rollout, Microsoft said.
Still, some cyber vendors aren’t convinced Microsoft will enable eBPF, considered the Linux method, or wall off kernel access, following the Apple method.
Alex Stamos, chief information security officer of CrowdStrike competitor SentinelOne, said regardless of the option Microsoft pursues, Microsoft’s own security product must use the method of access as outside competitors. If not, Microsoft gives itself an unfair advantage as both operating system maker and security vendor, he said.
Meanwhile, the pressure to find an alternative grows. In July, Germany’s powerful cybersecurity agency asked Microsoft to develop “more resilient architectures” for endpoint protection software with minimal kernel-level access.
Cisco also is working with Microsoft to build eBPF for Windows, Connors said, aiming to use the technology for its Windows antivirus software.
Any kernel overhaul requires a major product change and mind shift, said Gartner cybersecurity analyst Neil MacDonald. That’s time-consuming and costly for both Microsoft and its software partners. It might take the form of a larger Windows revamp, Stamos said, just as Apple made its change as part of a MacOS update and move to its own silicon.
“It’s somewhat painful, but it’s a necessary evolution,” MacDonald said.
Write to Belle Lin at belle.lin@wsj.com
