How a US Govt Board Helped the Open-Source Community Leap to Patch Log4j

LAS VEGAS–Six months ago, the federal government set up a new office and gave it a tough first assignment: Report back on the public-private response to the Log4j vulnerability that left a huge fraction of the web vulnerable to remote compromises.

At a panel discussion Wednesday at the Black Hat information-security conference here, that board’s chair and deputy chair shared the major lessons learned from that effort—starting with a welcome willingness among industry types to talk to a government organization.

“I think what surprised a lot of people was how deep the fact-finding could go,” Cyber Safety Review Board(Opens in a new window) Chair (and policy undersecretary at the Department of Homeland Security) Robert Silvers told panel moderator and Black Hat founder Jeff Moss. “We actually created a factual rollup of how the Log4J vulnerability disclosure process, the response, all went down.”

Co-chair Heather Adkins, VP of security engineering at Google, made the same point, calling herself “really pleasantly surprised” that 80-plus organizations and security researchers spoke to the board for the 52-page report(Opens in a new window) (PDF) it published in July. “We even heard from the People’s Republic of China.” 

The Cybersecurity and Infrastructure Security Agency (CISA) spun up the 15-member board in February, following a directive in the executive order on information security(Opens in a new window) that President Biden issued in May 2021. It’s roughly modeled after the National Transportation Safety Board, with the goal of bringing transparency to a field in which the targets of attacks have often retreated(Opens in a new window) into vague silence about what went wrong.

“Until the CSRB was created, there was really nobody whose job it is to convene 80 different companies and security researchers,” Silvers said. “Our charge was, let’s figure out what happened just so the community can know.” 

He and Adkins complimented such organizations as the Chinese e-commerce giant Alibaba and the open-source Apache Software Foundation for sprinting to patch the vulnerability in Log4j (people also refer to this “vuln” as Log4Shell or CVE-2021-44228(Opens in a new window)). Millions of sites use that open-source Java library developed by Apache to log their activities, but an undiscovered flaw allowed attackers to exploit it remotely to run arbitrary code on those servers. 

But having so many organizations rush out patches as sites rushed to install them—”this may have been the largest mass-scale cyber response in history,” Silvers said—did cause new complications. 

“There were a few iterations of the patch. We definitely found that this induced some patching fatigue,” he said. CISA attempted to ease this fatigue by posting a GitHub repository(Opens in a new window) of packages vulnerable to Log4j.

Adkins, in turn, commented that having these fixes “done in the open” unavoidably made some attackers more aware of the vulnerability: “We begin to see posts on WeChat in China, talking about this release candidate that has this fix.” 

Alibaba publicly reporting the vulnerability without notifying the Chinese government first also led to Beijing punishing the company(Opens in a new window).

And while early exploits may have just consisted of hackers snooping around, that was followed by remote installations of cryptocurrency-mining software, the sale of exploit kits and then use by nation-state attackers.

“How do we build a software ecosystem where things can happen quickly?” she asked. “How do we make the knowledge of the bug somewhat irrelevant?”

The remainder of the panel focused on how the board and the industry in general could make that happen. Adkins endorsed measures to help open-source foundations train developers and audit code that could make the group effort behind these projects more effective. 

As she put it: “We’re somewhat skiing on top of a pretty good avalanche of support among the community,”

Silvers said the government should require software transparency from vendors, including providing a software bill of materials (SBOM, pronounced “s-bomb”) for deliverables. “The board is a total believer in the SBOM concept, but it has to be evolved,” he said. “You have to know what you have and where.”