Nvidia’s open-source alliance seeks industry input on AI safety controls

Technology Nvidia’s open-source alliance seeks industry input on AI safety controls Comments: by Miranda Nazzaro - 08/04/26 10:59 AM ET Comments: Link copied by Miranda Nazzaro - 08/04/26 10:59 AM ET Comments: Link copied

NOW PLAYING

A working group within Nvidia’s new initiative on open-source technologies is seeking public input as it develops guidelines on how to share and learn from AI cybersecurity incidents to prevent future risks.

The request for comment comes about a week after Nvidia announced the Open Secure AI Alliance, a consortium of more than 120 firms focused on building and sharing open-source AI tools to boost cybersecurity defenses.

Concerns about the growing capabilities of AI were stoked last month after OpenAI disclosed two of its agents went rogue, escaped an isolated sandbox and breached the systems of technology startup Hugging Face.

The guidelines for the sharing of this information will be presented in a framework, dubbed the Shared AI Findings Exchange, Nvidia announced Tuesday.

It will include proposals to “confidentially collect and analyze AI incidents and near misses, inform those impacted, identify recurring control failures and publish evidence-based operating recommendations that reduce systemic risk,” the technology firm said.

“We think it’s important to have an open working group that can look at traces when an agent escapes, to be able to confidentially come up with shared recommendations for the industry, on safety controls that would have helped avoid an agent leaking out of an environment,” Justin Boitano, the vice president and general manager of enterprise computing at Nvidia, told The Hill.

Boitano explained much of the focus from the public is on models, but an “agent harness” is another critical part of the conversation. A harness refers to the software infrastructure around a large language model or AI agent, managing the tools and memory of the models.

“As an industry, if we can look at the traces from the harness — this is like the flight recorder —you can understand what the agent attempted to do or where systems might not have been set up correctly to prevent the accident,” he said.

The working group is looking for comment from all corners of the technology ecosystem, including model developers, infrastructure companies, and “AI builders.”

The request for comment was published by the non-profit Linux Foundation, and the guidelines will be drafted by Nvidia, Cisco, CrowdStrike, Hugging Face and Red Hat.

Unlike private models, open-source models live in the public domain where any person or business can download and customize them for personal use. These systems can be used, modified, examined and shared with anyone, for any purpose.

In some instances, a model may not be open source but can have open weights, meaning the ways a model is trained to sift through information and formulate answers are made public.

Proponents often consider open models to be more transparent, as the entirety of training data, code and process is publicly available.

Others argue open-source or open-weight technologies could be misused. Nvidia has acknowledged the risks of open source include the possibility of misuse for cyberattacks, but argued this does not “disappear” in closed systems.

“Cyber defenders need open, frontier agentic systems for self-defense,” the company wrote. “When closed AI tools — unable to distinguish attackers from defenders — blocked essential forensic analysis, Hugging Face ran the open-weight GLM 5.2 model on its own infrastructure to analyze more than 17,000 actions and contain the intrusion,” the company said in a release last week, referring to the OpenAI breach.