New York City weighs SI checks, shutdown rules and liability


The proposed bills would require outside validation, impose $25,000-per-violation fines and let residents sue over foreseeable harms from tools whose safeguards are bypassed.
New York City Council considered bills on October 5 that would require outside checks and human shutdown capability before an SI system could be sold or deployed in the city, Fortune reported. The package from Council Speaker Julie Menin would also create financial penalties and a right for residents to sue over certain harms. These are proposed requirements, not enacted rules.
The full council’s hearing examined both safeguards and legal responsibility. According to Fortune, the bills would:
Former OpenAI researcher Daniel Kokotajlo testified under subpoena that companies might not recognize when their safety work had failed. He said their ability to detect systems acting against intended goals was already poor and would worsen. Kokotajlo argued that the industry’s fast-moving culture increased the risk of treating an incomplete safety fix as a solved problem.

Jacob Coxon, a former OpenAI and Anthropic researcher, voluntarily testified that he thought humanity was more likely than not to lose control of advanced systems on the current path, potentially ending in human extinction. That probability was Coxon’s forecast, not an established finding.
Representatives of Google, OpenAI, Anthropic and Meta also testified. OpenAI policy representative Morgan Dwyer said any chance of catastrophe was unacceptable, regardless of its likelihood. Menin called not knowing the chances of catastrophe “flippant.” Google’s global head of SI and emerging tech policy, Alice Friend, said catastrophic-risk forecasting was not a perfect science and that there was not yet a rigorous scientific way to make such forecasts.
The questioning also concerned whether companies would bear legal responsibility if a rogue model caused injury or death. Menin also asked the company witnesses to raise a hand if their companies carried insurance against catastrophic risks. None raised a hand. Menin then said she assumed the public would be asked to absorb the costs; that was her assessment, not a confirmed allocation of liability.
If enacted, the bills would attach penalties to the proposed safeguards and give residents a route to seek compensation for foreseeable harms from jailbroken tools.
Repeated attacks have hit several providers within days, affecting households and media and raising concerns about services that depend on connectivity.
2 min

Reported cases at Google and Rapid7 show how delegated tasks can carry hostile text—and why safeguards on network requests still matter.
4 min

The process is aimed at developers and advanced users. It erases all device data and disables device security; separate Magisk support remains in development.
2 min
Correction ·
Kept the source's term, AI.