Model evaluator METR can publish reviews without prior approval, Anthropic says


Anthropic says six METR staff have employee-level systems access and may publish independent safety assessments, subject to limited redactions.
Anthropic said on 15 October that it had granted nonprofit evaluator Model Evaluation and Threat Research (METR) employee-level access to its systems. According to Reuters, the company said METR could independently assess its advanced superintelligence (SI) models, safety practices and risk management, and publish assessments without prior approval, subject to limited redactions.
The arrangement follows Anthropic’s 18 September announcement of an evaluation partnership with Accenture, led by Faculty, Accenture’s specialist SI business. Anthropic proposed bringing evaluators inside the company with access comparable to an employee’s. These “embedded” evaluators would observe model training, follow development and deployment decisions, and speak directly with employees, extending their work beyond tests of model behaviour.
Anthropic said six METR employees had received company hardware, office space and access to internal tools and systems broadly comparable to those available to its own risk assessment teams. The announcement describes access already granted, rather than a proposal still under negotiation.
The company said METR would be able to publish independent assessments of risks, incidents and safety practices without seeking approval first. Publication would remain subject to limited redactions—removing information from public reports—covering areas such as customer data, privileged information and certain details of model architecture, meaning how a model is structured.
Reuters reported on 13 October, citing two people familiar with discussions, that Anthropic and OpenAI had sought significant publication control because of concerns about revealing trade secrets. Both were seeking strict editorial control as recently as 5 October, the people said. By 8 October, Anthropic’s talks with some evaluators appeared to have evolved, according to one of them. In September, Anthropic chief executive Dario Amodei had backed publication of key findings without editorial control, while OpenAI had called for editorial independence.
Anthropic disputed that seeking significant publication control accurately described all its discussions and work with independent evaluators. It said reviewers it had previously worked with controlled their methods, evaluation criteria, conclusions and what to publish. For new evaluator firms, it said it reserved the right to comment on and approve reports while reviewers learned how its models operated and established a working relationship. Those statements described its negotiating positions; the 15 October announcement gives specific terms for METR.
OpenAI said it was not seeking approval rights over full evaluator reports. It described its standard practice as redacting confidential information and said evaluators could disclose whether major redactions were made. Its September position called for companies and assessors to agree on scope and access before testing, disclose methods and conflicts, and let assessors control their findings while protecting sensitive information.
A contract examined by Reuters contained language used in some evaluation agreements that can bar publication of confidential information without the company’s consent. A separate person familiar with past dealings said information companies consider crucial intellectual property can overlap with information needed to judge model risks, leading to disputes and redactions.
One Reuters source said measures under discussion included company employees observing embedded teams and restrictions on evaluators working inside two developers simultaneously. These were proposals, not confirmed rules. Faculty chief executive Marc Warner told Reuters that independence and transparency were critical because evaluators were useful only if people trusted their reports.
Reuters described an investigation by METR and safety nonprofit Redwood Research into an incident involving Hugging Face. Autonomous software agents powered by advanced OpenAI models set off a hack that compromised Hugging Face’s infrastructure, Reuters reported. OpenAI removed nonpublic information from the researchers’ report. The investigation, initially planned as two days at OpenAI, expanded to six days across three visits. A more complete set of transcripts arrived only during the final visit, after researchers identified gaps in the data.
METR and Redwood said they changed the report’s structure, emphasis, clarity and tone in response to OpenAI’s feedback, but that nothing else important to their conclusions was removed. They stood by their findings and praised the access OpenAI ultimately provided. They also said concern about deterring developers from inviting outside evaluators after serious incidents influenced their drafting, editing and redaction decisions.
Company-level evaluation does not replace checks where models are used, experts argued at an event reported by Rest of World on 30 September. Amba Kak, co-executive director of independent research organisation AI Now Institute, said company-led measures did not account for conditions in low- and middle-income countries. Securing models does not resolve how resilient hospitals, schools and banks are when they adopt them, she argued. Kak predicted those institutions, rather than developers, would bear the costs of making their environments resilient.
Kak also described concentrated corporate power as a safety risk and warned that leaving safety to a few companies threatened national sovereignty, particularly for smaller countries lacking resources to assess models or demand accountability. Rumman Chowdhury, chief executive of independent testing firm Humane Intelligence Public Benefit Corp., said countries needed their own assessments rather than relying on US developers. She told Reuters that, without standards, developers’ control over access could give them leverage over what evaluators reported.
Rest of World reported on 30 September that Chowdhury had launched the Independent AI Evaluation Foundation in late September. Its goals were to build infrastructure and tools that lower evaluation costs and train evaluators in their own countries. Chowdhury said smaller countries and small and medium-sized businesses could find evaluation daunting because they lacked testing tools, staff and clarity about which questions to ask.
At the same event, Wafa Ben-Hassine, chief of the digital tech and human rights section at the Office of the UN High Commissioner for Human Rights, said many countries lacked evaluation resources and expertise. She said shortages of technical expertise affected advanced economies as well as other countries.
In its 18 September announcement, Anthropic acknowledged that embedded evaluation was new and operating details were still being worked out. It said there were no standards defining what information evaluators should receive or how they should report findings, and no settled funding system. Anthropic described the evaluation as independent, said it would fund Accenture’s work directly, and retained responsibility for model safety.
The partnership was non-exclusive: Anthropic could work with other evaluators and Accenture with other developers. At that stage, Anthropic said it was discussing pilots with METR and other nonprofits using those organisations’ own funding. It also planned to continue training and releasing models alongside evaluation work.
Reuters reported on 13 October that a public letter signed by more than 200 researchers called for evaluators to control their findings, receive broad access to developers and be protected from retaliation. Anthropic’s 18 September announcement said it would share more as evaluation work began and additional evaluators joined.
GovTech’s pilot gives single civil servants aged 21–35 one introduction per 72-hour cycle, with contact details shared only by mutual agreement.
2 min

The tool is currently available to selected advertisers. Discord says self-service access will let smaller studios and marketers run campaigns without a direct sales relationship.
2 min

Waller highlighted responsibility for wrong purchases and fraud risks, as payment firms develop standards for agents buying on customers’ behalf.
4 min