The AI industry keeps asking Americans to trust its judgment, then keeps giving them reasons to doubt it.
The Future of Life Institute’s new AI safety scorecard grades nine leading developers across 37 indicators, and the best overall mark goes to Anthropic with a C+. OpenAI and Google DeepMind receive Cs. xAI, DeepSeek and Mistral fail. For policymakers and business leaders, that should end one comforting fiction: the companies building frontier systems can police themselves through press releases, voluntary principles and private assurances.
The central finding is broader than one company’s grade. Major developers have been retreating from prior safety pledges even as capabilities grow. That is exactly what competitive pressure predicts. If one lab promises to pause when danger thresholds appear, executives soon ask whether a rival will keep racing. The public gets solemn language about responsibility while the actual incentive pushes toward speed.
Why voluntary AI safety frameworks fall short
That makes today’s debate over corporate AI too narrow. NIST’s AI Risk Management Framework gives organizations a useful structure for mapping, measuring and managing AI risks. Yet NIST presents it as voluntary guidance. Voluntary guidance can improve the behavior of responsible firms. It cannot answer the harder question of what happens when a voluntary safety commitment threatens market share, geopolitical status or a multibillion-dollar launch.
An International AI Safety Report warns that general-purpose AI systems raise emerging risks and safety challenges across capabilities, deployment and governance. That matters because the policy argument should move beyond whether AI can cause harm in principle. The serious question is whether our institutions can verify that the most powerful systems have been tested honestly before they spread into critical workflows, consumer products, cybersecurity tools and military settings.
What should federal AI safety legislation require of frontier developers?
Congress should start with a federal floor for frontier AI, modeled in part on the strongest state efforts but designed to avoid a messy patchwork. California’s SB 53 requires large frontier developers to publish a framework for managing catastrophic risk, define capability thresholds, disclose summaries of risk assessments, report critical safety incidents and protect certain employees who raise concerns. Those elements belong in federal law because the products cross state lines the moment they launch.
The floor should have three parts. First, developers should publish real risk thresholds before major deployment decisions, not vague promises to be careful. The EU AI Act’s systemic-risk provisions require providers of general-purpose AI models with systemic risk to perform model evaluations, assess and mitigate systemic risks, track serious incidents and report relevant information. The United States does not need to copy Europe’s entire regulatory architecture. It does need the basic principle that danger thresholds should trigger duties rather than internal debate.
Second, Congress should require AI audits for qualifying frontier systems. Illinois’s Artificial Intelligence Safety Measures Act, signed this month, adds annual third-party audits for major AI developers and requires rapid reporting of potential large-scale harms. Proper audits would not let regulators micromanage model design. They would test whether companies followed their own frameworks, whether evaluators had enough independence and whether executives overrode internal safety findings without a documented basis.
Third, lawmakers should build whistleblower protection into the core of AI governance. The California Report on Frontier AI Policy recommended a “trust but verify” approach that recognizes both the value of innovation and the need to reduce material risks. Verification becomes hollow if employees who see dangerous shortcuts fear retaliation, nondisparagement clauses or career destruction. Congress should protect good-faith disclosures to federal authorities and require internal reporting channels for safety staff.
The White House has taken a partial step toward federal oversight by inviting leading developers to submit advanced models for national-security review before release. That voluntary process may produce useful information. Yet it leaves the central weakness untouched. A company facing intense competitive pressure can narrow cooperation, delay disclosure or simply decline to treat an uncomfortable result as a binding stop sign.
Would AI regulation hurt U.S. competitiveness against China?
Industry will warn that regulation could slow America’s lead over China. That concern deserves respect, but it should not become a veto. A narrow federal regime focused on the largest developers and most capable models would protect startups from needless burdens. It would also strengthen national competitiveness by making American AI more trustworthy in defense, finance, health care and public administration. Durable leadership requires systems that allies, agencies, businesses and citizens can rely on.
The alternative is a market where AI companies grade their own homework while asking society to absorb the downside. The European Commission’s General-Purpose AI Code of Practice shows how detailed safety, transparency and copyright practices can support compliance with broader law. America can choose a lighter approach, but it cannot choose no verification and still expect public confidence to survive the first preventable catastrophe.
What AI safety rules mean for employers buying AI tools
Business leaders should draw the same lesson. Responsible AI adoption at work requires more than buying the newest model and trusting the vendor’s branding. Organizations need procurement standards, incident escalation paths and clear limits on high-stakes use. If Washington creates a credible safety floor, employers will find it easier to separate serious vendors from those selling confidence without accountability.
The Future of Life Institute’s index does not prove that any one company will cause a disaster. It proves something more immediately useful: the current system rewards companies for moving faster than their safety commitments can withstand. Congress should stop treating that as a public-relations problem. It is a governance failure. And governance failures, unlike model weights, remain squarely in human hands.
