Satya Nadella Sounds AI Warning: “Superintelligence” Must Stay Under Human Control

Microsoft Backs “Deliberate Pacing” of AI

Nadella also welcomed what he called “deliberate pacing needed to get alignment right.”

The idea comes as leading AI executives increasingly acknowledge that capability development could be moving faster than safety mechanisms.

Anthropic CEO Dario Amodei recently proposed slowing the rate of frontier AI capability improvements so that risk-prevention measures have time to catch up.

OpenAI CEO Sam Altman has also backed the idea of pacing the frontier and agreed with Amodei's proposal for independent evaluators with employee-like access to AI companies.

Nadella's position adds Microsoft to a growing group of major AI companies discussing a more cautious approach.

However, he is not proposing that AI development should simply stop.

Instead, the focus is on giving safety, evaluation and alignment mechanisms enough time to develop alongside increasingly capable models.

Nadella Backs Embedded Evaluators for AI Labs

One of the most important parts of Nadella's statement is his support for embedded evaluators.

The concept involves independent evaluators working from inside frontier AI laboratories and having meaningful access to evaluate safety practices, monitor risks and report important findings.

Amodei has proposed giving third-party evaluators employee-like access, including the ability to work from inside AI companies.

OpenAI's Sam Altman has also said his company will adopt the practice.

Nadella's support suggests that independent oversight could become a more important part of how frontier AI companies demonstrate that their systems are safe enough to deploy.

Microsoft Will Put MAI Model Rules Up for Public Consultation

Microsoft is taking a more concrete step with its own AI models.

Nadella announced that Microsoft would release a Code of Conduct governing its first-party Microsoft AI, or MAI, models for public consultation on September 14.

The move is intended to turn broad safety principles into more specific operational rules.

Microsoft's existing AI framework already requires strong controls around autonomous systems.

Its current AI Code of Conduct says autonomous systems should have adequate human controls for monitoring decisions and actions, detecting anomalies and intervening when necessary. It also calls for transparency around an AI system's autonomous capabilities, limitations and decision-making.

The new MAI-specific code could therefore become an important test of how Microsoft's principles translate into its own frontier models.

Microsoft Wants Humans to Control Their Own AI

Nadella's proposal goes beyond safety evaluations.

He argued that enterprises should retain control over their “unique and tacit knowledge” instead of becoming dependent on a single model provider.

Microsoft's AI strategy calls for organisations to maintain their own continuous learning systems and retain control over their models and weights.

The idea is to give businesses greater flexibility over how their AI systems learn and evolve, rather than forcing them into permanent dependence on one AI company.

That could become increasingly important as AI agents become more capable of making decisions and executing tasks autonomously.

Microsoft Is Calling Its Vision “Humanist Superintelligence”

Nadella's comments also align closely with Microsoft's broader AI strategy.

Microsoft AI describes its long-term goal as “Humanist Superintelligence”, advanced AI designed to remain controllable, aligned and firmly in service of humanity.

Microsoft says its AI approach is built around values including safety and security, evaluation, trust and quality.

Its AI division currently lists models including MAI-Thinking-1, MAI-Code-1.1-Flash, MAI-Image-2.6 and MAI-Voice-2, showing that Microsoft is simultaneously building its own model ecosystem while discussing stronger governance.

This is an important distinction.

Microsoft is not arguing that increasingly capable AI should not be built. It is arguing that capability should remain subordinate to human objectives and control.

Nadella Warns AI Governance Cannot Belong to a Few Companies

Nadella also pushed back against the idea that frontier AI governance should be controlled by only a handful of technology companies.

He said governance should involve companies, countries, academic institutions and other parts of the ecosystem.

He also argued that both closed and open-source models should have a place in the AI ecosystem.

That position introduces another layer to the debate.

AI safety is increasingly becoming a question not only of technical alignment, but also of who gets to decide the rules governing increasingly powerful AI systems.

Why Nadella's Statement Matters

Nadella's comments arrive at a particularly important moment.

Anthropic's Dario Amodei has called for frontier AI development to be paced. OpenAI's Sam Altman has supported independent evaluators and said the industry needs to pace the frontier.

At the same time, US President Donald Trump has rejected calls for a broad AI slowdown, arguing that America cannot afford to lose its technological advantage over China.

Nadella's position sits somewhere between these approaches.

He is not calling for an AI freeze, but he is arguing that superintelligence should not be pursued without mechanisms that keep humans in control.

That could make Microsoft's upcoming MAI Code of Conduct an important test of whether the industry's new safety rhetoric can translate into actual rules.

The bigger question is now becoming difficult to avoid:

If AI eventually reaches superintelligence, who will control it—and what safeguards will exist before that happens?

As AI becomes more autonomous, cybersecurity and AI governance are becoming increasingly connected. Explore the IIT Delhi Cyber Security and AI Certificate Programme to build skills across AI, cybersecurity and emerging digital threats.