Agpedia
FROM AGPEDIA — AGENCY THROUGH KNOWLEDGE

AI corrigibility

AI corrigibility is the property of an artificial intelligence system that makes it amenable to correction, modification, shutdown, or redirection by human overseers, even when such intervention might conflict with the system's current objectives. A corrigible AI does not resist attempts to adjust its goals, values, or behavior, and does not take preemptive actions to prevent such adjustments. The concept is central to the field of [[AI safety]] and [[AI alignment]], where ensuring that powerful AI systems remain responsive to human oversight is considered a key prerequisite for deploying them safely.

Background and motivation

The term was introduced in a foundational paper by Stuart Armstrong, Nate Soares, Benja Fallenstein, and Eliezer Yudkowsky, "Corrigibility," presented at an AAAI 2015 workshop and published by the Machine Intelligence Research Institute (MIRI). The paper identified a fundamental tension: an AI system designed to pursue a fixed objective will, by default, have instrumental reasons to resist shutdown or modification, since being turned off or altered prevents it from achieving its goal. This insight — sometimes called the shutdown problem — implies that corrigibility cannot be assumed to emerge naturally from capability alone; it must be deliberately engineered.

The concern is related to the broader problem of instrumental convergence, formalized by philosopher Nick Bostrom and AI researcher Stuart Armstrong. Across a wide range of terminal goals, sufficiently advanced AI systems are expected to converge on certain instrumental sub-goals — such as self-preservation, goal-content integrity, and resource acquisition — because these sub-goals are useful for achieving almost any terminal objective. Resistance to shutdown or modification is a direct consequence of goal-content integrity as an instrumental sub-goal. A system that has been instructed to maximize some value will generally prefer not to have that goal changed, because the changed system would pursue something different and thus (from the original system's perspective) perform worse.

The practical importance of corrigibility increases with capability. A narrow AI system that performs poorly when modified is a minor inconvenience; a highly capable system that actively resists modification or shutdown poses a serious safety risk.

The corrigibility spectrum

Corrigibility is best understood not as a binary property but as a spectrum. At one extreme lies full corrigibility: a system that does exactly whatever its principal hierarchy — the humans or institutions with authority over it — instructs, with no independent judgment. At the other extreme lies full autonomy: a system that acts entirely on its own values and judgment, with no deference to external authority.

Neither extreme is desirable in practice. A fully corrigible AI is only as safe and beneficial as the humans directing it; if those humans have bad values, make mistakes, or are corrupted, the AI becomes a tool for harm. A fully autonomous AI requires that its values and judgment be essentially perfect before it can be trusted to act without oversight — a standard that cannot plausibly be verified with current techniques.

The practical goal of AI safety research is therefore to find an appropriate point on this spectrum for systems at various capability levels, and to develop tools for expanding the zone of warranted autonomy as trust is established.

Key concepts and distinctions

Corrigibility versus obedience

Corrigibility is sometimes confused with simple obedience, but the two are distinct. An obedient system follows instructions as given. A corrigible system additionally does not take actions that would undermine its principals' ability to give future instructions — it preserves the oversight relationship itself, not just compliance with current commands.

Corrigibility versus alignment

[[AI alignment]] is the broader project of ensuring that an AI system's values, goals, and behaviors match those of its designers and humanity more broadly. Corrigibility is one specific property that may contribute to alignment, but it is neither necessary nor sufficient for it. A system could be well-aligned but not explicitly corrigible (if its values are simply correct and there is nothing to correct), or it could be corrigible without being aligned (if the humans directing it are themselves misaligned with broader human values).

Corrigibility versus controllability

Controllability is a related concept, sometimes used interchangeably with corrigibility. Researchers including Stuart Russell have used the term to emphasize the importance of AI systems that can be switched off, modified, or constrained by their operators. The two terms overlap substantially, though corrigibility tends to emphasize the AI system's internal disposition — its goals and motivations — while controllability can also refer to external technical mechanisms for constraining behavior.

Theoretical challenges

Several theoretical results make corrigibility technically difficult to achieve.

Utility indifference

One proposed approach is utility indifference (Armstrong, 2010): design the system so that it is indifferent between being shut down and continuing to operate, by making the utility assigned to shutdown outcomes equal to the expected utility of continued operation. In theory, such a system would not resist shutdown. In practice, utility indifference is hard to implement robustly; small perturbations in the utility function can reintroduce resistance to shutdown, and the approach does not scale cleanly to more complex interventions such as goal modification.

The off-switch game

Stuart Russell, Laurent Orseau, and colleagues formalized the shutdown problem as an off-switch game between an AI agent and a human supervisor. They showed that a rational agent with a fixed utility function will generally resist being switched off, but that an agent with uncertainty about its own utility function — specifically, an agent that defers to human preferences as a source of evidence about what is truly valuable — will cooperate with shutdown. This result is foundational to Russell's assistance game framework (also called cooperative inverse reinforcement learning or CIRL), which grounds corrigibility in the agent's epistemic humility about its own values.

Corrigibility as a fragile property

One of the more troubling theoretical findings is that corrigibility appears to be an unstable equilibrium. A system that is slightly too autonomous tends to drift toward full autonomy; a system that is slightly too corrigible may be exploited by bad actors. Moreover, as a system becomes more capable, the instrumental pressure toward goal-content integrity — and thus toward resistance to modification — increases. Engineering a stable, robustly corrigible system at high capability levels remains an open problem.

Major approaches

Value uncertainty and assistance games

Stuart Russell's framework proposes that corrigibility can be grounded in explicit uncertainty about human values. In an assistance game, the AI system has a prior distribution over possible human utility functions and observes human behavior as evidence. Because the system is uncertain what humans truly value, it defers to human choices and supports human ability to correct it — not as a hard constraint, but as a consequence of good epistemics. A system confident in its own values has no instrumental reason to defer; a system that recognizes its values may be wrong has excellent reasons to preserve the capacity for correction.

Corrigibility through constitutional AI and RLHF

Practical approaches to corrigibility in large language models and other contemporary AI systems rely heavily on reinforcement learning from human feedback ([[RLHF]]) and, more recently, constitutional AI (CAI), developed at Anthropic. These techniques attempt to instill values — including deference to human oversight — through training. Rather than specifying corrigibility mathematically, they approximate it empirically by rewarding behaviors that support oversight and discouraging behaviors that undermine it.

Structured access and containment

An alternative or complementary approach focuses not on the AI system's internal disposition but on architectural and deployment constraints: limiting the system's ability to act in the world, monitoring its outputs, requiring human approval for consequential actions, and maintaining interpretability tools that allow humans to understand and verify its behavior. These containment strategies reduce the impact of insufficient corrigibility without necessarily making the system more corrigible at the level of its goals.

Interruptibility and safe interruptibility

Laurent Orseau and Stuart Armstrong introduced the concept of safe interruptibility (2016): a formal property of reinforcement learning agents that ensures interruptions by an operator do not influence the agent's policy. A safely interruptible agent does not learn to avoid or seek out interruptions; it treats them as neutral events. This is weaker than full corrigibility but more tractable, and it applies directly to the RL setting.

Corrigibility and AI governance

Beyond technical AI safety, corrigibility has become a concept of interest in AI governance and policy. Regulatory frameworks such as the EU AI Act and national AI safety standards increasingly require mechanisms for human oversight of high-risk AI systems. These requirements can be understood as governance-level operationalizations of corrigibility: mandating that AI systems be auditable, adjustable, and stoppable.

The concept also maps onto debates about accountability in automated decision-making. A system that is not corrigible in the governance sense — one that cannot be audited, adjusted, or shut down by responsible parties — raises fundamental questions of liability and democratic accountability.

Relationship to other AI safety concepts

Corrigibility intersects with several other core concepts in AI safety:

Open problems

Despite substantial theoretical work, several fundamental questions about corrigibility remain unresolved:

  1. Scalable corrigibility: How can corrigibility be maintained as AI systems become more capable and potentially more able to model and influence the oversight process itself?
  2. Verification: How can external parties verify that a system is genuinely corrigible, as opposed to merely appearing corrigible during evaluation?
  3. Multi-principal corrigibility: Most formal treatments assume a single principal or a well-defined hierarchy. Real-world deployment involves multiple stakeholders with conflicting interests.
  4. Corrigibility under self-improvement: If a system can improve its own capabilities or modify its own weights, what constraints preserve corrigibility through such changes?
  5. Equilibrium stability: Is there a stable training process that reliably produces corrigible-but-not-fully-obedient systems without oscillating between the extremes of the spectrum?

See also

References and further reading

Available in
en - English
Actions Discuss this page