What Is Governance For?

When progress outruns oversight

Imagine a frontier AI developer using increasingly capable AI systems to conduct research and develop their successors. The systems propose experiments, write code, evaluate changes, and recommend the next steps. Results improve. The people responsible receive reports and approve another round of development.

Then the pace of development exceeds their ability to check a consequential change. The AI supplies an explanation, but the reviewers cannot establish whether the safeguards on which approval depended still work. They have completed the required review. They have not resolved the question that makes continued authorization responsible.

The work continues. Better performance remains visible; the adequacy of oversight does not. The developer still owns the infrastructure and its executives still sign the approvals. Yet it can no longer adequately explain why the activity it is permitting remains within justified limits.

This is a hypothetical possibility, not an account of a particular laboratory. It draws its starting concern from AI 2027, a forecast scenario by Daniel Kokotajlo and his coauthors. They imagine increasingly automated research, with some employees “watching performance crawl up, and up, and up.” Their scenario also retains a contribution from some human researchers.¹

Delegating research can be valuable. The governing problem begins when those responsible cannot establish that the conditions for permission still hold. A familiar approval process can survive a change in what it must oversee; revising it needs appropriate authority and preservation of the protections it serves. Rising performance alone cannot establish adequate oversight.

Nor does the problem require a malicious AI. The improvements could be real and useful. A system could do exactly what its operator requested while producing consequences the operator cannot adequately oversee. Beneficial intentions would not supply the missing understanding, and ownership would not supply an effective response.

This is where the purpose of governance becomes urgent. We need arrangements that make consequential development answerable to people, including those outside the organization conducting it. Those arrangements must support worthwhile activity while preserving the ability to question its direction, constrain its effects, and correct its failures. Whether development should continue depends on more than whether it can continue.

Governing consequential power

Governance concerns the terms under which collective power is exercised. It establishes what an institution is for, who may decide, and how those decisions can be challenged. Its purpose extends from choosing a direction to ensuring that someone can act when the arrangement fails.

The working definition used throughout this book is: Governance is the set of arrangements through which a group or society sets direction, distributes and exercises authority, makes collective decisions, and determines how decisions and decision-makers can be challenged, corrected, or replaced.

That definition describes governance without assuming that it is good. An organization can have an elaborate governing arrangement and still pursue an unjust purpose. Its decisions may be orderly while its power is difficult to challenge. We must therefore ask whether the arrangement is adequate for the activity it governs, rather than treat its existence as an assurance of adequacy.

Managing development and governing it are connected responsibilities. Management organizes the work: allocating resources, scheduling experiments, and coordinating teams. Governance establishes and oversees the authority, direction, and conditions within which that work takes place. A research program can be efficiently managed while its purposes or permissions remain poorly justified.

The distinction becomes practical when a consequential question reaches the limits of someone's role. A researcher might recognize that an evaluation no longer supports its intended conclusion. A manager might be able to delay an experiment but lack authority to change the program's direction. Those with authority may lack access to the evidence. Governance must connect these responsibilities, rather than let each participant complete a narrow task while the unresolved matter passes between them.

AI development also raises questions whose consequences extend beyond the developer. Who benefits from a new capability? Who may be exposed to its effects? What interests must remain protected even when a project promises substantial gains? Expertise is essential to answering these questions, but technical achievement alone does not confer authority to decide them for everyone affected.

The people who build a system have knowledge others need. Workers, users, independent investigators, and affected communities may have knowledge the developer lacks. Public authorities have responsibilities that cannot simply be assigned to a supplier. A firm's ownership of a system, a government's legal powers, and a community's legitimate arrangements for collective decision-making can support different kinds of authority, each with its own limits.

These differences matter before we ask whether anyone retains control. Control must serve a legitimate purpose. An institution could direct its technology effectively while using it to diminish other people's agency. It could also announce a worthy purpose while lacking the means to carry it out. Good governance requires us to examine the justification for power and the capacity to exercise it responsibly.

Capability and protection

What, then, should governance make possible? This book's answer begins with collective capability: the practical ability of people and institutions to understand consequential situations, make legitimate choices, and carry those choices into effect. The capability includes recognizing when the activity or the governing arrangement needs to change.

Collective capability is broader than the developer's ability to produce a more powerful system. It includes independent scrutiny, informed public decisions, and protection against uses of power that others cannot meaningfully contest. Increasing one organization's capability may strengthen these capacities, weaken them, or do both in different respects. We need to ask whose ability to act is expanding and whose is being diminished.

Protection belongs within that account. A limit on data access can protect people from an intrusion they did not authorize. A route for challenging a decision can preserve agency when an institution makes a mistake. Restrictions on an activity can leave others freer to pursue their own purposes. Judging governance by how little it restrains would overlook what those restraints make possible.

The same reasoning applies to enabling worthwhile development. Clear permissions and dependable responsibilities can allow people to cooperate, invest effort, and pursue useful work. Governance should make a justified route to such activity possible. It should also make refusal meaningful where an activity cannot meet its obligations. The question is what we can responsibly accomplish, given the purposes and limits that should govern the effort.

There is no single measure that settles this. More investment, faster research, fewer incidents reported, and greater public confidence would each leave important questions open. What counts as improvement must be explained, including its effects on protected rights and on people's ability to choose. A benefit in one area cannot automatically compensate for a missing essential protection in another.

Purpose is itself a matter for governance. A developer may favor rapid capability growth; a public authority may place weight on security; those affected may be concerned about dependence or their ability to challenge decisions. Better evidence can inform these choices without eliminating their conflict. Legitimate decision-making requires opportunities for affected people to be heard, reasons for the choices made, and limits on the authority making them.

Participation does not require unanimous agreement or give every person a veto. It makes relevant interests and experience part of the governing question. The aim is not to manufacture acceptance of a decision already taken. It is to improve the grounds on which a decision can be made and challenged.

This has a humane significance even at the scale of frontier development. Governing capacity helps preserve people's ability to influence conditions that may otherwise be decided far beyond their reach. It makes useful cooperation possible without requiring everyone to become a technical specialist. People should be able to rely on institutions to perform responsibilities they cannot reasonably perform alone—and to question those institutions when that reliance is misplaced.

Why procedures remain valuable

The first priority in this book's manifesto is: We value collective capability over procedural compliance.

Procedural compliance is a genuine good. Stable requirements make duties intelligible and decisions more consistent. Records preserve reasons that can later be examined. A required review creates an opportunity to question a proposed action before its consequences become difficult to reverse. These arrangements can constrain arbitrary power, including the power of those who insist that their judgment makes ordinary rules unnecessary.

The priority asks what those procedures enable. In the opening scenario, receiving a report and obtaining an approval are steps in a review process. Their completion matters, but the responsible question is whether they provide an adequate basis for continuing development. If the process cannot answer that question, completing it again does not repair the gap.

We should be equally wary of the proposed shortcut. A developer might argue that its results are valuable, that delays are costly, or that reviewers cannot keep pace. Those concerns deserve consideration. They do not authorize the developer to discard obligations or expand its own permissions. Claiming to serve a larger purpose can become a convenient way to escape the limits that protect other people.

The strongest defense of procedure is therefore part of the argument for governing capability. Consistency protects against favoritism. Appropriate review makes disagreement consequential. Dependable limits allow people to cooperate without surrendering everything to the discretion of whoever currently holds power. A system of governance that gains speed by losing these protections may become more capable of acting while becoming less capable of acting legitimately.

Improvement requires a different response: examine why the procedure no longer serves its purpose, then revise it through authority appropriate to the change. Better evidence, different review methods, additional expertise, or a narrower permitted activity may help. Some protections may need to be retained precisely because pressure to relax them has increased. A failed method is a reason to reconsider the method, not automatic permission to abandon the obligation.

Approval and effective oversight

A completed approval records a decision. A justified approval needs adequate grounds. Effective governance must also give the decision practical force, including the ability to limit or stop the activity when its conditions no longer hold.

The distinction does not demand perfect foresight. An authorized decision can be reasonable on the available evidence and still have an unwelcome outcome. Conversely, an unauthorized action can produce a benefit without making the action legitimate. Governance must be able to examine the reasons, limits, and consequences rather than judge every decision solely by hindsight.

The trade-offs remain substantial. Additional scrutiny may slow useful work. Narrowing an activity may forgo a benefit. Keeping a stable requirement may preserve a protection while making adaptation more difficult. A clear account of these costs improves the argument; pretending they disappear weakens it. The purpose is to make justified choices under real conditions, with rights and binding duties kept in view.

Decisions that remain answerable

Purpose, direction, and particular action involve different governing decisions. Distinguishing them helps us see what kind of authority is needed, without prescribing the same institutional hierarchy everywhere.

Foundational decisions establish the arrangement's purpose, who may exercise authority, and what limits bind them. For frontier development, the questions include which interests must be protected, who may authorize consequential activity, and how that authority can be challenged. These are not settled merely by choosing a research method or obtaining better results.

Directional decisions concern what to pursue and where resources should go. An institution might invest in stronger evaluations, broader independent scrutiny, or a more limited development program. Choosing among these possibilities requires attention to the governing purpose, the available evidence, and the costs of each course. Such choices can legitimately involve disagreement.

Ongoing decisions apply those purposes and limits to particular matters. Should an experiment proceed? Does a change require additional review? What should happen when information on which permission depended is no longer reliable? These decisions need someone with responsibility and the means to act. They can also reveal that the broader arrangement needs reconsideration.

Accountability, learning, and stewardship matter throughout. Accountability concerns how power is justified and challenged. Learning allows experience to improve decisions and the questions institutions ask. Stewardship keeps attention on lasting interests and future choices. None can be postponed until after development has reached its destination.

The opening scenario leaves us with a practical distinction. Humans need not personally perform every research task to govern the work. They do need arrangements that maintain adequate understanding, allow independent challenge, and make intervention effective. Micromanaging work that AI systems or specialist teams could perform better would not necessarily strengthen those arrangements. Watching performance improve would not necessarily strengthen them either.

These responsibilities connect the book's three subjects: governance of AI; AI for governance; governance of AI-enabled governance. We must govern development and use. AI may help institutions analyze evidence and carry out authorized work. Those tools must themselves remain subject to scrutiny and limits, so that assistance does not become a new source of unanswerable power.

A responsible institution can acknowledge that its current understanding is insufficient without claiming that nothing can be done. It may investigate, strengthen oversight, reduce the permitted scope, or pause the affected activity as justified. The appropriate response depends on the stakes and obligations. Recognizing a limit can be an exercise of governing capacity rather than an admission that governance has become irrelevant.

That is the promise worth pursuing: institutions able to protect and enable under authority people can question. We should expect them to explain what their arrangements make possible, what they preserve, and how they recognize when those arrangements no longer suffice. The next question is which concerns about AI require that attention, and what evidence supports them.