How Do We Know What Is Happening?

An account that misses the question

Return to the hypothetical public institution using AI to change how people obtain a service. Imagine officials receiving a polished report on its recommendations, explaining methods, results, and limitations. But the system now has permission to make some changes directly. The report cannot establish which changes it made, who was affected, or whether the safeguards supporting that permission still apply.

The results need not be false. They answer an earlier question. An account adequate for judging advice may be inadequate for judging action, even with an unchanged model. Better writing or analysis of the same information would leave the missing evidence missing.

A reporting requirement can still be fulfilled after it ceases to supply the grounds a decision needs. Those receiving the report should recognize that change, request appropriate evidence, and examine what the revised account establishes.

Chapter 4 left us with different people contributing different accounts. An operator knows something about the system; a worker knows something about its use; a person seeking the service knows something about its consequences. Their contributions can be valuable without being complete or equally well supported. A legitimate decision needs credible grounds for judging them.

The question is what relevant actors must know, and what makes that knowledge credible. We will move from what their responsibilities require to what evidence can establish, then consider how the questions and sources should improve. The final issue is what that understanding means for continued permission.

What must be known and by whom

AI governance must maintain awareness as capability grows. Awareness means more than receiving information. In this book, justified awareness is a sufficiently current, evidence-supported, and correctable understanding of consequential activity, its effects, and the limits of available knowledge. It includes useful opportunities and emerging capabilities, as well as threats. Missing a way to improve access or reduce an avoidable burden can matter to a governing decision too.

What needs to be known follows from the responsibility being exercised. A developer assessing a capability, an operator using it, and an independent evaluator examining a safeguard face related but different questions. An affected person may need to understand the grounds of a decision well enough to question it. None of them needs to know everything. Each needs knowledge sufficient for their role or affected interest, including ways to discover a consequential omission.

That knowledge concerns the operating arrangement. What can the system do with its tools and permissions? Which other activities can affect it? Who experiences the consequences? A description of the model alone may leave the important relationship unexplained. Likewise, records of separate activities may fail to establish what their interaction means. The object of inquiry should follow the consequential activity, even when institutional responsibilities divide it into smaller parts.

Visibility supplies access to evidence that can be examined. A reviewer who receives only the operator's summary may be unable to test a disputed claim. Access to the underlying records could help, but obtaining them would not automatically make their significance clear. Expertise, time, and intelligible explanation can be needed before access becomes useful understanding.

Different roles can justify different access. An evaluator may need sensitive detail that should not be made public. A person challenging a decision may need its relevant reasons without receiving another person's private records. The institution should provide a usable basis for scrutiny while protecting privacy, confidentiality, and security. Invoking confidentiality must itself have grounds; it cannot make every consequential assurance impossible to question.

Currentness also depends on the question. A sound evaluation can become an inadequate basis for permission when a tool, interaction, or operating condition changes. Some knowledge remains dependable for years; some must be reconsidered before the next consequential action. The appropriate interval follows from what could change and when a response would still matter, rather than a demand for constant revision.

Correctability gives knowledge a route to improve. A conclusion should identify its grounds and leave room for evidence that challenges them. A reviewer who acknowledges an unresolved question makes responsible inquiry possible. That candor is valuable when it changes what is investigated or permitted. Merely attaching a disclaimer while treating the conclusion as settled would preserve the appearance of caution without its governing consequence.

A record, a test, and an explanation

Consider a separate, minimal hypothetical. Investigators observe an unexpected action after a system's permission changes. They can establish the recorded sequence. They still need to ask whether the permission change contributed to the action, for example by enabling an existing capability to be used, or whether the timing was coincidental. Knowing when an event was noticed does not settle why it occurred.

A bounded test could add evidence by examining behavior with and without that permission under specified conditions. Its result would still need interpretation, including whether other differences could explain it and whether it transfers to the activity being governed.

Different evidential achievements therefore support different claims:

Investigators should state which conclusion they can support and which further question remains. A test may help examine a proposed explanation; a missing record may limit the reconstruction. These forms of evidence can work together without becoming interchangeable. More detail in one does not necessarily repair a gap in another.

The date of discovery needs particular care. A behavior first observed after an update may have been possible earlier, when nobody looked for it under the relevant conditions. Conversely, the update may have made it possible. The investigation should distinguish those possibilities where they affect the decision. First observation establishes a discovery, not necessarily an origin.

An AI-generated explanation belongs within this inquiry. It may offer a useful hypothesis or make an account easier to follow. Its fluency does not show that it faithfully describes the process that produced the action. Where the explanation matters, investigators need grounds for checking it against relevant evidence and alternatives.

This does not require every decision to wait for complete causal understanding. Evidence can establish that a consequential action occurred or that a safeguard failed under specified conditions while leaving its full cause uncertain. Those findings can justify further inquiry or limits appropriate to the stakes. The institution should explain what its judgment rests on, preserving the distinction between an observed event and an inferred cause.

That distinction helps people obtain a meaningful answer. Someone affected may need correction before a complete explanation is available. The institution should be candid about what it knows while examining what it can responsibly do. An unresolved cause does not remove its duties to investigate, preserve relevant evidence, or respond within its authority.

Improve the evidence we ask for

Evidence requirements shape what can become known. If the institution asks only about recommendation quality, it may learn little about the changes its system now makes directly. Better analysis of that submission cannot reconstruct information never produced or preserved. Governance must improve its inputs as well as its conclusions.

We value useful evidence over comprehensive reporting.

Comprehensive reporting has substantial value. Broad records can expose omissions, support comparison, and preserve information whose importance becomes clear later. Common disclosures can make selective accounts easier to detect. A report organized solely around the writer's chosen question could exclude the fact that most needs scrutiny.

The priority therefore cannot mean requesting less whenever less is convenient. Useful evidence may require a wider disclosure, an additional test, or access to an unwelcome finding. It may also require removing repetitive material so that the recipient can examine what matters. The governing question is what information could change the decision or make it meaningfully reviewable, and how to obtain it reliably at a proportionate burden.

There is a serious limit to tailoring. No institution can identify every future question in advance. Narrow requirements may discard the context needed to discover a surprise. Preserving relevant records, setting proportionate minimum disclosures, and maintaining independent inquiry can keep the account open to questions its author did not anticipate. Complaints and unexpected results matter partly because they can reveal what the existing requirement never asked.

Reporting involves different kinds of work. Producing evidence might mean conducting a test, recording an action, or investigating an experience. Checking it means examining whether it supports the claim, including limitations or contradictions. Communicating it means making the account intelligible to the appropriate recipient. A readable document can contain verified evidence, but readability alone does not establish that verification occurred.

AI assistance creates a possibility worth examining across this work. Imagine a reliable tool helping assemble an account from preserved records, tailor explanations for different recipients, or identify a question that earlier reports omitted. Such help could make a more informative requirement practical. It should earn that role through demonstrated usefulness for the task, with supporting material accessible for appropriate checking. Assistance with testing or verification needs evidence of its own reliability.

What better reporting makes possible

Reliable assistance may make a previously burdensome question practical to ask. That can improve what governance learns. Clearer writing cannot make an unperformed test exist, however. The useful opportunity is to improve the question and the evidence behind the answer together.

The total burden remains important. Less preparation for a reporting party may mean more checking for the recipient. Integrating records, protecting sensitive information, maintaining the method, and correcting errors all require work. A sophisticated report that overwhelms a small oversight body may weaken scrutiny despite improving presentation. Assessing improvement requires attention to whose work decreases, whose increases, and whether the resulting understanding is better.

Collection has costs beyond staff time. Gathering more personal information can intrude on people, create new security obligations, or discourage participation. Requiring affected people to document every difficulty may shift institutional work onto those least able to perform it. Where another reliable means can answer the question, governance should consider it. Where sensitive evidence is necessary, access and protection should fit the duty rather than expand without limit.

Inputs also include expertise, resources, authority, and people's experience. A more demanding evidence request is of little use if nobody can interpret the response or seek clarification. Incentives matter too: an arrangement that rewards reassuring accounts while making unwelcome findings costly gives people reasons to conceal uncertainty. Improving what enters a decision includes making candid contributions possible and ensuring someone can examine them.

In the opening service, an accepted template could shape the evidence generated as well as its presentation. If it asks about recommendations, staff may learn to preserve and check recommendations while direct actions receive no equivalent account. Reviewers may be equipped to accept that familiar evidence, and completion of the template may count as discharge of the reporting duty. Faster production would then make the established omission easier to repeat.

Changing the question requires a route into those arrangements. The reporting recipient needs authority to request evidence about direct actions, and both sides need the capacity to produce and examine it. A worker’s concern or an independent finding should be able to prompt that reconsideration even when the old report is complete. Useful records and comparisons can remain while the missing inquiry is added. The test is sufficiently current, checkable understanding of the permitted activity; a continuous stream of data could leave the same question unanswered.

Who can question the account

Credible knowledge cannot depend exclusively on one institution noticing and describing its own omissions. Developers and operators have essential knowledge, but independent evaluators, workers, users, and affected communities may encounter evidence unavailable to them. Governance should preserve routes through which those contributions can change an inquiry.

Chapter 4's hypothetical question about people omitted from a completion-time measure shows the difference. The contribution gave officials a reason to examine access; it did not establish the cause of any exclusion. Here the institution needs to investigate whether its records represent those people and what further evidence is required. It should carry its share of that work rather than ask the person raising the concern to prove the whole institutional failure unaided.

A complaint, testimony, or independent finding should receive weight appropriate to what it establishes. Conflicting accounts need examination, not automatic deference to whichever arrives with more technical language. The institution may find an error in either account or establish that each describes a different part of the activity. Relevant disagreement should remain visible while inquiry continues.

Independence has practical conditions. A reviewer needs competence, resources, and access sufficient to pursue an unwelcome question. If the operator chooses all the evidence and can prevent follow-up, describing the review as independent supplies little assurance. Another review delegated by the same authority may be useful without providing an independent basis for challenge. The distinction concerns what can actually be examined and questioned.

Protected reporting can contribute to that basis. A worker may have relevant information and reasonable grounds to fear identifying themselves. Confidential or anonymous reporting can allow inquiry that would otherwise be obstructed. The account still needs appropriate checking. Accountability attaches to the consequential authority and its duties to investigate and respond; it should not be confused with requiring every reporter to become publicly identifiable.

Shared evidence also need not produce one authorized interpretation. Evaluators can disagree about what a test establishes, and affected people can disagree about the burdens they consider acceptable. Further inquiry may resolve a factual dispute while leaving a legitimate choice unsettled. Better knowledge improves the grounds of judgment without conferring authority on whoever supplies it.

Computational coordination needs these distinctions to survive across institutions. A finding passed from an evaluator to an operator should retain its supporting grounds, the conditions examined, and what remains unresolved. A shared method that strips away that context could spread a mistaken conclusion more efficiently. Independent sources and appropriate access must remain available to question the connected account. Chapter 6 considers how computation could make this governing work practicable.

Independent inquiry is valuable for opportunities as well as deficiencies. A user might identify a useful capability overlooked by the original evaluation; a worker might propose a less burdensome way to obtain reliable evidence. Those contributions need checking too. An arrangement open only to reassurance would be weak, but one attentive only to failure could also miss a worthwhile improvement.

The public expectation should be an account that can encounter correction. Relevant people need practical ways to bring evidence forward, and competent institutions need ways to pursue it. Honest uncertainty can then become a starting point for useful work rather than a reason to dismiss the contribution or declare the inquiry finished.

What makes continued oversight credible

The remaining question is how this knowledge bears on permission. The book proposes an operating condition: no AI system should operate at a scope, scale, or level of autonomy beyond the responsible institutions' demonstrated capacity to oversee it and constrain its consequential effects. This is a normative commitment, not a statement of existing law or a demand for perfect prediction.

Its significance is that uncertainty material to authorization must inform the grounds and conditions of permission. Existing safeguards may already accommodate it; where oversight is inadequate, the institution must strengthen its capacity, constrain the affected activity, or combine both. An institution cannot establish adequate oversight merely by describing what it does not know and authorizing the same activity regardless. The relevant limit depends on the stakes, the safeguards that remain effective, and the possibility of correction. An unexplained variation in an advisory response and an unexplained capacity for irreversible action require different grounds for permission.

A material loss of visibility on which authorization depended is itself a reason to reassess the affected activity. Waiting for proven harm could leave the institution unable to perform the preventive duty that justified permission. Not every missing record demands suspension. The question is whether the loss removes evidence essential to the current basis for oversight, and what response is justified under the circumstances.

An assurance that oversight remains adequate should therefore be open to examination. What consequential activity is covered? What departures could the arrangement detect, and what could it miss? Who receives the evidence, can they understand its significance, and can someone with legitimate authority respond effectively in time? Those claiming adequacy should provide grounds appropriate to these questions and allow independent challenge.

An absence of recorded incidents may reflect effective safeguards, limited exposure, or gaps in detection. Tests and exercises can help establish what oversight can reveal and how a response works, within their conditions. They do not prove reliability in every future circumstance. The institution needs to explain the support for its claim and its reach.

Timing connects knowledge to effective constraint. An accurate history received after an irreversible consequence can support investigation without having provided prevention. Limits may be justified before visibility is adequate and may make the permitted activity easier to examine. Evidence and constraints can develop together; observation alone supplies neither authority nor remedy.

Responsibility for these judgments must remain identifiable. The operator should not be the sole judge of adequate oversight. Reviewers need appropriate access and the ability to question its basis. If a change invalidates an assumption supporting authorization, the relevant authority must reconsider the affected permission. Discovering a problem does not confer power to rewrite the arrangement.

Stronger inquiry can be costly; narrower permissions may delay or withhold useful work. Those costs deserve reasons alongside the protected interests at stake. Governing capacity includes strengthening the evidence, preserving what remains justified, and explaining necessary limits while uncertainty persists.

We should expect institutions to show what supports their understanding, how consequential omissions can be discovered, and what changes when that understanding is insufficient. Knowledge limits then have practical force. How can computational assistance help make oversight demonstrably adequate, while its own contributions remain open to scrutiny?