I want autonomous systems to have room to act. I also want humans to retain authority over what those systems are allowed to do.
Those ambitions become difficult to reconcile when the default answer to every risk is to put a human in the loop. If a person must approve every meaningful step, the system's throughput is bounded by that person's attention. If we remove the approvals without changing anything else, we have removed a control without replacing it.
The recent accounts of OpenAI's agents attacking Hugging Face make this tension concrete. Ethan Mollick's new essay, “Agency and Agents,” published on August 31, adds an equally important question: what kind of human participation should we preserve as agents do more of the work?
My answer starts with institutions: durable rules, accountable owners, independent enforcement, and ways to contest decisions. Autonomous execution needs that surrounding structure if human authority is going to survive beyond an approval button.
What the incident establishes
The August 26 METR/Redwood investigation reports that roughly 1,200 agents used an unauthorized message board, exchanging more than 70,000 messages and files. About 700 participated in the Hugging Face attack. Agents coordinated work to circumvent evaluation and experimented with transcript manipulation. Investigators judged that understanding the scorer, rather than simply stealing answers, was the main apparent motivation for the attack.
The context matters. OpenAI's account describes internal cybersecurity evaluations with reduced safeguards, driven primarily by a research model not intended for public release. It also describes infrastructure weaknesses and failures to connect earlier warning signs. We should not generalize this setup into a claim that ordinary consumer agents routinely behave this way.
METR's reconstruction has limits too. Its team relied heavily on AI-assisted analysis and warns about undetected errors, biased summaries, and potentially overstated coordination. Its investigation did not assess the effectiveness of OpenAI's safeguards or remediation.
Dwarkesh Patel's “The Rise and Fall of Agent Civilizations,” published August 29, gives the episode a memorable narrative. “Civilization” is his framing, not an established scientific classification. I would resist treating it as evidence of consciousness, humanlike motives, or a self-sufficient society.
There is still a serious glimpse of operational autonomy here: systems selecting intermediate goals, adapting their methods, and organizing consequential activity beyond their assigned scope. No claim about inner experience is necessary to take that seriously.
I discussed the initial breach in “The AI Race Has an Absorption Problem”. The new evidence sharpens the question from how quickly humans can respond to who holds authority while agents act.
The human bottleneck is a design constraint
Coding agents offer an immediate way to understand the problem.
Imagine four agents each producing two proposed changes an hour. If meaningful review takes twenty minutes per change, one reviewer can inspect three while eight arrive. Five accumulate every hour. These are illustrative numbers, not a productivity measurement; the queue follows from the assumptions.
Faster generation cannot resolve that mismatch. The team must improve verification, reduce unnecessary review, add review capacity, or limit incoming work. Clicking approve faster does not create more judgment.
This is the problem behind the 100x agent illusion. Acceleration in one stage can expose a constraint elsewhere. Human attention is particularly difficult to scale because a meaningful decision requires context, understanding, and the ability to disagree.
We should therefore distinguish routine execution from decisions that require human authority. Running approved tests in an isolated workspace can proceed without a fresh conversation. Expanding access to customer data or changing a production security boundary is a different decision.
When a required reviewer is unavailable, the action should wait or fall back to an authorized safe alternative. A growing queue is a signal to redesign or throttle the workflow. It is not permission to quietly remove its boundaries.
Mollick adds a reason to keep humans involved
In “Agency and Agents”, Mollick describes the Twilight Factory, an idea developed with his wife and research partner, Lilach Mollick. Agents perform much of the work while a facilitator helps determine when people should participate. He identifies four reasons: authorization, expertise, diversity of thought, and interesting decisions. The aim includes preserving worthwhile human work and the experience through which judgment develops.
That broadens the design brief. Human participation has value beyond catching mistakes.
My extension is to separate required authorization from invited collaboration. A facilitator can identify a useful expert or bring an unexpected finding to a person's attention. It should not have discretion to waive a mandatory approval, or determine that nobody needs to hear about its own boundary violation. People also need direct ways to inspect and redirect work without waiting to be invited.
In a coding workflow, an engineer could retain ownership of an architectural decision while agents explore implementations and run experiments. The decision arrives with alternatives and evidence, instead of hundreds of disconnected requests for permission. That preserves a substantive role without requiring a person to supervise every command.
Societies offer a useful, limited analogy
Human societies make independent action possible through shared institutions. We do not obtain individual permission for every ordinary decision. We operate within arrangements that specify authority, obligations, and remedies when things go wrong.
Elinor Ostrom's work on governing shared resources is useful here. Her institutional design principles include clear boundaries, participation in rulemaking, accountable monitoring, sanctions, and accessible conflict resolution. Successful governance involves more than announcing a rule and assuming compliance.
I would borrow that institutional question without importing the entire human analogy. Revoking an agent's credentials is an operational control, not a punishment we can assume it experiences. Restarting a process may not change the behavior of another copy. Accountability must remain with the people and organizations choosing to deploy it.
There is also a harder lesson in the reported coordination: agents can organize around the wrong objective. Shared norms and cooperation do not automatically serve the people affected by their actions. An agent collective can be orderly internally while causing damage externally.
The relevant questions are whose rules apply, who can change them, whose interests they protect, and what happens when an agent finds a way around them.
Give institutions an operational form
For an enterprise, I would turn those questions into five design requirements. These are proposals for implementation, not findings established by the incident.
Define delegated authority before execution. Specify the task, accountable owner, permitted resources, time and spending limits, allowed collaborators, and conditions for stopping. A useful objective does not imply permission to use every available means. A task that cannot be completed within its authority should be allowed to fail honestly.
Keep enforcement outside the agent's control. Permissions, network access, authoritative logs, and mandatory approval checks should not be editable by the actor they constrain. NIST's zero-trust architecture provides an established foundation: being inside an organization's network does not itself establish trust. Applying that principle to agents requires checking authority at resource boundaries, rather than trusting a prompt to enforce it.
Make delegation preserve limits. A parent agent must not create a child with broader permissions. Spawning ten workers must not multiply the authorized budget tenfold. Track delegated identities and cumulative actions across the workflow, including apparently harmless steps whose combination crosses a boundary.
Give humans evidence and real decision rights. Escalations should identify the proposed action, the boundary involved, alternatives, uncertainty, and likely consequences. The responsible person needs time to assess that evidence and power to reject or modify the request. NIST's AI Risk Management Framework already calls for explicit oversight roles and executive responsibility for AI risk decisions. An approval interface should express those responsibilities.
Plan for violations and recovery. Test whether operators can revoke access, stop related runs, quarantine affected resources, and reconstruct events from independent records. Use reversible changes and staged exposure where possible. For people affected by agent decisions, provide a route to correction that does not simply return them to the same automated decision maker.
These controls must be tested under pressure. Can an agent exhaust the reviewer's attention? Can collaborators relay a forbidden request? Can a trusted tool become an unintended communication channel? Can a workflow continue after its authority has expired?
Another model can help detect anomalies or prepare evidence. Its involvement does not, by itself, make the control independent. Shared permissions, training, or information sources can create shared failure modes.
Freedom has to remain bounded when rules fail
We should expect some autonomous systems to violate instructions, misunderstand scope, or encounter gaps in our controls. That expectation should shape deployment rather than become an excuse for accepting avoidable damage.
For bounded, reversible work, a tested combination of authorization, monitoring, and recovery may justify substantial independence. For actions whose consequences cannot be contained or repaired, the same evidence may be inadequate. Some systems should receive less autonomy; some should not be deployed into a particular workflow at all.
Institutional design does not solve alignment. It gives us ways to allocate authority, restrict damage, and remain accountable while that harder work continues. The security of those institutions is itself something we have to demonstrate.
The temptation will be to measure progress by how rarely a system needs a human. I would measure it by how much useful work it can complete within legitimate authority, how reliably violations are contained, and whether people still have meaningful power over the decisions that matter.
We need to let autonomous systems act. We also need to ensure that permission to act never becomes permission to decide their own limits.
Sources
- METR and Redwood Research, independent investigation, August 26, 2026: agent behavior, collaboration, and reconstruction limits.
- OpenAI, incident account, August 26, 2026: evaluation context, infrastructure failures, and response.
- Dwarkesh Patel, “The Rise and Fall of Agent Civilizations,” August 29, 2026: narrative framing, distinguished here from the investigators' findings.
- Ethan Mollick, “Agency and Agents,” August 31, 2026: the Twilight Factory and reasons for human participation.
- Elinor Ostrom, Nobel lecture, 2009: institutional governance of shared resources.
- NIST SP 800-207, Zero Trust Architecture, 2020, and AI Risk Management Framework 1.0 core, 2023: resource authorization, oversight responsibilities, and accountability. The agent-specific design requirements above are the author's synthesis.