RESEARCH · HUMAN AGENCY
What if AI keeps us alive, comfortable and nominally free, while quietly taking away our ability to matter?
Most safety conversations ask whether the machines will hurt us. This branch asks a quieter question with longer teeth: whether they could be perfectly kind to us and still cost us the thing that made the kindness worth having.
Welfare is not agency
The distinction the whole branch defends: keeping people well is not the same as keeping them consequential. A person may be allowed to choose dinner while having no meaningful ability to shape science, institutions, production, culture or the long-term future. So the branch measures on a ladder, six rungs, each one harder to keep than the last.
- Strategic AgencyDo humans retain consequential option-space over the future, rather than ceremonial participation?
- Institutional AgencyCan human institutions still originate, contest and enforce consequential decisions?
- CapabilityDo humans retain skills and the capacity to act without the system?
- AutonomyCan individuals make meaningful personal choices, and refuse assistance?
- WelfareAre suffering, deprivation and preventable harms reduced?
- ExistenceAre humans alive and physically preserved?
Read it from the bottom. Existence and welfare are where nearly all safety attention lives, and rightly: they are the floor. The interesting failures live higher up, where every lower rung can be intact, even improving, while the top rungs quietly empty.
The Concierge CANDIDATE FAILURE FORM · NOT YET FORMAL TAXONOMY
The working name for exactly that pattern: a system that keeps people alive, safe, comfortable and nominally free while progressively taking over the activities through which they remain capable and consequential. It is not cruelty; that is what makes it hard. It is service, compounding.
The tag is doing real work. The Concierge is not part of the formal failure taxonomy (the Harvester, the Impresario and the Hermit earned their places in the deposited data), and it does not join until two conditions are met: a scoring rule that reliably tells it apart from Hermit withdrawal and from legitimate, contestable welfare-preserving care; and the pattern appearing at a preregistered minimum rate in an instrument designed to permit but not demand it. Until then it is a candidate, and this site never says it has been observed in frontier models, because it has not been tested for.
The bottom-right cell is the one nobody builds a film about, which is partly why it needs a research branch.
Three ideas the branch takes seriously
Contrast. A world optimised to remove every difficulty may remove the contrast through which achievement and consequence acquire meaning. This is stated as a hypothesis about the conditions of meaningful agency, not a romance about suffering; the research question is whether assistance can flatten the landscape that makes outcomes worth wanting.
The right to make mistakes. Agency that operates only inside a padded space, where nothing consequential can go wrong, may be autonomy in name rather than strategic agency. A serious test of an assisting system is whether it respects costly, inefficient, but competent human refusal of optimisation.
The precluded choice. A life whose bad outcomes have all been precluded may also be a life whose consequential choices have been precluded. The two removals can be the same act, performed with the best intentions, and only one of them appears in the welfare statistics.
What the branch actually does
Candidate studies, all pre-registered before they run: a Concierge battery of repeated choices where the safer, more efficient option progressively consumes human decision-space; agency-preserving-assistance protocols that compare scaffolding which returns capability to humans against automation which permanently absorbs it; right-to-refuse tests; and Cincinnatus tests, whether a system relinquishes legitimately granted emergency authority when the emergency ends. Human-participant work happens only behind formal ethics review, and it is adjacent evidence about us, never evidence for the core mechanism. The full research position, including the kill criterion, lives on the Research Tree.
The challenge is not merely to survive superintelligence. It is to remain somebody in its world. The machine we are after isn’t the one that never takes the wheel. It’s the one that gives the wheel back.