Instrument before you evaluate: write-path invariants for deployed human-agent systems
By Guglielmo Paesano and Romanus Berg. Accepted at the Human-AI Coevolution (HAIC) workshop, NeurIPS 2026.
Read it on OpenReview(opens in a new tab)The system it comes from
- Authors
- Guglielmo Paesano, Romanus Berg
- Venue
- Human-AI Coevolution (HAIC) workshop, NeurIPS 2026
- Accepted
- 30 Sep 2026
- Presentation
- Atlanta, 13 December 2026
- Full text
- OpenReview
What the paper argues
Whether a deployed human-agent system can be evaluated is decided by its write path: what it records, with what attribution, and what it may overwrite. The title is the recommendation. Decide what the system writes down before deciding how to grade it.
It is a field report from a production system in which people and AI agents both act. The paper names four write-path invariants and closes with a checklist a team can answer before it designs an evaluation.
What a write path is
The write path is everything a system does when it records something: which events it stores, who or what each record is attributed to, and which records it is allowed to change later.
Those three choices are usually made early, for engineering reasons, long before anyone asks how well the system works. By the time the question is asked, the answer can only come from what was kept.
Where it comes from
The system is SocrateOS, the production system of 1o1.org, a nonprofit building a dialectic AI platform for reflective decision making. I designed, built and run it. Romanus Berg, the founder of 1o1.org, is the co-author.
The SocrateOS case study describes the system itself: the three depths of dialogue, the memory graph, retrieval over a person's own documents and the checks that run before any model call.
How it reaches client work
Every agent I deliver keeps a record of each run: what came in, what the model was asked, what it answered and what the agent did, attributed and append-only. That record is what lets a client ask whether the system works and get an answer from data.
LLM evaluation for a system that is already running starts from the same place: what the system records today, and what it would need to record for the question to be answerable.
Cite as
Paesano, G., and Berg, R. (2026). Instrument before you evaluate: write-path invariants for deployed human-agent systems. Human-AI Coevolution (HAIC) workshop, NeurIPS 2026. openreview.net/forum?id=pExskhNGlq