Guglielmo Paesano AI engineer and founder of AIGROW

I build AI systems that do people's work, and I measure where they still need people.

contact@guglielmo.ioDiscord, 4K members

LLM evaluation for a system that is already running

I tell you, with numbers, where the AI system you already run can be trusted and where it still needs a person, starting from what it records.

You bring
Read access to the system, its logs and its data model
You get back
A written report: what can be measured, what was, and what to change
How it is cut
Two milestones: the instrumentation review, then the measurement

What you get

  1. A review of the write path: what the system records about each run, with what attribution, and what it is allowed to overwrite.
  2. The list of questions you cannot answer today because the data was never written, and the smallest change that fixes each.
  3. A test set built from your own cases, with the grading rule written down so someone else can apply it.
  4. Pass rates per task with their uncertainty, split by the cases where the system works alone and the cases where a person had to step in.
  5. A written recommendation: which steps to leave to the model, which to gate, which to give back to a person.

How the work is done

Instrument before you evaluate

Whether a deployed system can be evaluated at all is decided by its write path. If it overwrites a record instead of appending, or cannot say whether a person or the model produced a field, no metric computed afterwards means what it claims. So the first milestone reads the data model, not the model outputs.

Measure the task, not the model

A leaderboard score says little about your job. I grade the system on your cases, per task, because the same 36 models passed one office task 96 times in 100 and another 22 times in 100, and that split is what you need to know.

Report uncertainty

A pass rate from forty cases is an interval, not a point. Every figure in the report carries its sample size and its range, and the report says which differences are real and which are noise.

Where I have done this

Who it fits

A good fit when

  • The system is live, or about to be, and people act on what it outputs.
  • You can give read access to logs and the data model.
  • Somebody has to sign off on it and wants more than a demo to sign against.

Not a fit when

  • You want a benchmark number for a press release.
  • Nothing is built yet: then the right engagement is the build, with evaluation designed in.

Questions about AI evaluation

What is the difference between LLM evaluation and testing?
A test checks that code does what it did yesterday. An evaluation estimates how often a system that is not deterministic gets a task right, on cases that represent real use, with a stated uncertainty. You need both: the tests run on every push, the evaluation runs when the model, the prompt or the data changes.
What does "instrument before you evaluate" mean in practice?
That the first thing to check is what your system writes down. If a run does not record its input, the model output, what a person changed and who changed it, as separate facts that are never overwritten, then no later analysis can separate what the model did from what a person fixed. Adding those records is usually a small change, and it comes before any metric.
How long does an evaluation take?
The instrumentation review is short, because it reads a data model and a sample of records. The measurement depends on how many tasks there are and whether graded cases already exist. I scope both in writing after reading your brief.
Will you tell us if the system is not good enough to ship?
Yes. The report states where the system needs a person, with the numbers behind it. That is the part of the work I care about most, and it is the reason to ask someone outside the team.

Tell me what you need built.

Write what the system has to do, what exists already and when it has to be live. I read every brief myself and answer in writing, with a scope and a price for each milestone.

Or write to contact@guglielmo.io

The human check appears here.

No calls and no sales sequence. One written answer from me.