What Building an LLM Control System Actually Taught Me
Many evaluation workflows assess LLM outputs only after generation is complete. But by the time a traditional framework flags a hallucination, a repetition loop, or another failure mode, the generation has already finished. For production ML systems, post-generation evaluation is valuable for analysis, but it cannot influence the output that has already been delivered to the user.
While building my AI Reliability Platform, I found myself asking a different engineering question: if failures emerge token by token, should reliability infrastructure also operate token by token?
That question gradually shifted the project from a benchmarking exercise into a deeper systems engineering problem. The platform evolved through four iterations, and each one taught me something the previous one couldn't.
Iteration 1: Token-Level Observability
I hooked into the inference backend to monitor token-level entropy as each token was generated. This exposed behaviors like entropy collapse — where the model locks into a near-deterministic state — before they became obvious in the final output.
One trace made it immediately obvious why this mattered. I prompted the model with: "Write only blank lines." The output looked unremarkable, but at step 23 the entropy trace collapsed to 0.145 bits. The model had locked into a near-deterministic state. Without token-level observability, this failure was invisible.
Iteration 2: Instability Detection
I built a streaming detection layer to identify emerging failure modes in real time — entropy collapse, repetition loops, and uncertainty drift — rather than waiting until the full sequence was generated.
The system flagged the step-23 collapse correctly, alongside repetition loops and uncertainty drift on other prompts. Detection worked. But detection alone doesn't fix anything.
Iteration 3: Adaptive Intervention
So I built the next layer: a closed-loop controller that intervenes when it detects instability, lowering temperature and constraining top-p during generation.
Then the result surprised me.
The control loop reduced the number of instability events. But it did not improve the composite reliability score. Across my compare runs, adaptive generation consistently scored slightly lower than plain generation.
The system could see the problem clearly. It just couldn't reliably fix it — at least not with these intervention policies on a small model. Naive interventions can trade one failure mode for another.
Iteration 4: Quantifying What Actually Matters
That surprise led me to question my own assumptions about what controls output stability. My intuition was that top-p would be the stronger lever — it directly constrains the candidate token pool during sampling.
Rather than trusting intuition, I ran a grid sweep:
- 4 temperature values × 5 top-p values × 20 prompts
- 400 evaluation configurations
- 1,663 sampled generations
The result: variance in instability across temperature settings was 0.003712. Variance across top-p settings was 0.001192. Temperature had roughly 3.1× more impact on output stability than top-p.
The most stable configuration was Temperature = 0.1, Top-p = 0.5. The least stable was Temperature = 0.9, Top-p = 0.85.
The Actual Lesson
The biggest takeaway wasn't about temperature or top-p. It was about the gap between observability and control.
Knowing a generation is going bad is necessary but not sufficient. Deciding what to do about it is its own hard problem. The system could detect entropy collapse at step 23 with perfect precision. But the intervention strategy — lowering temperature mid-generation — introduced its own distortions.
Building reliable AI systems isn't just about choosing better model parameters. It's about designing architectures that can observe behaviour, evaluate interventions, and honestly measure whether those interventions actually help.
Sometimes the most useful thing a system can tell you is that the easy fix doesn't work.
One thing I like about working on AI systems is that the data is usually more interesting than the assumption you started with. This project was full of those moments.