Overview
Small local models fail in quiet ways. A server answers 200 with half the prompt cut, a model drops a tool call, or a big tool result is trimmed and the model invents the missing part. Agent Harness Lab turns each of those failures into code that can be tested, and adds an eval runner that says how often a model really gets a task right, with an interval. The only runtime dependency is Zod, and there is no framework.
Key Features
- Loop guards. A hard step limit, a guard against the same tool call repeating, a timeout per call and a total time budget, with retries only on transport errors and server errors.
- Context that stays honest. A tool result over a byte cap becomes its head, a visible marker and its tail, and the full text stays retrievable through a read tool, so a stub is never the only copy.
- A truncation check. It compares the prompt size the server reports with what was sent, and says so when the server cut the prompt.
- Structured output. The answer is validated against a schema, re-asked once with the exact error, and after that the run fails closed with a reason.
- An eval runner. It runs counted trials per case and configuration, reports 95 percent Wilson intervals, and counts broken trials (a server failure, a timeout, an empty reply) apart so they never read as a zero score.
Technical Achievements
On a small 4B model through Ollama with five tools, the model picked the right tool in every trial with the temperature set to 0, and missed in 7 of 90 trials with it left unset. The run also showed the opposite where I did not expect it: on one case, temperature 0 scored 0 of 30 and gave the same wrong answer every time, so greedy decoding turned a coin flip into a certain miss. The README reports that as an observation and does not claim a reason.
The first CI run on Linux found a bug that Windows had hidden. The call timeout timer was unref'd, so when a hung model call was the only thing pending, the event loop emptied and the process ended with no error. Another handle kept the loop alive on Windows. The timer now stays referenced, and a test runs the hang in its own process so it fails on any machine.
What I Learned
A rate without an interval misleads. The same cell moved between a run of ten and a run of thirty, and only the larger run could separate the effect of temperature from the number of tools. A suite that stays green with a guard removed is not testing that guard, which is why a mutation harness removes the guards one at a time and needs a test to fail each time.
Tech Stack
Language: TypeScript on Node.js, run directly with type stripping and no build step Validation: Zod Local models: Ollama Testing: node:test with generated table tests, seeded simulations and a mutation harness CI: GitHub Actions on Node 22