// field notes · build in public

Field Notes — Build in Public

Short, dated notes from the lab. Some are polished, some are raw — honesty is a feature. I ship the failures as loudly as the wins, because the dead ends are usually the part worth learning from.

This is the raw log behind everything on this site: evaluation harnesses, swarms that didn't negotiate, benchmarks I ran so you don't have to, and papers I read that changed how I build. Nothing here is a highlight reel.

Last updated · May 2026 — reverse chronological

Setting up an evaluation harness for agent tool-calls

An agent that can't be graded is a guess. This month I built a small harness that runs a tool-calling agent against a fixed task set and scores the outcome, not the text it narrates. The lesson: check the side effects — did the file actually get written, did the API actually get called — because a model will confidently describe work that never happened. It started as a script and quietly became a tiny framework, and now I wire it into the lab so every agent change gets a before/after number. No LLM-judge theater; just environment-grounded verifiers. It's unglamorous, and it's the most useful thing I've shipped all quarter.

What failed this week: the swarm negotiation loop

The idea was elegant — several agents, one shared core, message passing — and the failure was embarrassingly predictable. The agents agreed beautifully with each other and produced nothing useful. Consensus without a ground truth isn't coordination; it's a meeting. They negotiated for twenty-five turns and settled on a plan no one was accountable for. The fix was boring: assign ownership, add a verifier, cap the rounds. I'm writing it down because the next elegant idea deserves to fail this fast, not three weeks later. Build in public means showing the dead ends too.

RAG vs long-context: a quick benchmark I ran

Does RAG still matter when context windows are huge? Short answer — yes, and for an unglamorous reason: retrieval isn't about fitting text in, it's about where the model pays attention. I ran the same questions against a long-context model with a 200k window, once with the whole corpus stuffed in and once with RAG pulling the relevant slices. On factual lookup the gap was small. On multi-hop reasoning and traceability — knowing which source the answer came from — RAG won clearly. Long context is a big memory; RAG is a spotlight. I reach for both, and I've stopped pretending they're the same problem.

The verifier is the product

Spent the month on the unfashionable part of the stack: the check that decides whether an agent actually did the job. The pattern that keeps proving itself — model the environment as a first-class citizen, expose a clean action API, and grade the agent inside that world. No rubrics, no vibes: a sandbox that can say "no, that didn't work." The swarm is the fun part; the verifier is the product.

The connectomics essay took longer than expected

That's the whole note, honestly — I estimated a weekend and spent a month. The writing itself was the fast part. The slow part was deciding what I actually believe: whether the brain's sparse, deliberate wiring is a useful metaphor for multi-agent systems, or just a pretty analogy I wanted to be true. I rewrote it four times, cut half of it, and ended with something more honest and more boring than I planned. That's usually the version worth publishing. It became the seed of the whole swarm thesis on this site, so I wanted it right rather than quick.