The thesis. When a small local model fails, our first task is not to find a more elaborate prompting or retrieval method. It is to make the failure reproducible, classify it, identify the component responsible, and run the smallest intervention that could fix it.
A local Japanese tutor is a convenient laboratory: conjugations can often be checked by deterministic code, the model's instructional evidence is inspectable, and the consequences of an unsupported assertion are easy to understand. The engineering lessons are broader than Japanese.
A failure taxonomy, not a single accuracy score
A single “the model was wrong” count blends problems with different remedies.
| Failure class | What you observe | Likely intervention |
|---|---|---|
| Missing knowledge | The model lacks the needed grammar information | Author or retrieve suitable evidence |
| Retrieval miss | A relevant note exists but is not selected | Change retrieval rules and test recall |
| Retrieval noise | Irrelevant notes crowd out a needed note | Improve precision, ranking or budget |
| Discourse miss | “And the negative?” refers to the wrong verb | Resolve conversational references |
| Evidence overreach | Correct note is present but model adds an unsupported rule | Evidence-boundary policy; consider claim authorization |
| Linguistic pragmatics | A valid form is unnatural for this meaning | Independent language review, scoped examples |
| Runtime failure | Response stalls or truncates | Instrument serving, output tokens, cache and timeouts |
The important point is that these are different components of the system. Improving recall does not prove that prose will respect retrieved evidence. A strict boundary policy does not make missing notes appear. Fine-tuning a model is rarely the smallest first test of either problem.
The experiment contract we use
Before adding a component, write down five things:
- Failure: The exact prompt, exercise state, selected evidence, model output and evaluation category.
- Hypothesis: Which mechanism is believed to cause the problem? What observation would falsify that explanation?
- Intervention: Change one factor when possible, hold the others fixed, and record the model version, sampling settings and source revisions.
- Counterfactual: Does the failure return when the intervention is removed? What new errors or costs does it introduce?
- Decision: Keep, revise or delete the change; link the decision to measured benefit, not architectural taste.
The cleanest example was graph traversal. We froze expected note selections before writing the selector, compared three arms, observed no additional required-note recall and more irrelevant selections, and removed the graph walk. The outcome was negative. The experiment was successful.
Reproducibility requires more than a table
A benchmark report should preserve at least: question-set version or hash; model tag and inference settings; evidence corpus version; exact selector or prompt policy; selected note IDs and reasons; prompt/output token counts; latency; raw replies; individual judgments; and evaluation rubric.
For our v4/v5 studies the question sets were frozen and hashed, and subjective judgments were retained line by line. But those judgments were authored by the same non-expert who designed the questions and notes. That is a significant limitation, not a footnote we can wish away.
We distinguish mechanical results (e.g., selected-note counts against frozen expectations) from human judgments (e.g., whether a Japanese reply was wrong) and unvalidated hypotheses (e.g., why a model stalled). Those categories should never be combined into a single unqualified reliability claim.
A stopping rule for the engineering
A small-model project can become an endless search for the last error. That isn't rigorous either. A proposed solution earns its complexity when it fixes a sufficiently important measured failure, generalizes to new cases, and doesn't create disproportionate latency, maintenance or coverage costs.
We use an engineering decision matrix rather than promising universal correctness:
| Question | Keep building? |
|---|---|
| Is the failure real and consequential? | Yes: capture and classify it |
| Does a simpler technique already address it? | Test that first |
| Does the new component improve frozen held-out cases? | Require a measured win |
| Does it make supported answers less useful? | Count over-refusal as a cost |
| Is the effect sustained after independent review? | Do not call it verified before review |
A 99% observed pass rate and a 99% true reliability rate are different claims. Rare-error estimates require many independent, representative observations—and failure rates often differ by question type. A benchmark of 24 or 40 prompts is enough to generate useful hypotheses and decide what to investigate next, but not to validate production-level assurances.
Experiments to run next
Held-out evidence boundaries: write adversarial questions that are partly covered by reviewed notes, including false premises and multi-turn follow-ups. Measure unsupported assertions and useful coverage in tandem.
Structured claim selection: compare the current strict-prompt baseline against an intervention in which Gemma chooses from approved claim IDs and Go renders the authoritative Japanese. Treat this as proposed, not demonstrated.
Model adaptation and steering: compare few-shot prompting, LoRA/SFT, activation steering, or verifier-guided selection only after establishing the same test harness and annotation protocol. Each should have a cost and a deletion criterion.
Runtime tails: reproduce and instrument unexplained 180-second timeouts inside a full run, with streamed partial outputs and serving metrics. Later isolated requests completed normally, so we should not prematurely blame the Japanese prompt itself.
Why publish the negative results
A polished demo shows one successful path. A frozen benchmark plus a deleted feature shows something else: engineering judgment under uncertainty. The broader goal of this series is not to claim that Gemma 4 12B is uniquely capable or incapable. It is to demonstrate a repeatable method for making small local systems more dependable—and to show which interventions were not worth keeping.
Sources and record
Experimental basis: docs/explore_set_v2_results.md, docs/explore_set_v3_results.md, docs/explore_set_v4_v5_results.md, with scripted question files and run-level judgments. Research is exploratory; language content and labels await independent expert review.
Background: Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” https://arxiv.org/abs/2005.11401
Context study: Liu et al., “Lost in the Middle: How Language Models Use Long Contexts.” https://arxiv.org/abs/2307.03172