More context was not better

The question. Does adding more instructional context make a local 12B model a better teacher? We expected that relevant concept notes would help. The more interesting question was whether giving the model all our notes would help even more.

The finding. In one 22-turn exploratory test, supplying the notes linked to a teaching exercise reduced suspected errors from six replies to two or three. Supplying all twelve notes did not improve that result. It pushed median first-reply latency from 10.0 seconds to 60.1 seconds.

Decision: Retrieve a small set of relevant notes. Do not use the context window as a dumping ground.

The system under test

Our reference application is a local Japanese-language tutor backed by gemma4:12b-mlx and a deterministic Go engine. Go owns supported verb forms, grading, and exercise facts. The language model handles conversational explanations. The teaching notes are provisional: they were drafted with AI assistance and had not received independent proficient Japanese review at the time of these tests.

This distinction matters. The experiment measures what happens when an LLM receives different amounts of candidate instructional evidence. It does not establish that the resulting Japanese explanations are all correct.

We compared three otherwise similar prompt conditions. Each contained the exercise's reference facts. The difference was which concept notes accompanied them.

The model used a 32K context window and temperature 0.2. The questions were scripted. Each was run once in each condition. A single non-expert reviewer judged the answers.

Why linked evidence worked

Imagine an exercise asking for the て-form of 泳ぐ (oyogu, “to swim”). The correct form is 泳いで (oyoide, the て-form of “swim”). A note scoped to the ぐ → いで transformation supplies a rule the model can apply directly. A generic lesson about Japanese verb endings may be related, but it is less useful for this exact question.

Other improvements involved distinguishing godan and ichidan verbs and avoiding an invented historical explanation of んで (nde, a suffix pattern appearing in some て-forms). In these cases, the linked notes contained the relevant information or explicitly said that the historical explanation was not established.

The successful mechanism was not “more tokens.” It was higher evidence relevance at the point of generation.

exercise facts + question
           |
           v
     link-rule lookup
           |
           v
      relevant notes
           |
           v
     compact prompt to Gemma

Why the full corpus didn't help

The all-notes condition increased the mean prompt from about 2,165 tokens to nearly 16,800, yet produced no clear quality gain. It also introduced one cut-off reply and one reply with garbled kana. Those observations do not prove that longer prompts generally harm Japanese performance; they show that this corpus-wide context expansion was an expensive intervention with no demonstrated benefit.

Related research offers a plausible mechanism. Lost in the Middle found that some language models use information less robustly when it sits in the middle of long contexts. That paper is background, not a causal explanation demonstrated by our run. We did not randomize note position or isolate competing hypotheses such as attention dilution, prompt processing cost, or sampling variance. [1]

The latency cost was easier to identify. A separate probe using Ollama's counters measured roughly 400 input tokens per second and roughly 60 output tokens per second in this setup. The longer linked prompts increased prompt-reading time; output generation time changed much less. The all-notes arm's timing breakdown was not instrumented separately, so we do not claim a fully measured causal decomposition of its 60-second median.

What we changed afterward

Rather than write a larger prompt, we built a typed evidence selector and a compact renderer. The selector decides which notes are relevant to the current exercise and question. The renderer sends the model a subset of their teaching fields. The same source notes can therefore serve multiple exercises without duplicating them into every exercise record.

The useful engineering distinction is between authoring a knowledge base and constructing a turn-specific evidence packet. Those are different data products and should be measured separately.

When to use this technique

Use scoped retrieval and evidence packing when your model can answer some questions correctly, but performance deteriorates as you insert more reference material; when the same documents support many distinct user questions; or when context processing dominates latency.

It is a poorer fit when the question genuinely requires integrating many documents. In that case, arbitrarily capping retrieved notes can remove necessary information. Our two-note budget emerged as a local experimental design choice, not a universal optimum.

How we would challenge this finding

A stronger follow-up would use more questions, multiple samples, a proficient Japanese reviewer, and a frozen selection benchmark. We'd also test content order, matched token budgets, and the contribution of individual fields within each note. A good test would ask whether the linked context still wins when the total number of tokens is controlled, not just when content selection is changed.

Sources and record

Primary experiment: docs/explore_set_v2_results.md, run dated 2026-10-13; underlying question file testdata/explore/questions_v2.jsonl. The repository paths describe experimental artifacts and should be linked only after they are made publicly available.

[1] Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics (2024). https://arxiv.org/abs/2307.03172

Background: Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020). https://arxiv.org/abs/2005.11401