Smaller models.
Sharper experiments.
A technical field journal on making a local 12B language model more dependable: retrieval, conversational context, evidence boundaries, measurements—and the features we chose to delete.
Research field studies
Each study starts with a failure, changes one piece of the system, and records what improved, what got worse, and what remains unknown.
More context was not better
Selected notes helped. The entire corpus didn’t.
We added graph traversal. Then removed it.
A measured negative result, not an architectural opinion.
The note was right. The model still invented a rule.
Evidence boundaries and wider coverage solved different problems.
Stop tuning the demo. Measure the failure.
An engineering method for making failures reproducible.
Five levers. Five different failures.
The question is not which AI technique is fashionable. It is which component actually explains the observed failure.
Supply outside knowledge
Does the model lack the facts the task needs?
Select relevant evidence
Did the wrong note reach the prompt—or too many notes?
Resolve conversation
Does “that one” refer to the prior verb rather than the exercise?
Compress the context
Are prompt tokens and irrelevant fields dominating inference?
Limit claims
Does the model invent facts beyond a correctly retrieved note?
Question rules + compact notes + evidence boundaries
Built and experimentally compared around a deterministic Go grading and morphology engine. Model weight fine-tuning and activation steering have not been tested in this study series.
Typed graph-edge expansion
We tested one-hop graph expansion and removed it. The study did not establish that graph retrieval is universally ineffective.
A result is only as good
as its counterfactual.
We freeze questions where possible, preserve note-selection traces, and score selection independently from generated answers.
| Experiment | Question | Observation | Decision |
|---|---|---|---|
| v2 · Linked notes | Whole corpus or selected evidence? | 2–3/22 suspected errors linked versus 6/22 with none; all notes no better | Select, then compress |
| v3 · Question/graph | What happens when a learner switches topics? | Required-note recall 14/25 → 21/25; graph unchanged at 21/25 | Keep question rules; delete edge walk |
| v4 · Boundary prompt | Will the model stop when notes stop? | Partial-support overreach 5/8 → 0/8 in this sample | Keep as pilot candidate |
| v5 · More notes | Can a strict policy still be useful? | Strict supported-answer success 4/8 → 8/8 with new notes; three true unsupported leaks remained | Improve coverage, continue validation |
Selection is not correctness.
We score required-note recall and exact selections against predeclared expectations. A turn can retrieve every required note and still receive an incorrect reply.
Review is part of the experiment.
Language judgments were initially made by the experiment author, without proficient independent review. Those uncertainties are preserved rather than hidden in an accuracy headline.
Evidence should outlive
the demo.
The source reports identify frozen question sets, JSONL judgments, model settings and experimental trace directories. Public artifact links will be added when the repository is released.
v2, v3, v4 and v5
Run reports: docs/explore_set_v2_results.md, docs/explore_set_v3_results.md and docs/explore_set_v4_v5_results.md. These are local repository paths, not yet live public links.
What should we try next?
Independent Japanese review; held-out adversarial boundary tests; structured claim selection; SFT/LoRA and activation steering as controlled arms; reproducible timeout instrumentation.