The question. If a question-aware selector is already finding relevant notes, does walking typed relationships between notes retrieve useful missing context?
The finding. No, not in our frozen 29-turn experiment. Graph traversal found zero additional required notes, selected five more irrelevant notes, and reduced exact-selection performance from 20 of 29 turns to 15 of 29 turns. We removed the edge-walk implementation.
Decision: Prefer the smallest selector that improves held-out evidence selection. An appealing data structure is not evidence of a useful retrieval algorithm.
Why we tried a graph
Japanese grammar is relational. A construction can depend on a verb class, contrast with another construction, or be an exception to a broader pattern. We considered representing those relationships as typed edges between concept notes, then following one edge when assembling a teaching prompt.
The implementation was deliberately small: JSON-authored notes and links, typed Go structures loaded into memory, and a one-hop walk. No graph database was involved. It was still reasonable to ask whether the relationship expansion itself would add value.
The comparison had three arms:
- Row: choose notes from the current exercise's case rules.
- Question: add rules that inspect the learner's current question and references to earlier verbs in the conversation.
- Graph: keep the question rules and add one hop along typed note-to-note edges.
The expected notes for 15 scripted conversations—29 turns in total—were written before the selector code. That freezes the mechanical selection target, reducing the temptation to tune rules until they fit the observed examples.
What the selector actually retrieved
| Metric | Row | + Question | + Graph |
|---|---|---|---|
| Required notes selected | 14 / 25 | 21 / 25 | 21 / 25 |
| Turns containing every required note | 18 / 29 | 25 / 29 | 25 / 29 |
| Exact turns: all required, none irrelevant | 18 / 29 | 20 / 29 | 15 / 29 |
| Irrelevant notes selected | 3 | 8 | 13 |
The question rules were useful. They could follow a conversation away from the current exercise. For example, on a 読む (yomu, “to read”) exercise, the student might ask about 知っています (shitte imasu, “I know”). A row-only selector looks at the reading exercise. A question-aware selector notices the new verb and can retrieve its note instead.
The graph arm did not add any required note. Its additional candidates were general notes used to fill otherwise empty second slots.
More recall was not the whole story
Looking only at “turns containing every required note” would make question retrieval appear to jump from 18 to 25. But that number ignores distractions. When we also require no irrelevant selections, the improvement is only 18 to 20. The graph arm is worse on this stricter measure.
The two-note cap makes false positives consequential. When the learner asked about 分かっています (wakatte imasu, “I understand”) while on a 知る (shiru, “to know”) exercise, an over-broad て-form rule displaced the row’s 知る note. The model then presented 分かっていません (wakatte imasen, “has not understood”) where 分かりません (wakarimasen, “I don’t understand”) would normally be the more appropriate expression for the intended meaning. The former is valid in some contexts, so we later reclassified this from a clearly false answer to a possibly misleading usage choice. The selection failure itself remained: an irrelevant note occupied a limited slot.
Did the graph help the answers?
The answer-level comparison did not rescue it. The graph arm made no meaningful improvements on the key turns, incurred one additional timeout, and had slightly longer median first-reply latency (7.1 seconds, versus 6.7 seconds for the question-rule arm). Suspected-error replies were 2–3 for graph and 2 for question rules, judged by one non-expert reader.
| Measure | Row | + Question | + Graph |
|---|---|---|---|
| Suspected-error replies | 4–5 | 2 | 2–3 |
| 180-second timeouts | 1 | 1 | 2 |
| Median first reply | 6.8 s | 6.7 s | 7.1 s |
The run is small and not independently reviewed. We are not claiming graph-based retrieval is generally inferior. We're claiming this specific one-hop expansion provided no measured benefit in this setting.
Why the negative result matters
A graph might become necessary if a future benchmark contains a relationship that direct case and question rules cannot retrieve. But adding it before that failure exists increases maintenance burden, selection variance, and prompt noise without earning its complexity.
This also demonstrates a broader discipline: an experiment should be allowed to delete a component. We committed the experimental arm for reproducibility, then removed the edge walk in a separate change. The notes stayed in Git; question-aware retrieval continued working.
The reusable engineering pattern
Hypothesis: a one-hop graph retrieves missing notes
Prediction: required-note recall increases
Observation: 21 / 25 before, 21 / 25 after
Side effect: irrelevant notes 8 -> 13
Decision: remove graph expansion
A future graph experiment should start by defining a set of unreachable questions—questions known to require relationship traversal and not already solved by direct rules. Without that target, a graph can become an excuse to add plausible but unnecessary context.
Sources and record
Primary experiment: docs/explore_set_v3_results.md; testdata/explore/questions_v3.jsonl (question file hash beginning abe8a776); completed 2026-10-14. Selection expectations were frozen before implementation. The study used gemma4:12b-mlx, temperature 0.2 and a 32K context window.
Review limitation: one AI-assisted author also wrote the questions, expected note selections, and initial judgments. Notes and replies were not independently reviewed by a proficient Japanese speaker. No broad claims about GraphRAG are made from these results.