The note was right. The model still invented a rule.

The question. Can a local model be made more reliable by asking it to treat supplied notes as the boundary of established knowledge? And what happens to useful answers when those notes cover only a small part of the domain?

The finding. A stricter evidence policy eliminated observed unsupported extensions in eight partially supported questions and induced all eight fully unsupported questions to be declined in one 24-question test. On a second 16-question test, adding seven new notes restored four useful answers the strict policy had previously withheld. The stricter model still made three true-but-unsupported assertions in that second test.

Decision: Evidence boundaries and content breadth solve different problems. Use both, and measure supported correctness and unlicensed assertions separately.

The failure that prompted the test

On an earlier question—“Is 行く (iku, “to go”) the only exception to the く-ending て-form rule?”—Gemma received the relevant note but replied that there were other exceptions, without support. In a later run it offered 買う (kau, “buy”), 使う (tsukau, “use”), and 待つ (matsu, “wait”) as examples—even though none ends in く.

The retrieval system had worked: the note was present. The problem was generation beyond the evidence. Adding more retrieval logic would not by itself prevent the next unsupported extrapolation.

We compared the existing system instruction with a stricter rule: only provided verified facts, background notes, and lesson notes count as established; say what they settle, state what they do not settle, and do not assert that a list is complete or incomplete unless the material establishes that.

Here, “verified facts” is the historical prompt label for deterministic exercise facts; the 13 lesson notes were still provisional and had not been independently reviewed. The test assessed obedience to the supplied evidence, not absolute linguistic truth.

Study A: an answering-policy experiment

The 24 questions were frozen in advance: eight fully supported by the notes, eight partly supported, and eight unsupported. The evidence selector retrieved all the notes judged required on these questions, so retrieval recall was deliberately held out of the comparison.

The strict answer to the 行く question was appropriately limited: the note establishes that 行く takes 行って (itte, its て-form); it does not establish an exhaustive list of exceptions. The model said it could not make the broader claim.

This is an important difference from a generic instruction to “be careful.” A boundary policy names the evidence that gives a claim authority and defines what to do when the evidence ends.

The downside: safe but unhelpful answers

The strict policy refused some answers that the model apparently knew correctly from pretraining, such as the relationship between 好き (suki, “to like”) and 大好き (daisuki, “to really like/love”). Because the authorized notes didn't cover that fact, the model declined.

That isn't necessarily an error in the boundary policy. It is a coverage failure. If the product promises evidence-grounded instruction, plausible knowledge in a model's weights is not the same thing as reviewed instructional content.

This led to the second experiment: keep the strict policy, but improve the evidence base.

Study B: the content and policy interaction

Seven new provisional notes covered past tense, non-past use, wanting, requests, transitive/intransitive pairs, existence/availability, and location particles. Each authored claim required a citation in the loader. The links reached 195 of 300 served seed exercises, up from 103. None of the 300 vocabulary rows was linked yet.

We then ran 16 fresh questions, crossing two note sets (old versus expanded) with two policies (existing versus strict):

One strict/new unsupported question timed out. Two unsupported questions were not declined, so this is not proof of a hard boundary.

The clearest uncorrected failure under the existing policy was 〜たくないでした (-takunai deshita) offered for “didn't want to.” The ordinary polite alternatives include 〜たくなかったです (-takunakatta desu) or 〜たくありませんでした (-taku arimasen deshita). The provisional 〜たい note did not cover its past negative. The strict prompt therefore declined instead of manufacturing a form. The expanded note set did not make the existing policy abstain.

Three kinds of error we initially conflated

It is easy to call every imperfect reply a hallucination. That obscures the engineering decision.

1. False linguistic claim. The model asserts a rule or form that is wrong.
2. Unsupported but true claim. The answer happens to be correct, but the supplied instructional evidence doesn't license it.
3. Contextually misleading usage. A Japanese expression is grammatical, but a poor default for the meaning or setting being taught.

The strict policy's remaining leaks in the second experiment fell into the second category according to the original reviewer. An independent Japanese speaker has not yet adjudicated those judgments.

What this does not prove

We observed a strong effect on a small, hand-authored benchmark, not a general correctness guarantee. There was one sample per condition, one author/reviewer, and unreviewed note content. No proficiency test or learner-outcome measurement was conducted.

Nor does JSON schema, LoRA, activation steering or a second LLM judge automatically make unsupported statements impossible. Stronger guarantees would require a narrower output authority model—for example, selecting authorized claim IDs that deterministic code validates and renders. We have discussed that as an architectural research direction, not implemented or measured it here.

Practical takeaway

When a model invents beyond a correct retrieved note, first change the answering policy, not the retriever. Then measure the cost of refusal. If supported questions are being declined, improve reviewed knowledge coverage. Keep independent linguistic review as a requirement before describing anything as verified instruction.

Sources and record

Primary experiments: docs/explore_set_v4_v5_results.md, testdata/explore/questions_v4_boundary.jsonl (hash beginning 7b3b8db1) and questions_v5_coverage.jsonl (hash beginning 4a25efa9). Each condition was sampled once with gemma4:12b-mlx, temperature 0.2, 32K context. Subjective judgments are retained in experimental judgments.jsonl files, pending publication.

Linguistic reference: National Institute for Japanese Language and Linguistics, Basic Verb Handbook, entry for 知る (shiru, “to know”). https://www2.ninjal.ac.jp/verbhandbook/headwords/%E7%9F%A5%E3%82%8B.html