Management summary. This experiment tests a larger pool of possible next words without retraining the model. Among the two new variants, development data selected Complete words: small search. On 900 newly reserved examples, its exact top-three accuracy was 39.7%, compared with 39.4% for the existing generator. Candidate coverage changed from 81.0% to 82.3%.
Decision: Retain the existing default: the new generator does not pass the predeclared paired accuracy test. The requested 85% free-text target is not reached. More candidates do not guarantee correct rankings or 99.8% accuracy.
Earlier error analysis identified words missing from the candidate list. Some complete words require several model tokens, but a one-token proposal mechanism cannot assemble them. We therefore add a bounded search through a tree of words from the training vocabulary [1]. Only complete vocabulary words become proposals.
The existing generator retrieves 256 eligible token IDs, then adds up to 20 local n-gram words. Token aliases mean this is not necessarily 256 unique words. The alternatives explore 16 or 64 active token paths, with at most six tokens per new word and 128 completed proposals. All original words and their numerical scores survive unchanged. New words use the same likelihood and word-boundary scoring rule, with the same model weights and 128-token context [2,3].
Both alternatives use the same 600 development cases. Selection among the two new variants first maximizes exact top-three successes, then top-one successes, then measured speed; remaining ties favor the smaller beam. Timing independently reruns 60 cases, 20 per source, in alternating condition order.
knitr::kable(development[,.(Generator=unname(labels[candidate]),
`First word`=pct(top1), `First three`=pct(top3),
Coverage=pct(shortlist_recall), `Milliseconds per case`=round(milliseconds))])| Generator | First word | First three | Coverage | Milliseconds per case |
|---|---|---|---|---|
| Existing generator | 24.8% | 37.0% | 77.8% | 311 |
| Complete words: small search | 24.8% | 36.7% | 78.7% | 766 |
| Complete words: wide search | 24.8% | 36.7% | 79.5% | 1129 |
The word tree makes 32 of 33 previously unreachable development targets theoretically reachable. Bounded search is stricter: the two beams actually add only 5 and 10 observed targets to the candidate lists, respectively. Both yield zero top-three gains and two losses on development. Removing a structural restriction therefore does not establish an accuracy gain.
The selected generator was frozen before the new final test. Repeated development-set use can favor that sample; the fresh holdout protects this particular final comparison.
Every comparison below uses the same 900 cases, 300 each from blogs, news and Twitter. Previous evaluation cases and exact normalized training-line overlaps were excluded. Sources receive equal weight; this is not traffic-weighted app performance. Exact matching asks whether the recorded author’s word appears first or among the first three; other reasonable continuations still count as errors.
knitr::kable(final[,.(Generator=unname(labels[candidate]),
`First word`=pct(top1), `First three`=pct(top3),
Coverage=pct(shortlist_recall),
`Synthetic four-choice`=pct(synthetic_choice_accuracy),
`Milliseconds per case`=round(milliseconds))])| Generator | First word | First three | Coverage | Synthetic four-choice | Milliseconds per case |
|---|---|---|---|---|---|
| Existing generator | 25.3% | 39.4% | 81.0% | 89.6% | 326 |
| Complete words: small search | 25.3% | 39.7% | 82.3% | 89.6% | 780 |
The paired top-three change is +0.2 percentage points, with a 95% interval of +0.0 to +0.6 points. The exact paired p-value is 0.500. Promotion requires both a positive lower interval bound and p below 0.05. This tests evidence of improvement, not attainment of a chosen accuracy target.
Hover over a bar to see the exact score and number of examples.
chart <- melt(final,id.vars=c("candidate","cases"),
measure.vars=c("top3","shortlist_recall"),
variable.name="Measure",value.name="Rate")
chart[,Measure:=factor(Measure,levels=c("top3","shortlist_recall"),
labels=c("First three correct","Candidate coverage"))]
chart[,Generator:=factor(unname(labels[candidate]),
levels=unname(labels[c("baseline",selected_name)]))]
chart[,Tooltip:=paste0(Generator,"<br>",Measure,": ",pct(Rate),
"<br>Correct/covered: ",round(Rate*cases)," / ",cases)]
plot <- ggplot(chart,aes(Generator,Rate,fill=Measure,text=Tooltip))+
geom_col(position=position_dodge(.75),width=.65)+
scale_fill_manual(values=c("#236795","#7d8a94"))+
scale_y_continuous(labels=scales::percent,limits=c(0,1))+
labs(x=NULL,y="Proportion of final cases",fill=NULL)+
theme_minimal(base_family="Segoe UI")+theme(legend.position="bottom")
plotly::ggplotly(plot,tooltip="text")The new generator turns 2 previous top-three failures into successes, but turns 0 previous successes into failures. It newly covers 12 targets and loses coverage of 0; 2 accuracy gains concern previously missing targets. Additional candidates can displace an old correct suggestion even when every old score is preserved.
token_groups <- groups[group %in% c("single_token","multiple_tokens")]
knitr::kable(token_groups[,.(
`Target encoding`=fifelse(group=="single_token","One canonical token","Multiple canonical tokens"),
Cases=cases, `Old first three`=pct(baseline_top3),
`New first three`=pct(expanded_top3),
`Old coverage`=pct(baseline_coverage), `New coverage`=pct(expanded_coverage),
Gains=gains, Losses=losses)])| Target encoding | Cases | Old first three | New first three | Old coverage | New coverage | Gains | Losses |
|---|---|---|---|---|---|---|---|
| One canonical token | 812 | 43.5% | 43.5% | 87.3% | 87.3% | 0 | 0 |
| Multiple canonical tokens | 88 | 2.3% | 4.5% | 22.7% | 36.4% | 2 | 0 |
These groups use the canonical encoding of each target word, not alternative token aliases. Subgroup comparisons are descriptive. Four-choice outcomes are identical by design because answer options are scored separately with unchanged weights; they cannot improve through candidate generation. Synthetic alternatives are not real quiz results.
This comparison isolates generation rather than additional data or training. Higher coverage alone is insufficient: the ranking must place new correct words above plausible alternatives. Runtime is measured independently over all final cases after loading, including the separate four-choice scoring task; model/trie initialization and R shortlist construction add further app latency.
The training lexicon, bounded beam, six-token limit and canonical encoding constrain reach. Proposals are pruned before adding the final boundary score, so search can miss words that would rank well afterward. Near duplicates and unknown overlap with pretrained data remain possible. A single English holdout does not establish performance across all users or writing styles.
On this holdout, the extra search adds 12 covered words but only 2 additional top-three successes, from 355/900 to 357/900. First-word accuracy remains 25.3%. Mean measured time rises from 326 to 780 ms, or 2.39 times the original runtime. With only two discordant outcomes and p = 0.50, this is insufficient evidence of an accuracy improvement.
Conclusion: the registered test does not support replacing the existing generator. Coverage, errors and computational cost should guide the next separately validated experiment.
This report is knitted from
25_word_generation_report.Rmd using saved, audited results.
In RStudio, Knit rebuilds the report;
26_try_complete_words.R runs a new phrase comparison. The
reproduction
guide records the pipeline. An independent audit recomputes
individual ranks and comparisons from saved targets, candidates and
scores, without publishing full prefixes; its recorded status is
passed.
The initial neutral FP16 cached/full-sequence check failed its absolute log-score tolerance of 0.001. Further neutral checks reached 0.01571 maximum difference. Using the same merged weights in FP32 reduced the maximum to 1.24e-05, supporting the scoring formula while identifying finite-precision effects. Tested neutral rankings agreed. Before inspecting development outcomes, the numerical tolerance was registered as 0.02. This is not an accuracy tolerance: every original baseline score remains exactly unchanged in the union.
Independent development batches exceeded the neutral 0.02 reference in 371 of 22175 comparisons, with a maximum log-score difference of 0.05028. This was retained as a diagnostic, not used to alter selection. Rechecking the largest case with identical merged weights in FP32 reduced its cross-batch difference to 9.54e-06 and confirmed the full-sequence formula. These checks support finite-precision sensitivity; they do not guarantee identical rankings for every prompt. Published outcomes remain those of the frozen FP16 implementation.
The final numerical audit flags 1 of 1310 separately batched score comparisons above that reference, with maximum difference 0.02692. Neither of the two final accuracy gains lies within the audited close-boundary threshold. These diagnostics are published without retuning the frozen model.
Independent CPU tests cover complete-word termination, continuing terminal paths, deterministic beam ties, candidate preservation and option isolation. The trie contains 476,776 eligible multiple-token words. Its 4,692 over-depth entries are excluded from new proposals, not removed from the baseline. Vocabulary-type proportions do not estimate future token coverage.
knitr::kable(groups[group %in% c("blogs","news","twitter"),
.(Source=group,Cases=cases,`Old first three`=pct(baseline_top3),
`New first three`=pct(expanded_top3),Gains=gains,Losses=losses)])| Source | Cases | Old first three | New first three | Gains | Losses |
|---|---|---|---|---|---|
| blogs | 300 | 44.0% | 44.0% | 0 | 0 |
| news | 300 | 38.7% | 39.0% | 1 | 0 |
| 300 | 35.7% | 36.0% | 1 | 0 |
The paired bootstrap uses 10,000 replicates stratified by source. The exact two-sided McNemar test uses paired gains and losses. The interval describes sampling uncertainty for this source-balanced evaluation and fixed model; it is not a guarantee for all possible prompts. Secondary subgroup findings are not independent promotion tests.
rmarkdown::render("25_word_generation_report.Rmd")
source("26_try_complete_words.R")sessionInfo()## R version 4.6.1 (2026-06-24 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26200)
##
## Matrix products: default
## LAPACK version 3.12.1
##
## locale:
## [1] C
## system code page: 65001
##
## time zone: Europe/Amsterdam
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggplot2_4.0.3 data.table_1.18.6.1
##
## loaded via a namespace (and not attached):
## [1] gtable_0.3.6 jsonlite_2.0.0 dplyr_1.2.1 compiler_4.6.1
## [5] tidyselect_1.2.1 tidyr_1.3.2 jquerylib_0.1.4 scales_1.4.0
## [9] yaml_2.3.12 fastmap_1.2.0 R6_2.6.1 labeling_0.4.3
## [13] generics_0.1.4 knitr_1.51 htmlwidgets_1.6.4 tibble_3.3.1
## [17] bslib_0.12.0 pillar_1.11.1 RColorBrewer_1.1-3 rlang_1.3.0
## [21] cachem_1.1.0 xfun_0.60 sass_0.4.10 S7_0.2.2
## [25] otel_0.2.0 viridisLite_0.4.3 plotly_4.12.1 cli_3.6.6
## [29] withr_3.0.3 magrittr_2.0.5 crosstalk_1.2.2 digest_0.6.39
## [33] grid_4.6.1 lifecycle_1.0.5 vctrs_0.7.3 evaluate_1.0.5
## [37] glue_1.8.1 farver_2.1.2 rmarkdown_2.31 purrr_1.2.2
## [41] httr_1.4.8 tools_4.6.1 pkgconfig_2.0.3 htmltools_0.5.9ea980cb0a6c2ae4b936e82123acc929f1cec04c1.