This analysis supplies the known next word. Its results are hypothetical diagnostic counts, not deployable accuracy, quiz results or a new final test. The production predictor is unchanged.
Management summary. On the same 600 development examples, the existing generator misses 133 recorded target words. Supplying those words to the unchanged scoring rule would put 0 in the top three; 133 still rank lower. A further 245 targets were already available but ranked below third place. This distinguishes retrieval opportunities from limitations of the model and scoring rule.
The preceding complete-word experiment found little benefit relative to its extra runtime [1]. This diagnosis asks whether missing words would rank well if the search found them. It reuses the original candidate pools and numerical scores for both the existing generator and the previously selected 16-path word search.
No training, answer-option ranking, candidate regeneration or new final-test evaluation occurs. Both candidate pools are analyzed because an added competitor can lower the target’s rank. No interaction terms are fitted: this is a fixed-model intervention; token-length and source differences are descriptive sensitivity checks.
knitr::kable(overall[,.(
Pool=unname(labels[arm]),
`Already top three`=already_top3,
`Available but ranked lower`=covered_below_top3,
`Missing; would reach top three`=missing_reaches_top3,
`Missing; still ranked lower`=missing_still_below_top3)])| Pool | Already top three | Available but ranked lower | Missing; would reach top three | Missing; still ranked lower |
|---|---|---|---|---|
| Existing candidates | 222 | 245 | 0 | 133 |
| Complete-word candidates | 220 | 252 | 0 | 128 |
The existing pool’s target-aware count is 222/600, compared with 222/600 before insertion. For the complete-word pool it is 220/600, compared with 220/600. These are optimistic counts within fixed competing pools. Actual search must discover useful words without knowing the answer and could also introduce higher-scoring competitors. They are not attainable-performance promises or global vocabulary ceilings.
The closest missing target in the existing pool ranks 8, still 0.543 log-score point below the top-three cutoff. Thus none of the missing words meets the 0.10 precision-check rule; their FP16 scores should not be described as individually FP32-verified.
Hover over a segment to see its exact number of examples.
categories <- c("already_top3","covered_below_top3","missing_reaches_top3","missing_still_below_top3")
category_labels <- c("Already top three","Available but lower","Missing, would reach top three","Missing, still lower")
chart <- melt(overall,id.vars="arm",measure.vars=categories,
variable.name="Category",value.name="Count")
chart[,Category:=factor(Category,levels=categories,labels=category_labels)]
chart[,Pool:=factor(unname(labels[arm]),levels=unname(labels))]
chart[,Tooltip:=paste0(Pool,"<br>",Category,"<br>",Count," / 600 development examples")]
p <- ggplot(chart,aes(Pool,Count,fill=Category,text=Tooltip))+
geom_col(width=.6)+coord_flip()+
scale_fill_manual(values=c("#236795","#88959f","#71a9cb","#d2d8de"))+
labs(x=NULL,y="Number of development examples",fill=NULL)+
theme_minimal(base_family="Segoe UI")+
theme(legend.position="bottom",legend.text=element_text(size=9))+
guides(fill=guide_legend(nrow=2))
plotly::ggplotly(p,tooltip="text")16 distinct examples met the predeclared check rule. Their entire candidate union was rescored using the same merged parameter values promoted to FP32, with TF32 disabled and batches of four. No shortlist was regenerated. This tests numerical sensitivity, not whether a newly trained FP32 model would be better. Batched floating-point operations need not produce identical values [2].
knitr::kable(overall[,.(
Pool=unname(labels[arm]),`Examples checked`=fp32_checked,
`Target top-three membership changed`=membership_changes_in_checked_subset,
`Missing-word recoveries confirmed`=recoveries_confirmed_fp32)])| Pool | Examples checked | Target top-three membership changed | Missing-word recoveries confirmed |
|---|---|---|---|
| Existing candidates | 16 | 2 | 0 |
| Complete-word candidates | 16 | 1 | 0 |
For the target and the third-highest other word in each checked pool, cached scores were also compared with a direct full-sequence calculation. The largest absolute log-score discrepancy was 3.15e-05; the registered 0.0001 check passed for every comparison. This subset check cannot certify all unexamined prompts or eliminate every numerical effect.
In the existing pool, “have” moves from third to fourth and “here” from fourth to third under FP32. In the complete-word pool, only “here” crosses the cutoff. All these words were already present. The diagnosis identifies precision-sensitive individual rankings without establishing an overall FP32 accuracy gain.
Supplying the correct missing word addresses only retrieval. Under the current score rule, 378 of 600 examples still put at least three competing words ahead of the recorded target. Some alternatives may be linguistically reasonable; exact matching has only one observed continuation. Therefore these findings locate a limitation of this model, score rule and evaluation target, rather than proving that every lower-ranked word is linguistically wrong.
The next priority is the model and ranking criterion. In this diagnostic sample, even supplying every missing answer does not add a top-three success. Merely expanding retrieval therefore has no demonstrated upside under the current score rule. This is an inference for this fixed setting, not proof that search improvements can never help. A different model or scoring method needs its own controlled development comparison and a newly reserved test before promotion. The current diagnosis supplies no evidence for 85% or 99.8% free-word accuracy.
Repeated use of development data makes this exploratory. The known target is deliberately introduced, some targets may fall outside the training vocabulary, and results depend on the fixed competing pools. Word-boundary scoring and canonical tokenization remain modeling assumptions. The production default and all previous test results are retained.
This HTML report is knitted from
28_target_rank_diagnostic.Rmd. Expand Code
to see the table and figure calculations. Source
27_target_rank_diagnostic.R in the same RStudio project to
verify saved evidence and knit this report; GPU calculation is needed
only for a fresh diagnosis. The Python
analysis, case
table, registered
procedure and independent
audit retain ranks, score differences, candidate evidence and input
fingerprints. Full text prefixes and model weights are not
republished.
knitr::kable(groups[group!="All",.(Pool=unname(labels[arm]),Group=group,Cases=cases,
`Already top three`=already_top3,`Available but lower`=covered_below_top3,
`Missing; would reach top three`=missing_reaches_top3,
`Missing; still lower`=missing_still_below_top3)])| Pool | Group | Cases | Already top three | Available but lower | Missing; would reach top three | Missing; still lower |
|---|---|---|---|---|---|---|
| Existing candidates | single_token | 549 | 220 | 241 | 0 | 88 |
| Existing candidates | multiple_tokens | 51 | 2 | 4 | 0 | 45 |
| Existing candidates | blogs | 200 | 74 | 86 | 0 | 40 |
| Existing candidates | news | 200 | 82 | 75 | 0 | 43 |
| Existing candidates | 200 | 66 | 84 | 0 | 50 | |
| Complete-word candidates | single_token | 549 | 218 | 243 | 0 | 88 |
| Complete-word candidates | multiple_tokens | 51 | 2 | 9 | 0 | 40 |
| Complete-word candidates | blogs | 200 | 73 | 88 | 0 | 39 |
| Complete-word candidates | news | 200 | 82 | 76 | 0 | 42 |
| Complete-word candidates | 200 | 65 | 88 | 0 | 47 |
cases[,Rank:=cut(diagnostic_rank,breaks=c(0,1,2,3,10,50,Inf),
labels=c("1","2","3","4-10","11-50","51+"))]
distribution <- dcast(cases[,.(Examples=.N),by=.(arm,Rank)],Rank~arm,value.var="Examples",fill=0)
knitr::kable(distribution)| Rank | baseline | beam16 |
|---|---|---|
| 1 | 149 | 149 |
| 2 | 37 | 36 |
| 3 | 36 | 35 |
| 4-10 | 80 | 80 |
| 11-50 | 104 | 104 |
| 51+ | 194 | 196 |
Target membership in the training vocabulary is recorded for every example. Among the existing generator’s hypothetical recoveries, 0 targets are outside that vocabulary. Such words require a vocabulary change as well as a retrieval change.
sessionInfo()## R version 4.6.1 (2026-06-24 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26200)
##
## Matrix products: default
## LAPACK version 3.12.1
##
## locale:
## [1] C
## system code page: 65001
##
## time zone: Europe/Amsterdam
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggplot2_4.0.3 data.table_1.18.6.1
##
## loaded via a namespace (and not attached):
## [1] gtable_0.3.6 jsonlite_2.0.0 dplyr_1.2.1 compiler_4.6.1
## [5] tidyselect_1.2.1 tidyr_1.3.2 jquerylib_0.1.4 scales_1.4.0
## [9] yaml_2.3.12 fastmap_1.2.0 R6_2.6.1 labeling_0.4.3
## [13] generics_0.1.4 knitr_1.51 htmlwidgets_1.6.4 tibble_3.3.1
## [17] bslib_0.12.0 pillar_1.11.1 RColorBrewer_1.1-3 rlang_1.3.0
## [21] cachem_1.1.0 xfun_0.60 sass_0.4.10 S7_0.2.2
## [25] otel_0.2.0 viridisLite_0.4.3 plotly_4.12.1 cli_3.6.6
## [29] withr_3.0.3 magrittr_2.0.5 crosstalk_1.2.2 digest_0.6.39
## [33] grid_4.6.1 lifecycle_1.0.5 vctrs_0.7.3 evaluate_1.0.5
## [37] glue_1.8.1 farver_2.1.2 rmarkdown_2.31 purrr_1.2.2
## [41] httr_1.4.8 tools_4.6.1 pkgconfig_2.0.3 htmltools_0.5.9ea980cb0a6c2ae4b936e82123acc929f1cec04c1.