Management summary. We compared seven scoring rules using identical model weights and candidate lists. Development data selected Original score. On 900 new test examples, it placed the observed word among its first three suggestions in 37.1% of cases. First-suggestion correctness was 23.8%. Confidence was fitted separately on 900 calibration examples and assessed only afterward on the test set.

Decision: Keep the original ranking: none of the alternatives won the predeclared development comparison.

A displayed probability is an estimated chance that the first suggestion matches the observed word. It is not a guarantee for an individual phrase. Top-three accuracy is reported separately.

The confidence map remains a research result in this report. It has not been enabled in the standard prediction interface.

Design and reasoning

The preceding diagnosis found that supplying missing target words did not improve development top-three counts [1]. We therefore isolated the score rule. Candidate generation remained target-blind: it proposed eligible model tokens plus the same R n-gram shortlist. No known-answer insertion, answer options, extra training or different base model was used.

Three data roles prevent a confidence map from being assessed on its own fitting examples:

Role Examples Purpose
Development, previously reused 600 Compare seven fixed rules and select one
New calibration set 900 Fit confidence for the fixed selected rule and original control
New final test 900 Evaluate ranking, confidence and predeclared abstention thresholds

Each role balances blogs, news and Twitter. The new examples exclude all 9,300 earlier evaluation lines and used training lines. These checks address exact normalized duplicates, not near duplicates or unknown overlap with the model’s pretraining.

Which ranking rule helped?

tab <- merge(dev,paired,by="variant",sort=FALSE)
tab <- tab[match(names(labels),variant)]
knitr::kable(tab[,.(Rule=unname(labels[variant]),`First correct`=pct(top1),
  `First three correct`=pct(top3),Gains=gains,Losses=losses)])
Rule First correct First three correct Gains Losses
Original score 24.8% 37.0% 0 0
No word-boundary term 24.8% 36.5% 0 3
Half word-boundary term 24.8% 37.0% 0 0
Mean per model token 24.8% 36.8% 3 4
Token mean without boundary 25.0% 36.7% 3 5
Character-length adjustment 23.8% 33.5% 8 29
Two valid tokenizations 24.8% 37.0% 0 0

All rules share 77.8% candidate coverage on development. Removing or halving the word-boundary term tests its influence. Token means and a square-root character adjustment test length sensitivity. A combined token-mean/no-boundary rule checks a prespecified interaction. The tokenization variant sums two distinct paths that decode to the same word; it does not change the tokenizer or exhaust every possible segmentation.

Selection maximized the number of top-three successes, then first-suggestion successes, then a fixed simplicity order favoring the original. These are predefined rules, not a fitted reranker. A future learned reranker must use training-only, target-blind candidate generation.

The combined token-mean/no-boundary rule adds one first-suggestion success (150 versus 149), but reduces top-three successes from 222 to 220. The registered top-three priority therefore retains the original rule. This small tradeoff does not establish that every alternative is uniformly worse.

New final test

knitr::kable(final[,.(Rule=unname(labels[variant]),Cases=cases,
  `First correct`=pct(top1),`First three correct`=pct(top3),
  `Target in candidate list`=pct(coverage))])
Rule Cases First correct First three correct Target in candidate list
Original score 900 23.8% 37.1% 78.9%

Because the original rule won development selection, there is no challenger in this final test. It estimates the retained predictor’s performance and confidence reliability; no self-comparison is presented as evidence of improvement.

knitr::kable(timing[,.(Rule=unname(labels[variant]),`Timed examples`=cases,
  `Mean milliseconds`=round(mean_ms,1))])
Rule Timed examples Mean milliseconds
Original score 60 305.9

Timing reruns 60 balanced test cases independently after loading, with synchronized GPU work. It includes candidate generation and the applicable score calculation, but excludes model loading and the R shortlist. Earlier studies included a separate multiple-choice task, so their timing numbers are not directly comparable.

Are the confidence estimates reliable?

The calibration target is first-suggestion correctness across all cases, including those whose answer is missing from the shortlist. A two-parameter logistic map converts the shortlist’s relative score share into an estimated correctness probability [2]. Its fixed slope penalty limits extreme fits. It changes the displayed estimate, not the word ranking. Only the separate calibration set was used to fit it.

knitr::kable(confidence[,.(Rule=unname(labels[variant]),
  Estimate=ifelse(kind=="calibrated","Calibrated","Uncalibrated shortlist share"),
  `Brier score`=round(brier,4),`Log loss`=round(log_loss,4),
  `10-bin calibration error`=round(ece_10_equal_width,4))])
Rule Estimate Brier score Log loss 10-bin calibration error
Original score Uncalibrated shortlist share 0.1498 0.4702 0.0549
Original score Calibrated 0.1460 0.4606 0.0285

Lower Brier score and log loss indicate better overall probability predictions; they do not, on their own, prove calibration. The reliability plot compares average estimated confidence with observed correctness. Sparse bins are less certain. The 10-bin error depends on the chosen binning [2].

All three descriptive probability metrics improve on this test, but reliability is uneven. The highest calibrated bin contains only 11 cases: mean estimated correctness is 94.8%, while 9 of 11 predictions are correct (81.8%). More high-confidence observations are needed before making a service-level claim.

Hover over a point to see its confidence, observed accuracy and example count.

reliability <- rbindlist(lapply(seq_len(nrow(confidence)),function(i) {
  x <- as.data.table(confidence$reliability[[i]])
  x[,`:=`(variant=confidence$variant[i],kind=confidence$kind[i])]
  x
}))
chart <- reliability[variant==chosen & cases>0]
chart[,Estimate:=factor(kind,levels=c("raw_shortlist_share","calibrated"),
                        labels=c("Uncalibrated share","Calibrated"))]
chart[,Tooltip:=paste0(Estimate,"<br>Estimated: ",pct(mean_probability),
  "<br>Observed: ",pct(observed_accuracy),"<br>Examples: ",cases)]
p <- ggplot(chart,aes(mean_probability,observed_accuracy,color=Estimate,
                     shape=Estimate,linetype=Estimate,text=Tooltip))+
  geom_abline(slope=1,intercept=0,color="#aab2b9",linetype=3)+
  geom_line()+geom_point(size=2.7)+
  scale_color_manual(values=c("#7d8a94","#236795"))+
  scale_x_continuous(labels=scales::percent,limits=c(0,1))+
  scale_y_continuous(labels=scales::percent,limits=c(0,1))+
  labs(x="Mean estimated first-suggestion correctness",y="Observed first-suggestion accuracy",color=NULL,shape=NULL,linetype=NULL)+
  theme_minimal(base_family="Segoe UI")+theme(legend.position="bottom")
plotly::ggplotly(p,tooltip="text")

What happens when uncertain predictions are skipped?

The following thresholds were fixed before the final test. Answer coverage is the proportion of all 900 cases answered. Selective accuracy uses only answered cases as its denominator. Raising a threshold may improve conditional accuracy while leaving many users without a suggestion.

display <- copy(selective[variant==chosen])
display[,Interval:=vapply(wilson95,fmt_interval,character(1))]
knitr::kable(display[,.(
  `Minimum estimated confidence`=pct(threshold),Answered=answered,
  `Answer coverage`=pct(coverage),`First-suggestion accuracy`=pct(accuracy_first),
  `95% interval for that accuracy`=Interval,`Top-three accuracy among answered`=pct(accuracy_top3))])
Minimum estimated confidence Answered Answer coverage First-suggestion accuracy 95% interval for that accuracy Top-three accuracy among answered
0.0% 900 100.0% 23.8% 21.1% to 26.7% 37.1%
50.0% 80 8.9% 72.5% 61.9% to 81.1% 80.0%
70.0% 35 3.9% 85.7% 70.6% to 93.7% 88.6%
85.0% 15 1.7% 80.0% 54.8% to 93.0% 86.7%
95.0% 4 0.4% 100.0% 51.0% to 100.0% 100.0%

At the 95% estimated-confidence threshold, 4 of 900 cases receive an answer. Observed first-suggestion accuracy is 100.0%, with interval 51.0% to 100.0%. The interval concerns uncertainty in measured accuracy; it is different from the model’s confidence threshold. A high point estimate from very few cases does not establish a reliable 95% service.

Discussion and conclusion

The fixed candidate pool separates score changes from search improvements. Source and token-length results below indicate whether gains are concentrated in one type of text. The combined length/boundary variant tests one interaction; the comparison does not exhaust all possible scoring methods.

The selected rule achieves 37.1% top-three accuracy across all 900 cases. Calibration does not increase that accuracy; it assesses when the first suggestion is more or less trustworthy. Its test Brier score changes from 0.1498 to 0.1460, while answer coverage exposes the cost of abstention. These results must be interpreted together, without presenting selective accuracy as overall performance.

This final sample differs from earlier studies, so differences from their reported percentages do not establish model improvement or deterioration.

The reused development set, one balanced English test sample, one observed continuation per phrase, finite-precision scoring and uncertain pretraining overlap limit generalization. The sigmoid map may be misspecified and can drift on different user text. A comparison with another base model belongs in a separate experiment. Neither the 85% free-word target nor a reliable 95%-confidence claim follows automatically from these results.

Reproducibility

This report is knitted from 30_ranking_and_calibration.Rmd, with code available through Code buttons. 31_verify_ranking_study.R verifies the saved results, prints the main tables and rebuilds the HTML. Source, data fingerprints, case-level scores, calibration coefficients, prediction rows and independent audits are retained in the GitHub project. The reproduction guide records the order of selection, calibration and final testing.

An independent R optimizer reproduced the Python calibration coefficients within 0.000000004 before final testing. Separate checks reconstruct the ranking formulas, prediction metrics, reliability bins and selective-accuracy intervals from saved case-level evidence.

Scoring formulas, subgroup results and technical checks

Let L be the canonical token-sequence log likelihood, B the following boundary log mass, n the number of model tokens, and c the number of characters. The rules are L+B, L, L+0.5B, L/n+B, L/n, L/sqrt(c)+B, and a log-sum-exp over two distinct valid tokenizations with their boundary scores. The original rule preserves its archived floating-point operation order exactly. Neutral alternate-path scores were checked against full-sequence calculations before development evaluation. These heuristic variants are not all normalized word-probability distributions [3].

knitr::kable(groups[split=="final",.(Rule=unname(labels[variant]),Group=group,Cases=cases,
  `First correct`=pct(top1),`First three correct`=pct(top3),Coverage=pct(coverage))])
Rule Group Cases First correct First three correct Coverage
Original score single_token 808 26.5% 41.0% 86.1%
Original score multiple_tokens 92 0.0% 3.3% 15.2%
Original score blogs 300 25.7% 41.7% 81.7%
Original score news 300 22.0% 35.7% 79.7%
Original score twitter 300 23.7% 34.0% 75.3% 
sessionInfo()
## R version 4.6.1 (2026-06-24 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26200)
## 
## Matrix products: default
##   LAPACK version 3.12.1
## 
## locale:
## [1] C
## system code page: 65001
## 
## time zone: Europe/Amsterdam
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] ggplot2_4.0.3       data.table_1.18.6.1
## 
## loaded via a namespace (and not attached):
##  [1] gtable_0.3.6       jsonlite_2.0.0     dplyr_1.2.1        compiler_4.6.1    
##  [5] tidyselect_1.2.1   tidyr_1.3.2        jquerylib_0.1.4    scales_1.4.0      
##  [9] yaml_2.3.12        fastmap_1.2.0      R6_2.6.1           labeling_0.4.3    
## [13] generics_0.1.4     knitr_1.51         htmlwidgets_1.6.4  tibble_3.3.1      
## [17] bslib_0.12.0       pillar_1.11.1      RColorBrewer_1.1-3 rlang_1.3.0       
## [21] cachem_1.1.0       xfun_0.60          sass_0.4.10        S7_0.2.2          
## [25] otel_0.2.0         viridisLite_0.4.3  plotly_4.12.1      cli_3.6.6         
## [29] withr_3.0.3        magrittr_2.0.5     crosstalk_1.2.2    digest_0.6.39     
## [33] grid_4.6.1         lifecycle_1.0.5    vctrs_0.7.3        evaluate_1.0.5    
## [37] glue_1.8.1         farver_2.1.2       rmarkdown_2.31     purrr_1.2.2       
## [41] httr_1.4.8         tools_4.6.1        pkgconfig_2.0.3    htmltools_0.5.9

References

  1. Sahebzad, S. (2026). Target-aware development diagnosis.
  2. scikit-learn documentation. Probability calibration, reliability diagrams and proper scoring rules.
  3. PyTorch 2.8 documentation. Numerical accuracy.
  4. Johns Hopkins University and SwiftKey. Official corpus.
  5. Qwen team. Qwen3-1.7B-Base, revision ea980cb0a6c2ae4b936e82123acc929f1cec04c1.