Validation Record

What every statistic in Stratum is checked against — and where Stratum deliberately differs from other packages.

Every analysis is verified against values computed outside Stratum: by R, SciPy, NumPy, scikit-learn, or the certified NIST Statistical Reference Datasets. Those reference values are committed to the source tree and asserted on every build, so a result cannot quietly stop matching them.

We do not claim that Stratum “matches R”. That claim is not well defined. R's own functions disagree with each other in places — quantile() and boxplot() use different quartiles — and the same rule names mean different things in different packages. What is published here instead is the specific claim, analysis by analysis, naming the tool and the function, together with every place we chose differently and why.

49 analyses. 20 checked against an outside tool · 12 partly checked · 3 pinned by a published formula · 14 no statistic of its own. This page is generated from the same ledger the test suite emits, so it cannot describe evidence that is not there. Last generated 2026-09-10. Reference tool versions: ca (R package) 0.71.1, car (R package) 3.1.5, dunn.test (R package) 1.4.1, effectsize (R package) 1.0.3, effsize (R package) 0.8.1, foreign (R package) 0.8.91, MASS (R package) 7.3.65, MBESS (R package) 5.0.1, mpmath 1.4.1, numpy 2.4.6, pingouin 0.6.1, R 4.6.1, rpart (R package) 4.1.27, scikit-learn 1.9.0, scipy 1.16.3, statsmodels 0.14.6.

How to read this

Checked against an outside tool (20)
Committed reference values from a named function of a named package at a recorded version, asserted on every build.
Partly checked (12)
Some published outputs are verified and others are not — and the entry says which. A fitted random forest cannot be matched tree-for-tree against another library, so its diagnostics are checked and the model itself rests on invariants.
Pinned by a published formula (3)
No package computes the quantity, so the published formula is transcribed independently twice and the two must agree.
No statistic of its own (14)
The chart draws values it was handed. A scatter plot has nothing to verify beyond the numbers given to it.

“Partly checked” is not a disclaimer added after the fact — it names work that is still outstanding, deliberately, because a verification claim later found to be wrong costs more than one never made.

Deliberate divergences

If you reconcile a Stratum result against another package and the numbers differ slightly, these are the likely reasons. Each is a documented choice.

Analysis by analysis

Analysis Status Checked against
SummaryVerified
  • scipy.stats.skew(bias=False)
  • scipy.stats.kurtosis(bias=False, fisher=True)
  • R stats::shapiro.test (Royston 1995, AS R94), cross-checked against scipy.stats.shapiro
FrequenciesVerified
  • R foreign::read.spss(use.missings=TRUE) — R's own C reader applying the .sav dictionary's missing declaration — over five variables of Data/GSS2024.sav, the reviewer's own file
  • R table() / prop.table() / cumsum() for the counts and all three percentage columns
  • R table(x[g == level]) per split cell, each cell's table computed independently, over deliberately unequal cells — so a within-cell denominator cannot be borrowed from the whole file unnoticed
  • R chisq.test(table) and chisq.test(table, p = p/sum(p)) for the goodness-of-fit statistic, its df and its p
  • R binom.test for the exact two-sided binomial offered at two levels; the SPSS one-tailed convention at p ≠ 0.5 is recorded as a divergence rather than reconciled
HistogramVerified
  • R: ceiling(log2(n)+1), 3.49*sd*n^(-1/3), 2*IQR*n^(-1/3) from its own sd/IQR; nclass.Sturges / nclass.scott / nclass.FD recorded beside them (nclass.scott uses 3.5, not 3.49)
  • numpy: the same three rules spelled independently
  • the KDE overlay routes through the shared KernelDensityEngine
Heatmap/HexBinVerified
  • bins BOTH axes with the Freedman-Diaconis rule, checked against R nclass.FD and numpy
MosaicVerified
  • R stats::chisq.test(N, correct=FALSE) — $expected and $residuals, the PEARSON residuals (O-E)/sqrt(E), which is what Stratum reports; $stdres is a different quantity and is recorded alongside for contrast
  • scipy.stats.chi2_contingency for the expected counts, with the residuals formed from them independently
Box-WhiskerVerified
  • R stats::quantile(type=7) — NOT fivenum/boxplot, which use Tukey's hinges and give different quartiles at some sample sizes
  • numpy.quantile (linear == type 7)
  • scipy.stats.skew / kurtosis (bias=False)
  • R moments::skewness / kurtosis, converted from the biased g1/b2 to the unbiased G1/G2
  • McGill-Tukey-Larsen (1978) for the notch, pinned by formula
ViolinVerified
  • the density envelope uses the same kernel-density estimator as the Density chart; the inner box statistics are Box-Whisker's, covered there
DensityVerified
  • R: 0.9*min(sd, IQR/1.3489795)*n^-0.2 and (4/(3n))^0.2*sd from its own sd/IQR, plus the exact kernel sum mean(dnorm((t-x)/h))/h
  • numpy: the same rules and kernel sum spelled independently
  • R bw.nrd0 / bw.nrd recorded alongside — they divide by 1.34, not the exact 1.3489795, so they sit 0.67% away once the IQR branch wins
ECDFVerified
  • R stats::ecdf(x) evaluated at the distinct values and outside both ends
  • numpy (x <= t).sum()/n
CorrelationVerified
  • R stats::cor.test (pearson / spearman / kendall)
  • scipy.stats.pearsonr / spearmanr / kendalltau
  • Bonett-Wright (2000) for the Spearman interval
One-Way ANOVAVerified
  • scipy.stats.f_oneway
  • scipy.stats.kruskal
  • eta^2 / omega^2 / epsilon^2 in exact Fraction arithmetic
  • pingouin.welch_anova (cross-checked against the definition)
  • pingouin.pairwise_gameshowell
  • scipy.stats.studentized_range.sf / .ppf
  • scipy.stats.levene(center="median" / "mean") for the Levene's Test section, both centring variants
  • R dunn.test(method="bonferroni") for Dunn's Test — the pairwise follow-up to Kruskal-Wallis, which scipy does not implement at all. Its p-values are ONE-TAILED and Stratum's are two-sided, so the factor of 2 is asserted per pair; the z and the two-sided p are also re-derived from the definition, and the Kruskal-Wallis statistic the section sits under is cross-checked against scipy.stats.kruskal so a disagreeing ranking cannot pass
Two-Way ANOVAVerified
  • statsmodels.stats.anova_lm(typ=3 / typ=2 / typ=1) on a sum-coded model (== R car::Anova type III / type II / R aov)
  • pooled within-groups SS in exact rational arithmetic
MANOVAVerified
  • statsmodels.multivariate.manova.MANOVA.mv_test()
  • 50-digit mpmath for Rao's F degrees of freedom
Two-SampleVerified
  • scipy.stats.mannwhitneyu
  • scipy.stats.wilcoxon
  • scipy.stats.ttest_ind
  • scipy.stats.norm.sf
  • scipy.stats.t.ppf for the 90/95/99% difference-interval bounds
  • R effectsize::cohens_d(ci=) and MBESS::ci.smd / ci.sm for the NONCENTRAL-t interval on Cohen's d and Hedges' g, with the expected value from base R uniroot on pt(q, df, ncp) — the algorithm both packages run — and scipy.stats.nct as a third, non-R implementation
  • R pt(q, df, ncp) (Lenth 1989, AS 243) and scipy.stats.nct.cdf over a 399-point grid for the noncentral t itself, including the |ncp| > 37.62 range where AS 243 underflows and R is the lane that is wrong
  • Hedges (1981) J = Γ(df/2)/(√(df/2)·Γ((df-1)/2)) — the EXACT correction, asserted equal to R effectsize::hedges_g per case and to 60-digit mpmath. It was Hedges & Olkin's (1985) approximation 1 - 3/(4·df - 1); effsize::cohen.d(hedges.correction=TRUE) still uses that and rides along as the lane Stratum deliberately no longer matches
  • R t.test(mu=) / wilcox.test(mu=) conventions for the hypothesized difference
  • R stats::shapiro.test for the Shapiro-Wilk normality column
  • Schuirmann (1987) TOST — both one-sided tails from scipy.stats.t.sf / .cdf, on the same moments as the t-test
  • scipy.stats.levene(center="median") for the equal-variance column (Brown-Forsythe), which also decides the pooled / Welch choice under the Auto variance mode
  • base R t.test(var.equal=TRUE) and t.test() for the FULL SPSS independent-samples block — both t-tests, fixed and named, printed beside the one in force — with scipy.stats.ttest_ind(equal_var=True/False) as the second lane on each row
DifferencesVerified
  • R stats::chisq.test(correct=FALSE) with V = sqrt(chi2/(n*(k-1)))
  • scipy.stats.chi2_contingency and scipy.stats.contingency.association(method='cramer')
Contingency TableVerified
  • scipy.stats.chi2_contingency(correction=False)
  • scipy.stats.chi2.sf
Principal ComponentsVerified
  • R stats::prcomp(center=TRUE, scale.=TRUE/FALSE)
  • sklearn.decomposition.PCA
  • both re-anchored to Stratum's sign convention, so loadings and scores compare element by element rather than as magnitudes
Linear RegressionVerified
  • NIST StRD Linear Least Squares — certified to 15 digits (Norris, NoInt1, Filip, Longley, Wampler1-5)
  • R car::vif — the per-TERM generalized VIF of Fox & Monette (1992) and its GVIF^(1/(2·Df)) scale, over factor, polynomial and near-singular designs
  • R car::vif on the same design with every column its own term, cross-checked against statsmodels variance_inflation_factor, for the per-DESIGN-COLUMN VIF
Process CapabilityVerified
  • c4 via scipy.special.gammaln, and the half-step gamma ratio it shares with Hedges' J against 60-digit mpmath across df = 1…1e12 — the range where the plain lgamma difference cancels
  • d2 by Gauss-Legendre quadrature of the range distribution
  • numpy sample SD (ddof=1)
  • R MASS::fitdistr (Weibull MLE)
  • R plnorm / pweibull / pexp for the fitted tails and quantiles
  • scipy.stats.weibull_min.fit(floc=0) as the second lane
  • Anderson-Darling critical values re-derived by Monte Carlo (20,000 samples at n = 50, per family) rather than transcribed; normal, log-normal and exponential agree within noise, and the Weibull row's published cutoffs measure about 2% low, which is recorded rather than silently corrected
  • Stephens (1974) JASA 69(347) 730-737 and Stephens (1977) Biometrika 64(3) 583-588, collected in D'Agostino & Stephens (1986), for those cutoffs and the per-family A*² modifications
  • Akaike (1974) IEEE TAC 19(6) 716-723 and Hurvich & Tsai (1989) Biometrika 76(2) 297-307 for the AICc that ranks the families, with its recovery rate measured against the raw-A² rule it replaced (82% against 17% on exponential data at n = 80)
  • ISO 22514-2:2017 for the percentile definition of the indices
Regression PlotVerified
  • R lm(): fitted(), resid(), hatvalues(), rstandard() (internally studentized), rstudent() (externally studentized), cooks.distance() — the plots arrange these and compute nothing of their own
Q-QPartly verified
  • R ppoints(): p = (i-a)/(n+1-2a) with a = 3/8 at n <= 10 and a = 1/2 above, which is the rule Stratum uses; qnorm / -log(1-p) / p / log(-log(1-p)) for the theoretical axes
  • R qqnorm() run on the case's own data — the whole plot from R, not R inverting a position computed here
  • the superseded (i-0.5)/n vector recorded alongside: identical above ten points, different at or below, so a regeneration against the old rule cannot pass
  • normalQuantile pinned to Acklam's published 1.15e-9 RELATIVE bound against R qnorm

Not yet covered: The CONFIDENCE BAND (QQConfidenceBand) has no fixture — only the points and the theoretical axes are covered.

TrendPartly verified
  • R stats::lm for the four forms that are linear in their parameters — raw-power polynomials and lm(y ~ log(x))
  • R stats::nls (algorithm 'port') for the power and exponential forms, y = b0·x^b1 and y = b0·e^(b1·x): nonlinear least squares on the original y scale, started from OLS on the logs — the same estimator nls(), SciPy's curve_fit and JMP's Nonlinear platform compute
  • SciPy optimize.curve_fit (MINPACK Levenberg-Marquardt) as a second independent optimizer on those two, agreeing with nls on the residual sum of squares to 1e-8 — an iterative fit is where a single reference is weakest, because 'converged' is a choice each library makes differently
  • fixtures chosen where the answer is decided by which estimator is running: homoscedastic error across three decades of y, and an outlier set on both forms. Each case pins the original-scale least-squares curve on data where a log-scale fit lands somewhere else entirely, so what is certified is the estimator and not merely that some curve fits
  • r-squared on the ORIGINAL y scale, measured against the back-transformed curve on screen, which is what Stratum reports. R's model-scale summary()$r.squared is recorded alongside every case, so a reading taken from R can be reconciled against it

Not yet covered: The non-parametric smoothers — LOWESS, kernel smoother and moving average — have no fixture. Their implementations differ in detail between packages, so pinning one spelling of LOWESS would pin R's choices rather than a shared definition; closing this means deciding which definition Stratum claims first.

Contour PlotPartly verified
  • grid geometry from the engine's own 2-D rule, h = 1.06*max(sd, range*1e-3)*n^(-1/6)*scale, pad = clamp(2.5h, 3%, 15%) — the bandwidth is not exposed, but x0 and dx are functions of it, so the geometry pins the rule exactly
  • a DIRECT 2-D Gaussian kernel sum in numpy, compared in shape against the engine's binned separable blur within a measured bound (6%, 12% on the coarse grid)

Not yet covered: The blur is an APPROXIMATION to a 2-D KDE, so the surface is checked for shape against the exact estimator rather than value-for-value — the bound is a recorded measurement, not an equality. The value and aggregate modes (with a Z column) and their 2% masking threshold have no fixture. Note this chart does NOT share KernelDensityEngine: it uses the 2-D n^(-1/6) exponent where Density uses n^(-1/5), so it inherits nothing from that fixture.

Decision TreePartly verified
  • sklearn.metrics.roc_curve(drop_intermediate=False) / roc_auc_score
  • sklearn.inspection.partial_dependence(kind='average')
  • R rpart::rpart(...)$cptable — the cross-validated error curve, its xstd, and rpart's own 1-SE selection off it

Not yet covered: The DIAGNOSTICS (ROC, AUC, calibration binning, partial dependence) are pinned, and so now is the PRUNING SELECTION: the standard error of the cross-validated loss reproduces rpart's xstd to the last printed digit on every fixture cptable row, and the 1-SE rule picks the row rpart's own rule picks, on five curves spanning classification and regression (defect A14). The fitted TREE is still not pinned. A single tree can be matched to sklearn exactly given the same split rule and deterministic tie-breaking, so this is a gap to close, not an impossibility — and it is why the end-to-end check holds Stratum to rpart's leaf count only on kyphosis, where rpart's answer is a stump under both rules and Stratum's defaults reach it.

Random ForestPartly verified
  • sklearn.metrics.roc_curve / roc_auc_score

Not yet covered: Diagnostics only. An ensemble cannot be matched tree-for-tree against sklearn — the RNG streams and split tie-breaking differ by construction — so the model itself needs invariants (OOB convergence, importance monotonicity) rather than a value comparison.

Boosted TreesPartly verified
  • sklearn.metrics.roc_curve / roc_auc_score

Not yet covered: Diagnostics only — same reasoning as Random Forest.

Correspondence AnalysisPartly verified
  • R ca::ca()
  • an independent numpy implementation from the definition (Greenacre), not a transcription of the engine
  • scipy.stats.chi2_contingency — pins the identity n * total inertia == chi-square

Not yet covered: The DISPLAY CAP is not covered. Above StratumLimits.maxReportGroups the engine pools the tail into 'Other' and decomposes the POOLED grid, which moves both the inertia and the degrees of freedom; every fixture table is well under the cap. Row/column contributions and quality (cos²) have no fixture yet either.

ClusteringPartly verified
  • R stats::hclust(dist(X), method=single/complete/average/ward.D2)
  • scipy.cluster.hierarchy.linkage
  • R stats::kmeans(nstart=25) under BOTH its Hartigan-Wong default and algorithm='Lloyd', asserted to agree with each other and with sklearn.cluster.KMeans before being used (well-separated data only)
  • sklearn.cluster.KMeans (well-separated data only)
  • R stats::kmeans(X, centers=k) across 40 seeds — R's literal single-start default, carried as a SPREAD rather than a value because on hard data that call has no single answer
  • local optimality: nearest-centroid assignment, and centroid is the cost-minimising centre — mean under Euclidean, median under Manhattan
  • R stats::hclust ON THE DISCLOSED SAMPLE for a run past the O(n²) row cap: hclust has no cap and clusters all n, so a capped run is comparable to it only on the rows Stratum says it clustered. The fixture reproduces the selection rule (rows ordered by their own values, then taken at uniform stride) and hands R exactly those rows, which pins the sample as well as the linkage

Not yet covered: The k-means PARTITION is compared to scikit-learn and R only where the optimum is unambiguous. On ambiguous data, and under the Manhattan cost (which sklearn's KMeans does not implement), the evidence is invariants plus R's own objective as an upper bound, rather than a partition — because no outside implementation reaches a canonical answer there, not because the check was skipped. Elbow and silhouette have no fixture yet. TWO DEFAULTS DIVERGE FROM R BY DESIGN and are documented rather than changed (defect A13): Stratum standardises before clustering and R clusters the scale it is handed, and Stratum runs Lloyd from k-means++ with 10 restarts where R runs Hartigan-Wong from one random start. Neither is an arithmetic difference — both minimise the same total within-cluster sum of squares — and the fixture measures both: the same 45 rows are clustered twice, once on the raw scale R would use and once standardised the way Stratum does, each partition matched to R and sklearn on its own scale, with 21 rows landing in a different cluster between them. On the ALGORITHM half the evidence is asymmetric and the record says so. Where the three references agree what the optimum is, Stratum's 10 restarts reach it and R's single default start frequently does not — on three tight, well-separated blobs R's default lands at 35.05 instead of 0.050 on some seeds, and on the four-cluster 3-D case it reaches nine distinct optima across 40 seeds (best 1.84, worst 32.7). Where the optimum is AMBIGUOUS neither default is authoritative and Stratum is not always ahead: on the overlapping-ridges case R's default reaches four local optima (35.594, 35.665, 35.764, 39.269) and Stratum's 10 restarts land on a fifth at 35.802 — 0.11% above R's median start and 0.58% above the 35.594 both programs reach given more starts. So the assertion carried there is only that Stratum is no worse than the WORST R's own default hands a user, beside local optimality; raising Restarts closes the gap (25 → 35.659, 200 → 35.594). Matching R's ALGORITHM NAME would not make Stratum match R's answer — R's default seeds at random and its RNG stream cannot be reproduced here, and as that spread shows R does not match itself either — so the divergence is recorded, not papered over.

I-MR ChartPartly verified
  • d2 by Gauss-Legendre quadrature
  • c4 via scipy.special.gammaln

Not yet covered: The unbiasing CONSTANTS are pinned to first principles; the Nelson / Western Electric run rules are not, and they are a documented convention rather than a computed quantity.

X-bar ChartPartly verified
  • d2 / c4 as for I-MR

Not yet covered: Constants pinned; run rules are convention. Same as I-MR.

P ChartPartly verified
  • binomial limits

Not yet covered: Attribute-chart limits are not yet cross-checked against an outside tool.

C ChartPartly verified
  • Poisson limits

Not yet covered: Attribute-chart limits are not yet cross-checked against an outside tool.

ParetoBy formula

ranking and a running total are exact arithmetic, so there is no outside tool to disagree with. What is pinned is falsifiable and was: the tie-break is the CATEGORY LABEL (so equal frequencies order deterministically rather than by dictionary order), and topN truncation does NOT renormalise — the cumulative line runs against the FULL total and stops short of 100%. Renormalising it trips 42 assertions.

CUSUM ChartBy formula

Montgomery's CUSUM recursion — C+ = max(0, C+ + z - k), C- = max(0, C- - z - k), FIR headstart h/2 — transcribed INDEPENDENTLY in R and numpy and asserted equal before either was recorded. No qcc/spc is installed, so this is the published formula rather than a package's implementation; installing either upgrades this to external. The sigma it standardises by is MRbar/d2, whose unbiasing constant is pinned to first principles under I-MR Chart. NOT covered: the ARL.

EWMA ChartBy formula

Montgomery's EWMA recursion with the TIME-VARYING limit width L*sigma*sqrt(lambda/(2-lambda)*(1-(1-lambda)^(2i))), transcribed independently in R and numpy. The test additionally asserts the band widens monotonically, so substituting the asymptotic width fails even where the two happen to be close. Same qcc/spc caveat as CUSUM.

DataNo statistic

The data sheet. Displays stored values.

Dot/StripNo statistic

Plots raw values along one axis.

PieNo statistic

Shares of a supplied total.

TreemapNo statistic

Areas proportional to a supplied value.

BarNo statistic

Plots the aggregate the control bar selects; the aggregates themselves are Summary's, and covered there.

Dot PlotNo statistic

Plots raw values along one axis.

DumbbellNo statistic

Draws two supplied values per category.

SlopegraphNo statistic

Draws two supplied values per category.

ScatterNo statistic

Plots raw x/y pairs. Any fitted overlay is a separate mode with its own entry.

BubbleNo statistic

Plots raw x/y pairs with a size channel.

PCA Scree PlotNo statistic

Plots the PCA eigenvalues, which are covered under Principal Components.

PCA BiplotNo statistic

Plots the PCA loadings and scores, which are covered under Principal Components.

Correspondence MapNo statistic

Plots the correspondence-analysis coordinates, which are covered under Correspondence Analysis.

Parallel PlotNo statistic

Draws one polyline per row over scaled axes.

Finding a disagreement

If a Stratum result differs from another package in a way this page does not explain, please tell us — send the data and both results to support.