Does the Drug Work? It Depends Who You Ask

Here is a fact from a real, landmark drug trial that should stop you cold. In ACTG 175 — 2,139 HIV patients, randomized in the mid-1990s to one of four drug regimens, immune (CD4) cell counts measured at baseline and again 20 weeks later — the counts, on average, went up: 350 → 371, a gain of about 21 cells. The drugs work. Ship the slide.

Except they don't all work. Break that one average out by treatment arm:

• ZDV + ddI: CD4 change +54.5
• ddI alone: +26.9
• ZDV + ddC: +19.3
• ZDV alone — the standard of care, the drug nearly every HIV patient in the world was taking — −17.1. Counts went down.

The +21 overall wasn't a result. It was three arms winning loudly enough to drag a harmful arm into positive territory. A rushed analyst reports "the therapy improves immune function," attaches a chart, and is confident, defensible-looking, and wrong for a huge slice of the patients.

A good analyst gets suspicious of any single average that has to speak for four very different groups. "Does the drug work" is not a question with one number attached. This is the story of finding the real answer — a tour of the tools (a paired t-test, an effect size, a responder analysis, ANOVA, MANOVA and an ANCOVA regression) you need to do it on your own trial data.

Step 0 — Two minutes of housekeeping, and a check that randomization worked

The file arrives with coded factors already decoded to readable labels. Two small setup steps pay for themselves all the way down. First, prior-treatment history is ordinal — naive < 1–52 weeks < >52 weeks — so drag the levels into that order once (Data ▸ Set Category Order) and every table below reads in sequence, not alphabetically. Second, one derived column from the formula engine — cd4_change = cd4_week20 − cd4_baseline — the hinge of the whole analysis.

Now the check that makes everything afterward causal: average baseline CD4 by arm is ≈ 350 in all four groups (347–353). The arms start even, so any gap at week 20 is caused by the treatment, not by who landed where.

Step 1 — The paired test, asked twice

Every patient has two measurements — baseline and week 20 — so the right tool is a paired t-test, comparing each person to their own earlier self (Analyze ▸ Two-Sample, pick the two columns, turn on Pair Data). Across all 2,139 patients: mean change +20.8, t ≈ 7.9, p ≈ 5.7 × 10⁻¹⁵, and because CD4 counts are skewed, the Wilcoxon signed-rank agrees. Counts rose. Statistically ironclad — and misleading, for the exact reason the cold-open chart showed.

So ask a better question. Same test, but first filter to a single arm — ZDV alone — and re-run it. Now the mean change is −17.1 cells, p ≈ 0.0002. Same paired test, same trial, opposite conclusion (how the paired t-test works). The test was never wrong; the population you run it on is the entire ballgame.

Step 2 — Effect size, not just significance

Put the two worlds head to head — ZDV alone vs. everything else — with a two-sample Welch test on CD4 change (equal-variances off, because the groups differ in size and spread). Monotherapy averages −17.1; the other regimens average +33.3 — a 50-cell gap, t ≈ 9.2, p ≈ 3 × 10⁻¹⁹, with Mann–Whitney concurring without trusting the shape of the data.

But with 2,139 patients, everything is significant. The number that matters is the effect size: Cohen's d = 0.42 (95% CI 0.32–0.52) — a real, moderate separation, roughly four-tenths of a standard deviation between taking the old drug alone and taking almost anything else.

Step 3 — What fraction of patients actually got better?

Doctors don't treat averages. Define a responder — CD4 higher at week 20 than baseline — as one formula column (if(cd4_week20 > cd4_baseline, "Responder", "Non-responder")), and cross-tabulate it against arm. For a binary outcome versus a group, the honest, direct tool is a contingency table — no logistic regression required.

χ² ≈ 50.5, p ≈ 6 × 10⁻¹¹, Cramér's V ≈ 0.15 — a modest but genuine association. Put plainly: move a patient from the old monotherapy to the best combination, and you convert roughly one in five non-responders into a responder. That's not a p-value; that's a treatment decision.

Step 4 — Depends who you ask

Here is the heart of it. A one-way ANOVA on CD4 change by arm is overwhelming (F ≈ 32, p ≈ 3 × 10⁻²⁰), and a Tukey test pins ZDV-alone significantly below all three other arms (one-way ANOVA, explained). But "the arms differ" is still an average across every kind of patient. Re-run the same ANOVA with a By-Variable on prior-treatment history, and the drug stops having one answer:

Same drug, same dose. For a patient who'd never taken it, ZDV alone is roughly neutral; for a patient who'd taken it before — and grown resistant — it actively lowers CD4. The combinations help both groups, and help the naive ones most. Averaging those patients together is how you manufacture a confident report that's wrong for almost everyone in it.

Step 5 — More than one endpoint: MANOVA

Immune health isn't one number — there's CD4 and CD8. Testing them separately and hoping they agree is sloppy; MANOVA tests them jointly, accounting for the fact that they move together. On CD4 and CD8 at week 20 by arm: Wilks' Λ ≈ 0.971, F(6, 4268) ≈ 10.4, p ≈ 0. The arms occupy genuinely different positions in the joint immune profile, not just one marker.

Step 6 — The precise, defensible number: ANCOVA

Filtering proves the point one slice at a time; a regression holds everything constant at once. Model week-20 CD4 from the arm alone and R² is a pathetic 0.02. Add baseline CD4 as a covariate and R² leaps to 0.36 — a patient's starting count (coefficient 0.71) is by far the strongest predictor of where they end up. That's the variable a plain group comparison forgets to adjust for (building a regression that isn't fooled).

This is analysis of covariance, run in Stratum as an ordinary regression with a categorical arm plus a numeric baseline. Read the coefficients at equal baseline:

Holding starting CD4 fixed, ZDV + ddI buys 70 more CD4 cells at week 20 than ZDV alone — tight confidence interval, p at the floor (reading a regression you can trust). That's the sentence for the regulatory filing: not "the drugs work," but "for a patient at this baseline, this regimen delivers 70 cells the old one doesn't."

What we actually did

A paired test that flipped sign when we asked it the right question, an effect size to separate real from merely significant, a responder table for the number a clinician can use, a subgroup ANOVA that found the drug means different things to different patients, a MANOVA across two endpoints, and an ANCOVA that turned "it works" into "70 cells at equal baseline" — six lenses on one real trial, no code, about twenty minutes.

We started with "does the drug work" and a chart that said yes. We finished knowing it depends on which drug, which endpoint, and above all which patient — with a precise number on each. That's the difference between a chart and an analysis.