Purpose

This analysis applies multiple comparison correction methods to 84 educational hypothesis tests comparing student performance across demographic groups (lunch type, test prep, parental education, race/ethnicity). The objective is to control family-wise error rate (FWER) and identify which differences remain statistically significant after accounting for the inflated Type I error risk inherent in conducting 84 simultaneous tests.

Key Findings

• Uncorrected FWER: 0.987 – Without correction, there is a 98.7% probability of at least one false positive across all 84 tests, demonstrating critical need for adjustment
• Holm-Bonferroni Rejections: 36 significant findings – Holm method retains 36 rejections while controlling FWER at α=0.05
• Bonferroni Rejections: 29 findings – More conservative; Holm gains 24% additional power over Bonferroni
• BH (FDR) Rejections: 56 findings – Most liberal method, controlling false discovery rate rather than FWER
• Practical Significance Alignment: 36 tests flagged as practically significant (effect size ≥0.2), perfectly matching Holm rejections, indicating strong concordance between statistical and practical significance

###

Purpose

This section documents the data preprocessing pipeline for 84 hypothesis tests undergoing multiple comparison corrections. Perfect data retention (100%) indicates no observations were excluded during cleaning, which is critical for maintaining the integrity of family-wise error rate (FWER) control calculations that depend on the complete test set.

Key Findings

• Initial & Final Rows: 84 observations retained across both stages—no data loss occurred during preprocessing
• Retention Rate: 100% indicates complete dataset integrity with zero exclusions
• Rows Removed: 0 deletions suggests either pristine input data or that no quality thresholds triggered removal criteria
• Train/Test Split: Not applicable, as this is a multiple comparison correction analysis rather than a predictive modeling task

Interpretation

The perfect retention rate ensures that all 84 tests remain eligible for correction methods (Holm, Bonferroni, BH, Hochberg). This is essential because FWER calculations depend on the complete family of tests; removing even one test would alter the stepdown thresholds and rejection decisions. The absence of preprocessing transformations preserves the raw p-values and effect sizes needed for accurate multiple comparison control.

Context

No train/test split is applicable here since the objective is statistical inference across a fixed test family, not predictive generalization. The analysis assumes all 84 tests

EXECUTIVE SUMMARY: MULTIPLE COMPARISON CORRECTION ANALYSIS

Purpose

This analysis applied four statistical correction methods to 84 hypothesis tests comparing student performance across demographic groups (lunch type, test prep, race/ethnicity, parental education). The objective was to identify which findings remain statistically significant after controlling for multiple testing inflation, which would otherwise produce a 98.7% probability of at least one false positive.

Key Findings

• Uncorrected FWER: 98.7% — Without correction, the analysis would almost certainly report false positives across the 84 tests
• Holm-Bonferroni Rejections: 36 tests — Maintains family-wise error rate at 5% while rejecting 7 more tests than strict Bonferroni (24.1% power gain)
• Bonferroni Rejections: 29 tests — Most conservative FWER method; rejects only the strongest signals
• BH (FDR) Rejections: 56 tests — Less stringent control; allows up to 5% false discovery rate rather than protecting against any false positive
• Effect Size Alignment: 36 tests with practically significant effects (≥0.2) match Holm rejections, indicating statistical and practical significance overlap

Interpretation

The analysis successfully

Purpose

The Holm step-down procedure controls family-wise error rate (FWER) by sequentially testing hypotheses in order of increasing p-value, with thresholds that become progressively more lenient. This section demonstrates how Holm allocates the significance budget across 84 tests while maintaining strict control over false positives—critical for identifying genuinely significant effects amid multiple comparisons.

Key Findings

• Holm Rejections: 36 of 84 tests rejected—57% retention rate reflects conservative FWER control
• Threshold Range: 0 to 0.05, decreasing as α/(m-i+1), creating a staircase pattern that tightens early tests most severely
• Marginal Test: Writing: associates degree vs Some high school (p=0.0009) marks the rejection boundary; tests beyond this fail their thresholds
• Raw p-value Distribution: Mean=0.09, median=0, indicating strong clustering of significant results at the lower tail

Interpretation

Holm's sequential rejection strategy is more powerful than Bonferroni (24% power gain) because it relaxes thresholds for weaker tests while maintaining FWER at α=0.05. The 36 rejections represent findings robust to multiple comparison correction. The marginal test at

Purpose

This section compares how four multiple-comparison correction methods adjust raw p-values to control error rates across 84 simultaneous hypothesis tests. It demonstrates the trade-off between statistical conservatism (fewer false positives) and statistical power (ability to detect true effects). Understanding these differences is critical for determining which findings remain credible after correcting for multiple testing.

Key Findings

• Holm vs. Bonferroni: Holm rejected 36 tests versus Bonferroni's 29—a 24.1% power gain while maintaining identical FWER control, demonstrating Holm's superiority as a step-down procedure.
• BH (FDR) Method: Rejected 56 tests, the most permissive approach, because it controls False Discovery Rate rather than Family-Wise Error Rate, allowing more discoveries at the cost of accepting some false positives.
• Hochberg Performance: Matched Holm's 36 rejections despite weaker assumptions, suggesting strong positive dependence among test statistics in this dataset.
• Uncorrected FWER: At 0.987, the probability of at least one false positive without correction is nearly certain across 84 tests.

Interpretation

The analysis reveals a clear hierarchy: Bonferroni is most conservative, Holm and

Purpose

This section visualizes which of the 84 hypothesis tests remain statistically significant under four different multiple comparison correction methods. It reveals the power-conservativeness tradeoff: stricter methods (Bonferroni, Holm) reject fewer tests but provide stronger Type I error control, while less conservative methods (BH, Hochberg) detect more signals by controlling different error rates. Understanding method agreement identifies robust findings versus marginal discoveries.

Key Findings

• Bonferroni Rejections: 29 tests (35%) — most conservative FWER control; highest false negative risk
• Holm Rejections: 36 tests (43%) — 24% power gain over Bonferroni while maintaining FWER control
• BH Rejections: 56 tests (67%) — most liberal; controls FDR rather than FWER, detecting 93% more signals
• Hochberg Rejections: 36 tests (43%) — matches Holm; stepdown improvement over Bonferroni
• Perfect Agreement: First 5 tests rejected by all methods; last 5 retained by all methods — indicating clear signal/noise separation at distribution extremes

Interpretation

The decision matrix demonstrates that 36 tests form a robust consensus across FWER-controlling methods

Purpose

This section compares the statistical power and rejection rates across four multiple comparison correction methods applied to the same 84 hypothesis tests. It demonstrates the trade-off between controlling false positives (Type I error) and detecting true effects (statistical power), which is central to choosing an appropriate correction strategy for the analysis objective.

Key Findings

• Benjamini-Hochberg (BH) Rejections: 56 tests (66.7%) — highest rejection rate with 1.93× power relative to Bonferroni, controls False Discovery Rate rather than Family-Wise Error Rate
• Holm & Hochberg Rejections: 36 tests each (42.9%) — equivalent rejection counts despite different algorithmic approaches; Holm uses step-down logic while Hochberg uses step-up
• Bonferroni Rejections: 29 tests (34.5%) — most conservative method with lowest power; provides strongest control but sacrifices detection ability
• Power Differential: 27-test gap between BH (most powerful) and Bonferroni (least powerful) reflects fundamental difference between FDR and FWER control philosophies

Interpretation

The 93% increase in rejections from Bonferroni (29) to BH (56) illustrates why method selection depends on study context.

Purpose

This section validates whether statistically significant findings (after Holm-Bonferroni correction) also represent meaningful, real-world differences. Statistical significance alone can be misleading with large sample sizes; effect size analysis ensures that rejected hypotheses reflect substantive rather than trivial differences. This bridges the gap between p-values and practical importance.

Key Findings

• Practical Significance Alignment: 36 of 84 tests (43%) are both statistically significant (Holm-corrected) AND practically significant (Cohen's d ≥ 0.20), indicating robust, meaningful effects.
• Mean Effect Size: 0.35 (medium effect), with median 0.34 and range 0.02–0.94, showing considerable variation in effect magnitudes across comparisons.
• Confidence Interval Coverage: 69% of tests flagged as "Practically significant" have confidence intervals excluding zero, strengthening evidence of real differences.
• Small Effect Concern: 26 tests show small effects (d < 0.20), suggesting some Holm-rejected findings may be statistically robust but practically negligible due to large sample sizes.

Interpretation

The analysis reveals a healthy concordance between statistical and practical significance for the majority of findings. The 36 practically significant results represent genuine, meaningful differences in student outcomes across

Purpose

This section quantifies the multiple testing problem: conducting 84 independent hypothesis tests at α=0.05 without correction inflates the probability of at least one false positive to 98.7%. This demonstrates why statistical correction methods are essential and motivates the application of Holm-Bonferroni and alternative procedures in the overall analysis.

Key Findings

• Uncorrected FWER: 98.7% — nearly certain to observe at least one false positive by chance alone
• Mathematical Driver: FWER = 1 − (0.95)^84 = 0.987, showing exponential accumulation of error across tests
• Critical Threshold: Even 5 tests yield 23% FWER; by 80+ tests, false positive risk approaches certainty
• Correction Necessity: Holm-Bonferroni reduces FWER back to the nominal 5% level through adjusted p-value thresholds

Interpretation

Without correction, the 84 comparisons across Math, Reading, and Writing assessments would produce a 98.7% probability of spurious significance—rendering raw p-values unreliable for decision-making. The FWER curve illustrates that error accumulation is non-linear: the first 10 tests contribute modest inflation, but by test

Purpose

This section presents the complete numerical results from all 84 hypothesis tests with raw and adjusted p-values across four correction methods. It enables users to identify which findings remain statistically significant after controlling for multiple comparisons—the core objective of the analysis. By displaying all rejection decisions side-by-side, it facilitates comparison of method stringency and power.

Key Findings

• Holm Rejections: 36 of 84 – Conservative stepdown procedure retains 43% of significant findings while controlling family-wise error rate
• BH Rejections: 56 of 84 – False discovery rate control yields 93% more rejections than Bonferroni, reflecting its higher power
• Effect Size Alignment: 36 tests – Exactly match Holm rejections with practically significant effects (Cohen's d ≥ 0.2), indicating statistical and practical significance align
• Uncorrected FWER: 0.987 – Without correction, 98.7% probability of at least one false positive across 84 tests, justifying the correction approach

Interpretation

The analysis reveals a clear hierarchy: Bonferroni (most conservative, 29 rejections) < Holm/Hochberg (moderate, 36 rejections) < BH (least conservative, 56

Purpose

This section compares four multiple comparison correction methods to demonstrate why Holm-Bonferroni was selected as the primary approach. It shows the trade-off between statistical power (ability to detect true effects) and error control (preventing false positives) across methods with different assumptions and objectives. Understanding these differences is critical for interpreting which of the 36 Holm-rejected hypotheses represent genuine findings versus false discoveries.

Key Findings

• Bonferroni (29 rejections): Most conservative FWER method; 24% less powerful than Holm but requires no dependence assumptions
• Holm (36 rejections): 24% more powerful than Bonferroni with identical FWER control; assumes independence or positive dependence
• Hochberg (36 rejections): Matches Holm's power but requires stronger assumptions; ranks 2nd in power
• Benjamini-Hochberg (56 rejections): 93% more powerful than Bonferroni but controls FDR (looser standard) rather than FWER; allows ~5% false discovery rate among rejected tests

Interpretation

The 7-test gap between Holm (36) and Bonferroni (29) represents the power gain from assuming test dependence structure.