Executive Summary

Readiness of 5 columns for parametric analysis

Columns Screened

Approximately Normal

Non-Normal or Skewed

Extreme Outliers Flagged

Nonparametric Recommended

Most Non-Normal Column

Latency ms

Of 5 screened columns, 2 look approximately normal and 3 are skewed or heavy-tailed. The single most non-normal column is Latency ms (Shapiro-Wilk p < 0.001). Across the columns, 12 extreme and 36 mild outliers were flagged. 3 columns would be better served by a nonparametric method or a transformation; the rest are fit for parametric methods.

Interpretation

Of 5 screened columns, 2 are approximately normal and 3 are skewed or heavy-tailed. Score A and Score B pass normality checks (Shapiro-Wilk p = 0.511 and 0.319 respectively), while Latency ms, Monthly Income, and Sensor Reading depart significantly (all with Shapiro-Wilk p < 0.001). Across all columns, 12 extreme and 36 mild outliers were detected. Latency ms is the most non-normal column (skewness = 8.43, excess kurtosis = 72.63). The recommendation is clear: 3 columns warrant nonparametric methods or transformation; the remaining 2 are fit for parametric analysis.

Overview

Analysis Overview

Normality and outlier screening across 5 columns and 250 observations.

N Observations250

N Columns5

N Normal2

N Nonparametric3

Interpretation

Parametric tests (t-test, ANOVA, linear regression) rest on the assumption that data are approximately normal and free of extreme outliers. This screening examines all 5 columns across 250 observations to detect departures before analysis begins. The process measures shape (skewness and excess kurtosis), runs Shapiro-Wilk normality tests, and flags outliers using both the 1.5 and 3 times interquartile-range fences and a Grubbs test. The goal is diagnostic: to identify which columns are safe for parametric methods and which require nonparametric alternatives or transformation. Screening reveals problems but does not resolve them—any remedial action remains a deliberate analytical choice.

Data Preparation

Data Quality

Column typing, constant-column handling, and any Shapiro-Wilk sampling.

Initial Rows250

Final Rows250

Columns Screened5

Columns Dropped0

Interpretation

All 250 rows loaded successfully with 5 numeric columns screened. No columns were dropped as non-numeric, and no constant columns were flagged. Every mapped column was numeric, varied, and complete; no missing values or data cleaning was required. The dataset was small enough (n = 250) that Shapiro-Wilk testing could be applied directly to each column without random sampling. Data quality was clean across the board, allowing direct assessment of normality and outlier patterns without preprocessing complications.

Data Table

Normality by Column

Shape, Shapiro-Wilk normality test, and a verdict for each column.

Column	N	Skew	Kurtosis	Shapiro P	Verdict
Score A	250	0.1	0.24	0.511	approximately normal
Score B	250	0	-0.15	0.319	approximately normal
Monthly Income	250	3.25	18.6	<0.001	right-skewed
Latency ms	250	8.43	72.63	<0.001	right-skewed
Sensor Reading	250	1.04	10.88	<0.001	right-skewed

Interpretation

Normality assessment reveals a split dataset. Score A (skew = 0.1, kurtosis = 0.24, p = 0.511) and Score B (skew = 0, kurtosis = −0.15, p = 0.319) both meet normality criteria with symmetric distributions and Shapiro-Wilk p-values well above 0.05. The other three columns show pronounced right-skew: Monthly Income (skew = 3.25, kurtosis = 18.6, p < 0.001), Latency ms (skew = 8.43, kurtosis = 72.63, p < 0.001), and Sensor Reading (skew = 1.04, kurtosis = 10.88, p < 0.001). Latency ms stands out with extreme positive skewness and kurtosis, indicating a highly right-skewed distribution with very heavy upper tails. All five columns had n = 250 observations evaluated.

Data Table

Outliers by Column

IQR-fence outlier counts and a Grubbs test on the most extreme value.

Column	Mild Outliers	Extreme Outliers	Most Extreme Value	Grubbs P
Score A	3	0	17.4	0.184
Score B	3	0	138.14	1.000
Monthly Income	11	4	24,507.76	<0.001
Latency ms	3	3	175	<0.001
Sensor Reading	16	5	159.56	<0.001

Interpretation

Outlier detection using IQR fences (1.5× and 3× the interquartile range) and Grubbs testing identified 12 extreme and 36 mild outliers across the dataset. Sensor Reading carries the highest extreme outlier count (5 beyond 3× IQR) alongside 16 mild outliers. Monthly Income has 4 extreme outliers (most extreme value = 24,507.76, Grubbs p < 0.001) and 11 mild outliers. Latency ms shows 3 extreme and 3 mild outliers (most extreme = 175, Grubbs p < 0.001). Score A and Score B are cleaner, each with 3 mild outliers and no extreme outliers; their most extreme values (17.4 and 138.14) do not reach statistical significance on the Grubbs test (p = 0.184 and 1.000). The three columns with genuine outliers align with those failing normality tests.

Visualization

Skewness by Column

Absolute skewness per column against the plus-one readiness line.

Interpretation

Absolute skewness provides a practical rule of thumb: columns above ±1 lean far enough that parametric methods become risky. Three columns exceed this threshold. Latency ms shows the steepest lean (absolute skewness = 8.433), followed by Monthly Income (3.253) and Sensor Reading (1.043). Score A (0.096) and Score B (0.001) remain well below the guideline, confirming their suitability for parametric analysis. For the three non-normal columns, either a nonparametric alternative (Mann-Whitney U, Kruskal-Wallis, Spearman rank correlation) or a transformation (log, Box-Cox) is the safer analytical path. Parametric methods applied to Latency ms, Monthly Income, or Sensor Reading risk inflated Type I error and unreliable confidence intervals.

Methodology

Statistical methodology and diagnostics for Normality & Outlier Screening

Statistical Method

Normality & Outlier Screening

Standard-library analysis: before you run a t-test, ANOVA, or regression, check whether each numeric column is fit for those methods. For every column it reports the mean, median, spread, skewness and kurtosis, runs the Shapiro-Wilk normality test, counts outliers with the 1.5 and 3 times IQR rules plus a Grubbs test on the single most extreme value, and gives a clear parametric-versus-nonparametric recommendation. Works on any dataset: map the numeric columns you plan to test.

Data

N = 250 observations

Assumptions

Each screened column is numeric or cleanly convertible to numeric
Rows are independent observations

Limitations

Normality tests grow very sensitive on large samples — a tiny, harmless departure can register as significant, so read the skewness and kurtosis alongside the p-value
The Grubbs test checks only the single most extreme value; several outliers can mask one another
Screening flags problems but does not fix them — transforming or trimming is a separate decision

Software & Citation

MCP Analytics · mcpanalytics.ai

Code Appendix

Analysis Code

Complete R source code for this analysis

Normality & Outlier Screening — Is Your Data Ready?

Screens each numeric column the user maps before a parametric test: per column it reports n, mean, median, standard deviation, skewness and excess kurtosis, runs the Shapiro-Wilk normality test, counts outliers with the 1.5 and 3 times IQR fences plus a Grubbs test on the single most extreme value, flags robust-z outliers (median/MAD), and returns a clear parametric-versus-nonparametric recommendation.

Why This Method?

The t-test, ANOVA, correlation and linear regression all lean on roughly normal, outlier-free columns. A five-minute screen up front — normality test, skew and kurtosis, and an outlier count — tells you whether those assumptions hold or whether a nonparametric method or a transformation is the safer path, before a single misleading p-value is reported.

What This Analysis Covers

Per-column shape: mean, median, spread, skewness, excess kurtosis
Shapiro-Wilk normality test (sampled to 5,000 when a column is larger)
Outliers by the 1.5 and 3 times IQR fences, plus a Grubbs test
A parametric-versus-nonparametric recommendation for each column

Standard Library

Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {column_1..column_N}. All narrative is derived from the user's own column names and computed values.

suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))

Core Analysis Pipeline

p-value: invert G -> t, then Bonferroni over n observations (two-sided)

u <- (n * G^2) / ((n - 1)^2)
  if (u >= 1) {
    p <- 0
  } else {
    t2 <- u * (n - 2) / (1 - u)
    tval <- sqrt(t2)
    p <- n * 2 * pt(-tval, df = n - 2)
  }
  p <- min(1, max(0, p))

critical G at alpha via the two-sided t quantile

tcrit <- qt(alpha / (2 * n), df = n - 2, lower.tail = FALSE)
  crit <- ((n - 1) / sqrt(n)) * sqrt(tcrit^2 / (n - 2 + tcrit^2))

  list(G = G, crit = as.numeric(crit), p = p,
       value = as.numeric(x[i]), flagged = isTRUE(G > crit))
}

compute_shared <- function(df, params, col_map = list()) {
  # === SHARED EXPORTS ===
  #   initial_rows/final_rows/rows_removed  $ row accounting
  #   col_names          $ named character — semantic -> humanized names
  #   used_cols          $ character — semantic names screened (incl. constant)
  #   dropped_cols       $ character — excluded (non-numeric / empty)
  #   stats              $ per-column list of computed stats (aligned to used_cols)
  #   normality_df       $ column, n, skew, kurtosis, shapiro_p, verdict
  #   outliers_df        $ column, mild_outliers, extreme_outliers,
  #                        most_extreme_value, grubbs_p
  #   skew_by_column_df  $ column, abs_skew  (bar; constant/NA -> 0)
  #   n_used/n_constant  $ counts
  #   n_normal/n_nonnormal/n_nonparam  $ verdict + recommendation counts
  #   total_mild/total_extreme         $ outlier totals across columns
  #   n_sampled          $ how many columns were Shapiro-sampled at 5,000
  #   most_name/most_p   $ single most non-normal column + its Shapiro p
  #   metrics / json_output
  # === /SHARED EXPORTS ===

Step 1: Discover mapped columns (column_1..column_N)

initial_rows <- nrow(df)
  col_cols <- grep("^column_[0-9]+$", names(df), value = TRUE)
  col_cols <- col_cols[order(as.integer(sub("^column_", "", col_cols)))]
  if (length(col_cols) < 1) {
    stop("column_mapping must map at least one numeric column(column_1).")
  }
  col_names <- setNames(humanize_semantic(col_cols, col_map), col_cols)

Step 2: Coerce numeric (95% rule); drop non-numeric and report them.

Unlike relationship tools, screening keeps the actual values — missing entries are set aside per column (na.omit), never imputed, so the shape and outlier counts describe the data as it stands.

dropped_cols <- character(0)
  for (cc in col_cols) {
    v <- df[[cc]]
    if (!is.numeric(v)) {
      conv <- suppressWarnings(as.numeric(as.character(v)))
      n_orig <- sum(!is.na(v) & as.character(v) != "")
      if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
        df[[cc]] <- conv
      } else {
        dropped_cols <- c(dropped_cols, cc); next
      }
    }
    v <- suppressWarnings(as.numeric(df[[cc]]))
    if (all(is.na(v))) { dropped_cols <- c(dropped_cols, cc); next }
    df[[cc]] <- v
  }
  used_cols <- setdiff(col_cols, dropped_cols)
  if (length(used_cols) < 1) {
    stop(paste0("Screening needs at least one usable numeric column; none of ",
                "the mapped columns(",
                paste(unname(col_names[col_cols]), collapse = ", "),
                ") were numeric."))
  }
  final_rows <- initial_rows
  rows_removed <- 0L
  if (final_rows < 10) stop(sprintf("Only %d rows — screening needs at least 10.", final_rows))

Step 3: Per-column screen

n, moments, Shapiro-Wilk (sampled >5,000, seed 42), IQR + Grubbs + robust-z outliers, verdict, and a parametric-vs-nonparametric call.

stats <- list()
  for (cc in used_cols) {
    hn <- col_names[[cc]]
    x_all <- suppressWarnings(as.numeric(df[[cc]]))
    x <- x_all[is.finite(x_all)]
    n <- length(x)
    n_missing <- sum(!is.finite(x_all))

    mean_v <- if (n > 0) mean(x) else NA_real_
    median_v <- if (n > 0) median(x) else NA_real_
    sd_v <- if (n > 1) sd(x) else NA_real_
    constant <- (n < 2) || !is.finite(sd_v) || sd_v == 0

    sk <- if (constant) NA_real_ else skewness(x)
    ku <- if (constant) NA_real_ else excess_kurtosis(x)

Shapiro-Wilk: valid for 3 <= n <= 5000; sample larger columns.

shapiro_p <- NA_real_
    sampled <- FALSE
    shap_n <- n
    if (!constant && n >= 3) {
      xs <- x
      if (n > 5000) {
        set.seed(42)
        xs <- x[sample(n, 5000)]
        sampled <- TRUE
        shap_n <- 5000L
      }
      sp <- tryCatch(shapiro.test(xs)$p.value, error = function(e) NA_real_)
      shapiro_p <- as.numeric(sp)
    }

IQR fences: mild = beyond 1.5 x IQR, extreme = beyond 3 x IQR.

mild <- 0L; extreme <- 0L
    if (n > 0 && !constant) {
      q <- quantile(x, c(0.25, 0.75), names = FALSE, type = 7)
      iqr <- q[2] - q[1]
      if (is.finite(iqr) && iqr > 0) {
        mild <- as.integer(sum(x < q[1] - 1.5 * iqr | x > q[2] + 1.5 * iqr))
        extreme <- as.integer(sum(x < q[1] - 3 * iqr | x > q[2] + 3 * iqr))
      }
    }

Grubbs on the single most extreme value.

gr <- grubbs_one(x)

Robust-z outliers: |(x - median) / (1.4826 * MAD)| > 3.5.

rz_flagged <- 0L
    if (n > 0 && !constant) {
      med <- median(x)
      mad_s <- 1.4826 * median(abs(x - med))
      if (!is.finite(mad_s) || mad_s == 0) mad_s <- sd_v
      if (is.finite(mad_s) && mad_s > 0) {
        rz_flagged <- as.integer(sum(abs((x - med) / mad_s) > 3.5))
      }
    }

    verdict <- if (constant) {
      "constant(no variation)"
    } else if (!is.na(sk) && sk > 1) {
      "right-skewed"
    } else if (!is.na(sk) && sk < -1) {
      "left-skewed"
    } else if (!is.na(ku) && ku > 2) {
      "heavy-tailed"
    } else if (!is.na(shapiro_p) && shapiro_p >= 0.05) {
      "approximately normal"
    } else if (!is.na(shapiro_p) && shapiro_p < 0.05) {
      "mildly non-normal"
    } else {
      "approximately normal"
    }

    nonparam <- (!is.na(shapiro_p) && shapiro_p < 0.05) ||
                (!is.na(sk) && abs(sk) > 1)
    recommendation <- if (constant) {
      "constant — excluded from testing"
    } else if (nonparam) {
      "consider a nonparametric method or transformation"
    } else {
      "parametric methods are appropriate"
    }

    stats[[cc]] <- list(
      name = hn, n = n, n_missing = n_missing,
      mean = mean_v, median = median_v, sd = sd_v, constant = constant,
      skew = sk, kurtosis = ku, abs_skew = if (is.na(sk)) 0 else abs(sk),
      shapiro_p = shapiro_p, sampled = sampled, shap_n = shap_n,
      mild = mild, extreme = extreme,
      grubbs_G = gr$G, grubbs_crit = gr$crit, grubbs_p = gr$p,
      grubbs_value = gr$value, grubbs_flagged = gr$flagged,
      most_extreme_value = gr$value, robust_z_flagged = rz_flagged,
      verdict = verdict, recommendation = recommendation, nonparam = nonparam
    )
  }

Step 5: Roll-up counts + single most non-normal column (NA-guarded)

constant_vec <- vapply(used_cols, function(cc) isTRUE(stats[[cc]]$constant), logical(1))
  verdict_vec <- vapply(used_cols, function(cc) stats[[cc]]$verdict, character(1))
  shapiro_vec <- vapply(used_cols, function(cc) stats[[cc]]$shapiro_p, numeric(1))
  absskew_vec <- vapply(used_cols, function(cc) stats[[cc]]$abs_skew, numeric(1))
  nonparam_vec <- vapply(used_cols, function(cc) isTRUE(stats[[cc]]$nonparam), logical(1))

  n_used <- length(used_cols)
  n_constant <- sum(constant_vec)
  n_testable <- n_used - n_constant
  n_normal <- sum(verdict_vec == "approximately normal")
  n_nonnormal <- max(0L, n_testable - n_normal)
  n_nonparam <- sum(nonparam_vec)
  total_mild <- sum(vapply(used_cols, function(cc) stats[[cc]]$mild, integer(1)))
  total_extreme <- sum(vapply(used_cols, function(cc) stats[[cc]]$extreme, integer(1)))
  n_sampled <- sum(vapply(used_cols, function(cc) isTRUE(stats[[cc]]$sampled), logical(1)))

Most non-normal = lowest Shapiro p among testable columns; NA-filter BEFORE which.min (LAT-1445 crash class). Fall back to largest |skew|.

testable_idx <- which(!constant_vec)
  most_name <- "none"; most_p <- NA_real_
  if (length(testable_idx) > 0) {
    sp_ok <- testable_idx[is.finite(shapiro_vec[testable_idx])]
    if (length(sp_ok) > 0) {
      mi <- sp_ok[which.min(shapiro_vec[sp_ok])]
    } else {
      as_ok <- testable_idx[is.finite(absskew_vec[testable_idx])]
      mi <- if (length(as_ok) > 0) as_ok[which.max(absskew_vec[as_ok])] else testable_idx[1]
    }
    most_name <- hn_vec[mi]
    most_p <- shapiro_vec[mi]
  }

  metrics <- list(
    `Columns Screened`         = as.integer(n_used),
    `Approximately Normal`     = as.integer(n_normal),
    `Non-Normal or Skewed`     = as.integer(n_nonnormal),
    `Extreme Outliers Flagged` = as.integer(total_extreme),
    `Nonparametric Recommended`= as.integer(n_nonparam),
    `Most Non-Normal Column`   = most_name
  )

  json_output <- list(
    answer = paste0(
      "Screened ", n_used, " numeric column", .s(n_used), " across ",
      format(final_rows, big.mark = ","), " row", .s(final_rows), ": ",
      n_normal, " look", if (n_normal == 1) "s" else "",
      " approximately normal and ", n_nonnormal, " ",
      if (n_nonnormal == 1) "is" else "are", " skewed or non-normal. ",
      total_extreme, " extreme outlier", .s(total_extreme),
      " were found across the columns. ",
      if (most_name != "none") paste0(
        "The single most non-normal column is ", most_name,
        " (Shapiro-Wilk ", fmt_pp(most_p), "). ") else "",
      n_nonparam, " column", .s(n_nonparam), " ",
      if (n_nonparam == 1) "is" else "are",
      " better served by a nonparametric method or a transformation."
    ),
    cards = lapply(
      c("tldr", "overview", "preprocessing", "normality_table",
        "outlier_summary", "recommendation"),
      function(cid) list(id = cid, metrics = metrics)
    )
  )

  list(
    initial_rows = initial_rows, final_rows = final_rows,
    rows_removed = rows_removed,
    col_names = col_names, used_cols = used_cols, dropped_cols = dropped_cols,
    stats = stats,
    normality_df = normality_df, outliers_df = outliers_df,
    skew_by_column_df = skew_by_column_df,
    n_used = n_used, n_constant = n_constant, n_testable = n_testable,
    n_normal = n_normal, n_nonnormal = n_nonnormal, n_nonparam = n_nonparam,
    total_mild = total_mild, total_extreme = total_extreme,
    n_sampled = n_sampled, most_name = most_name, most_p = most_p,
    metrics = metrics, json_output = json_output
  )
}

Your data has more stories to tell. Run any analysis on your own data — 60+ validated R modules, interactive reports, AI insights, and PDF export. 500 free credits on signup.

Try Free — No Signup Sign Up Free

Something wrong? Report it and we'll fix it — or get your credits back

Download PDF · Print · Share

What's wrong with this card?

Executive Summary

Analysis Overview

Data Quality

Normality by Column

Outliers by Column

Skewness by Column

Methodology

Statistical Method

Analysis Code

Normality & Outlier Screening — Is Your Data Ready?

Why This Method?

What This Analysis Covers

Standard Library

Core Analysis Pipeline

Step 1: Discover mapped columns (column_1..column_N)

Step 2: Coerce numeric (95% rule); drop non-numeric and report them.

Step 3: Per-column screen

Step 5: Roll-up counts + single most non-normal column (NA-guarded)

Report an Issue