[{"content":"Back in 2023, I wrote a post using SymPy to compute the second order expansion of the mean squared error (MSE) of the sample standard deviation up to $O(n^{-2})$. Earlier this year, I revisited the problem in MSE of sample standard deviation (Gen AI) and showed that a modern LLM can derive the entire second order expansion directly from a prompt.\nWhen an AI produces a long asymptotic derivation with dozens of polynomial moment terms, how can we trust that every coefficient is right and that no terms of order $n^{-2}$ were dropped? To check the derivation rigorously, I formalized and verified the entire proof in Lean 4 with Mathlib.\nThe complete Lean 4 formalization is a single file, SampleSdMse.lean. It builds with Lean v4.35.0 and Mathlib v4.35.0 (commit 56a0058). A complete human readable proof follows the Lean formalization step by step.\nSummary of Results Let $X_1, X_2, \\ldots$ be i.i.d. with mean $\\mu$, variance $\\sigma^2 \u0026gt; 0$, and finite moments. We compare two estimators of the population standard deviation $\\sigma$:\n$S_n = \\sqrt{\\frac{1}{n-1}\\sum _ {i=1}^n (X_i - \\bar X)^2}$ (Bessel corrected, using $n - 1$ in the denominator) $\\hat \\sigma_n = \\sqrt{\\frac{1}{n}\\sum _ {i=1}^n (X_i - \\bar X)^2} = \\sqrt{\\frac{n-1}{n}} S_n$ (unadjusted, using $n$ in the denominator) Let $\\lambda_k = \\frac{E[(X - \\mu)^k]}{\\sigma^k}$ denote the standardized central moments, with $\\lambda_3$ the skewness, $\\lambda_4$ the kurtosis, and $\\lambda_6$ the sixth moment. The general second order expansions are:\n$$ E(S_n) = \\sigma - \\frac{\\sigma}{8n} (\\lambda_4 - 1) + \\frac{\\sigma}{128n^2} \\left(8 \\lambda_6 - 15 \\lambda_4^2 + 6 \\lambda_4 - 48 \\lambda_3^2 - 31\\right) + o(n^{-2}) $$$$ E(\\hat \\sigma_n) = \\sigma - \\frac{\\sigma}{8n} (\\lambda_4 + 3) + \\frac{\\sigma}{128n^2} \\left(8 \\lambda_6 - 15 \\lambda_4^2 + 14 \\lambda_4 - 48 \\lambda_3^2 - 55\\right) + o(n^{-2}) $$$$ MSE(S_n) = \\frac{\\sigma^2}{4n} (\\lambda_4 - 1) + \\frac{\\sigma^2}{64n^2} \\left(48 \\lambda_3^2 + 15 \\lambda_4^2 - 6 \\lambda_4 - 8 \\lambda_6 + 31\\right) + o(n^{-2}) $$$$ MSE(\\hat \\sigma_n) = \\frac{\\sigma^2}{4n} (\\lambda_4 - 1) + \\frac{\\sigma^2}{64n^2} \\left(48 \\lambda_3^2 + 15 \\lambda_4^2 - 14 \\lambda_4 - 8 \\lambda_6 + 55\\right) + o(n^{-2}) $$ Taking the difference gives:\n$$ MSE(S_n) - MSE(\\hat \\sigma_n) = \\frac{\\sigma^2 (\\lambda_4 - 3)}{8n^2} + o(n^{-2}) $$ This leads to three conclusions:\nThreshold at $\\lambda_4 = 3$: When the kurtosis is $\\lambda_4 = 3$, the $1/n^2$ leading difference term vanishes: $$ MSE(S_n) - MSE(\\hat \\sigma_n) = o(n^{-2}) $$ For heavier tails ($\\lambda_4 \u0026gt; 3$), $\\hat \\sigma_n$ has smaller MSE. For lighter tails ($\\lambda_4 \u0026lt; 3$), $S_n$ has smaller MSE.\nShared expansion at $\\lambda_4 = 3$: $$ MSE(S_n) = MSE(\\hat \\sigma_n) + o(n^{-2}) = \\frac{\\sigma^2}{2n} + \\frac{\\sigma^2}{16n^2}\\left(12\\lambda_3^2 - 2\\lambda_6 + 37\\right) + o(n^{-2}) $$ with expected value: $$ E(S_n) = \\sigma - \\frac{\\sigma}{4n} + \\frac{\\sigma}{32n^2}\\left(2\\lambda_6 - 12\\lambda_3^2 - 37\\right) + o(n^{-2}) $$ Normal distribution special case ($\\lambda_3 = 0, \\lambda_4 = 3, \\lambda_6 = 15$): $$ E(S_n) = \\sigma - \\frac{\\sigma}{4n} - \\frac{7\\sigma}{32n^2} + o(n^{-2}) $$ $$ MSE(S_n) = MSE(\\hat \\sigma_n) + o(n^{-2}) = \\frac{\\sigma^2}{2n} + \\frac{7\\sigma^2}{16n^2} + o(n^{-2}) $$ Formalization in Lean 4 with Mathlib Assumptions. The only assumptions are the textbook i.i.d. model, stated with Mathlib\u0026rsquo;s standard notions of independence (iIndepFun), identical distribution (IdentDistrib), and variance:\nclass SampleModel (Ω : outParam Type*) [MeasurableSpace Ω] (μ : outParam (Measure Ω)) [IsProbabilityMeasure μ] where X : ℕ → Ω → ℝ measurable : ∀ i, Measurable (X i) indep : ProbabilityTheory.iIndepFun X μ ident : ∀ i, ProbabilityTheory.IdentDistrib (X i) (X 0) μ μ moments : ∀ k : ℕ, Integrable (fun ω =\u0026gt; X 0 ω ^ k) μ var_pos : 0 \u0026lt; ProbabilityTheory.variance (X 0) μ In words: on a probability space $(\\Omega, \\mu)$, the observations $X_0, X_1, \\ldots$ are measurable, mutually independent, identically distributed, have all moments finite, and have positive variance. The proof only uses moments up to order 16; assuming all moments are finite keeps the formalization simple.\nDefinitions. Everything in the main theorem is a plain textbook definition. Expectation is Mathlib\u0026rsquo;s Bochner integral:\ndef mean : ℝ := ∫ ω, SampleModel.X 0 ω ∂μ def σ : ℝ := Real.sqrt (ProbabilityTheory.variance (SampleModel.X 0) μ) def lam (k : ℕ) : ℝ := ∫ ω, ((SampleModel.X 0 ω - mean (μ := μ)) / σ (μ := μ)) ^ k ∂μ def Xbar (n : ℕ) (ω : Ω) : ℝ := (1 / (n : ℝ)) * ∑ i : Fin n, SampleModel.X i ω def sampleSD (n : ℕ) (ω : Ω) : ℝ := Real.sqrt ((1 / (n : ℝ)) * ∑ i : Fin n, (SampleModel.X i ω - Xbar (μ := μ) n ω) ^ 2) def besselSD (n : ℕ) (ω : Ω) : ℝ := Real.sqrt ((1 / ((n : ℝ) - 1)) * ∑ i : Fin n, (SampleModel.X i ω - Xbar (μ := μ) n ω) ^ 2) def expect [SampleModel Ω μ] (T : Ω → ℝ) : ℝ := ∫ ω, T ω ∂μ def mse [SampleModel Ω μ] (T : Ω → ℝ) (θ : ℝ) : ℝ := ∫ ω, (T ω - θ) ^ 2 ∂μ Here lam k is $\\lambda_k$, sampleSD is $\\hat \\sigma_n$, and besselSD is $S_n$. Asymptotic statements use Mathlib\u0026rsquo;s standard little $o$ notation along atTop for integer sample sizes. For example, the MSE difference is the following part of the main theorem textbook_certificate, which assumes nothing beyond SampleModel:\n(fun n : ℕ =\u0026gt; (mse (besselSD n) σ - mse (sampleSD n) σ) - σ ^ 2 * (lam4 - 3) / (8 * (n : ℝ) ^ 2)) =o[atTop] (fun n : ℕ =\u0026gt; 1 / (n : ℝ) ^ 2) Proof outline. Lean checks every step of the following argument:\nStandardize $Z_i = (X_i - \\mu)/\\sigma$ and write $\\hat \\sigma_n = \\sigma \\sqrt{1 + D_n}$ and $S_n = \\sigma \\sqrt{1 + \\Delta_n}$, where $D_n = U_n - V_n^2$ with $U_n = \\frac{1}{n}\\sum (Z_i^2 - 1)$ and $V_n = \\bar Z$, and $1 + \\Delta_n = \\frac{n}{n-1}(1 + D_n)$. Compute the exact finite sample moments of $U_n$ and $V_n$ from independence, including the exact $1/n^3$ correction in the fourth moments. Expand $\\sqrt{1 + x}$ to fourth order with an exact remainder, and bound the expected remainder using $|\\sqrt{1 + x} - 1| \\le |x|$ and the Cauchy Schwarz inequality. Prove the moment growth bound $E[T_n^m] = O(n^{m/2})$ for partial sums $T_n = \\sum _ {i=1}^n W_i$ of i.i.d. mean zero $W_i$, from the recursion $E[T_{n+1}^m] = \\sum_j \\binom{m}{j} E[T_n^j] E[W^{m-j}]$, which holds because $T_n$ is independent of $W_{n+1}$. This gives every higher order remainder as $o(n^{-2})$ without any extra rate assumption. Kernel Axiom Audit Running lake build SdMseLean compiles the library with no sorry, and #print axioms shows that every theorem depends only on the standard Lean 4 axioms:\ninfo: SdMseLean/SampleSdMse.lean:2511:0: \u0026#39;SdMseLean.SampleModel.textbook_certificate\u0026#39; depends on axioms: [propext, Classical.choice, Quot.sound] info: SdMseLean/SampleSdMse.lean:2512:0: \u0026#39;SdMseLean.SampleModel.remainder_order\u0026#39; depends on axioms: [propext, Classical.choice, Quot.sound] info: SdMseLean/SampleSdMse.lean:2514:0: \u0026#39;SdMseLean.SampleModel.taylorSqrt4\u0026#39; depends on axioms: [propext, Classical.choice, Quot.sound] info: SdMseLean/SampleSdMse.lean:2518:0: \u0026#39;SdMseLean.sampleAvg_moment1\u0026#39; depends on axioms: [propext, Classical.choice, Quot.sound] info: SdMseLean/SampleSdMse.lean:2519:0: \u0026#39;SdMseLean.sampleAvg_moment2\u0026#39; depends on axioms: [propext, Classical.choice, Quot.sound] info: SdMseLean/SampleSdMse.lean:2520:0: \u0026#39;SdMseLean.sampleAvg_moment3\u0026#39; depends on axioms: [propext, Classical.choice, Quot.sound] info: SdMseLean/SampleSdMse.lean:2521:0: \u0026#39;SdMseLean.sampleAvg_moment4\u0026#39; depends on axioms: [propext, Classical.choice, Quot.sound] Build completed successfully (3047 jobs). Lean\u0026rsquo;s kernel checks every coefficient in the expansions above, under exactly the assumptions listed in SampleModel. What remains a human judgment is whether those Lean statements say what we mean, which is why the assumptions and definitions are kept as close to the textbook as possible.\n","permalink":"https://randylai.info/post/2026/mse_of_sample_sd_lean/","summary":"\u003cp\u003eBack in 2023, I wrote a \u003ca href=\"/post/2023/mse_of_sample_sd_cont/\"\u003epost\u003c/a\u003e using SymPy to compute the second order expansion of the mean squared error (MSE) of the sample standard deviation up to $O(n^{-2})$. Earlier this year, I revisited the problem in \u003ca href=\"/post/2026/mse_of_sample_sd_gen_ai/\"\u003eMSE of sample standard deviation (Gen AI)\u003c/a\u003e and showed that a modern LLM can derive the entire second order expansion directly from a prompt.\u003c/p\u003e\n\u003cp\u003eWhen an AI produces a long asymptotic derivation with dozens of polynomial moment terms, how can we trust that every coefficient is right and that no terms of order $n^{-2}$ were dropped? To check the derivation rigorously, I formalized and verified the entire proof in \u003ca href=\"https://lean-lang.org/\"\u003eLean 4\u003c/a\u003e with \u003ca href=\"https://github.com/leanprover-community/mathlib4\"\u003eMathlib\u003c/a\u003e.\u003c/p\u003e","title":"MSE of sample standard deviation (Lean by AI)"},{"content":"In the era of gen AI, I am trying to use LLM to compute the second order expansion of $MSE(S)$ that I did in 2023.\nThe standard sample s.d. is given by S=sqrt(\\sum_{i=1}^n(x_i - \\bar x)^2/ (n-1)). Without assumption the underlying distribution, compute the MSE(S) of it in terms of the first few moments of x_i up. Expand the terms up to 1/n^2.\nAnd it just nails it: https://gemini.google.com/share/fa35ab43436f\n","permalink":"https://randylai.info/post/2026/mse_of_sample_sd_gen_ai/","summary":"\u003cp\u003eIn the era of gen AI, I am trying to use LLM to compute the second order expansion of \u003ca href=\"https://randylai.info/post/2023/mse_of_sample_sd_cont/\"\u003e$MSE(S)$\u003c/a\u003e that I did in 2023.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eThe standard sample s.d. is given by  S=sqrt(\\sum_{i=1}^n(x_i - \\bar x)^2/ (n-1)). Without assumption the underlying distribution, compute the MSE(S) of it in terms of the first few moments of x_i up. Expand the terms up to 1/n^2.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eAnd it just nails it: \u003ca href=\"https://gemini.google.com/share/fa35ab43436f\"\u003ehttps://gemini.google.com/share/fa35ab43436f\u003c/a\u003e\u003c/p\u003e","title":"MSE of sample standard deviation (Gen AI)"},{"content":"Continue our previous post of computing the MSE of sample standard deviation. We want to have higher order expansion of $E(S_n)$. The algebric computation is a bit tedious, we will use sympy to help us.\nOpen notebook in full screen.\nThe result is quite interesting. For normal distribution, the MSE of $S_n$ and $\\hat \\sigma$ are the same up to the order $1/n^2$. If the distribution has a thicker tail than normal, i.e. kurtosis \u0026gt; 3, then $\\hat \\sigma_n$ will have a smaller MSE than $S_n$. On the other hand, if the distribution has a lighter tail than normal, than $S_n$ will have a smaller MSE.\n","permalink":"https://randylai.info/post/2023/mse_of_sample_sd_cont/","summary":"\u003cp\u003eContinue our \u003ca href=\"https://randylai.info/post/2023/mse_of_sample_sd/\"\u003eprevious\u003c/a\u003e post of computing the MSE of sample standard deviation. We want to have higher order expansion of $E(S_n)$. The algebric computation is a bit tedious, we will use \u003ca href=\"https://www.sympy.org/\"\u003esympy\u003c/a\u003e to help us.\u003c/p\u003e\n\u003cp\u003e\u003cfont style=\"font-size: small\"\u003e\u003ca href=\"/post/2023/sample_sd_bias.html\" target=\"_blank\"\u003eOpen notebook in full screen.\u003c/a\u003e\u003c/font\u003e\u003c/p\u003e\n \u003ciframe\n       src=\"/post/2023/sample_sd_bias.html\"\n       width=\"100%\"\n       height=\"1000px\"\n       style=\"border:none;\"\u003e\n \u003c/iframe\u003e\n\u003cp\u003eThe result is quite interesting. For normal distribution, the MSE of $S_n$ and $\\hat \\sigma$ are the same up to the order $1/n^2$.\nIf the distribution has a thicker tail than normal, i.e. kurtosis \u0026gt; 3, then $\\hat \\sigma_n$ will have a smaller MSE than $S_n$. On the other hand, if the distribution has a lighter tail than normal, than $S_n$ will have a smaller MSE.\u003c/p\u003e","title":"MSE of sample standard deviation (cont')"},{"content":"In a previous post, we have discussed the MSE of sample variance. One could argue that we should compute the MSE of the sample standard deviation as sample s.d. has the same unit as the measurements.\nLet $S_n$ be the sample s.d. and $\\sigma$ be the true s.d., the goal is to compute\n$$ MSE(S_n) = E(S_n-\\sigma)^2 = E(S_n^2) - 2 \\sigma E(S_n) + \\sigma^2 = 2 \\sigma^2 - 2 \\sigma E(S_n). $$ To compute $E(S_n)$, first, we consider Taylor\u0026rsquo;s expanding $g(x) = \\sqrt{x}$ about $x=\\sigma^2$, we have $$ g(x) = \\sigma + \\frac{1}{2 \\sigma}(x-\\sigma^2) - \\frac{1}{8 \\sigma^3}(x-\\sigma^2)^2 + R(x), $$ where $R(x) =- \\left(\\frac{1}{8 \\tilde \\sigma^3} - \\frac{1}{8 \\sigma^3}\\right)(x-\\sigma^2)^2$ for some $\\tilde \\sigma$ between $\\sqrt{x}$ and $\\sigma$.\nLet $\\kappa = E(X - \\mu)^4 / \\sigma^4$ be the kurtosis. It could be shown that $E\\left[\\sqrt{n}(S_n^2 - \\sigma^2)\\right]^2 \\rightarrow \\sigma^4(\\kappa-1)$ and $n ER(S_n^2) \\rightarrow 0$ (and the proofs are beyond the discussion of this post).\nThus, $$ E(S_n) = Eg(S_n^2) =\\sigma + \\frac{1}{2 \\sigma} E(S_n^2 - \\sigma^2) - \\frac{1}{8\\sigma^3} E(S_n^2 - \\sigma^2)^2 + o(n^{-1}). $$ $$ = \\sigma - \\frac{\\sigma}{8}\\left(\\frac{\\kappa - 1}{n}\\right) + o(n^{-1}). $$ Therefore, $MSE(S_n) = \\frac{\\sigma^2}{4}\\left( \\frac{\\kappa - 1}{n}\\right) + o(n^{-1})$.\nSimilar calculation could be done for $\\hat \\sigma_n = \\sqrt{\\frac{\\sum(X_i - \\bar X)^2}{n}}$,\n$$ \\begin{align*} MSE(\\hat \\sigma_n) \u0026amp; = E\\left[ \\sqrt{\\frac{n-1}{n}} S_n - \\sigma \\right]^2 = E\\left[\\left(1 - \\frac{1}{2n} + o(n^{-1})\\right) S_n - \\sigma\\right]^2 \\\\ \u0026amp;= \\frac{\\sigma^2}{4}\\left( \\frac{\\kappa - 1}{n}\\right) + o(n^{-1}). \\end{align*} $$ Unfortunately, MSE of $S_n$ and $\\hat \\sigma_n$ match at the first order. We will need to consider higher order expansion next time.\n","permalink":"https://randylai.info/post/2023/mse_of_sample_sd/","summary":"\u003cp\u003eIn a \u003ca href=\"https://randylai.info/post/2020/mse_of_sample_variance/\"\u003eprevious\u003c/a\u003e post, we have discussed the MSE of sample variance. One could argue that we should compute the MSE of the sample standard deviation as sample s.d. has the same unit as the measurements.\u003c/p\u003e\n\u003cp\u003eLet $S_n$ be the sample s.d. and $\\sigma$ be the true s.d., the goal is to compute\u003c/p\u003e\n$$\nMSE(S_n) = E(S_n-\\sigma)^2 = E(S_n^2) - 2 \\sigma E(S_n) + \\sigma^2 = 2 \\sigma^2 -  2 \\sigma E(S_n).\n$$\u003cp\u003eTo compute $E(S_n)$, first, we consider Taylor\u0026rsquo;s expanding $g(x) = \\sqrt{x}$ about $x=\\sigma^2$,\nwe have\n\u003c/p\u003e","title":"MSE of sample standard deviation"},{"content":"In introductory statistics courses, we often focus on unbiased estimators such as the sample variance, $$ S^2 = \\frac{\\sum(X_i - \\bar X)^2}{n - 1} $$ rather than a more initiative estimator $$ \\hat \\sigma^2 = \\frac{\\sum(X_i - \\bar X)^2}{n}. $$ The common argument is that $S^2$ is unbiased whereas $\\hat \\sigma^2$ is biased for estimating $\\sigma^2$. While it is true, this statement is a bit over-stressed. A more statistical way to evaluate performance of estimators is to compare the mean squared error (MSE) of the estimators.\nNormal Population If the population is normal, using the fact that $\\sum(X_i - \\bar X)^2 / \\sigma^2 \\sim \\chi^2_{n-1}$, one could conclude that $Var[\\sum(X_i - \\bar X)^2] = 2(n-1) \\sigma^4$ thus $$ MSE(S^2) = \\frac{2 \\sigma^4}{n-1}. $$ and $$ MSE(\\hat \\sigma^2) = Var(\\hat \\sigma^2) + Bias(\\hat \\sigma^2)^2 = \\frac{2 (n-1) \\sigma^4}{n^2} + \\left[\\frac{-\\sigma^2}{n}\\right]^2 = \\frac{(2n-1) \\sigma^4}{n^2}. $$ As $MSE(\\hat \\sigma^2) \u0026lt; MSE(S^2)$, we conclude that $\\hat \\sigma^2$ is a better than estimator than $S^2$ for normal data.\nHow about when the population is not normal? There are a few online resources that have already covered this case.\nhttps://davegiles.blogspot.com/2013/05/whats-variance-of-sample-variance.html https://math.stackexchange.com/a/2476662/136006 https://mathworld.wolfram.com/SampleVarianceDistribution.html We need $Var\\left[\\sum(X_i - \\bar X)^2\\right]$ to compute $MSE(S^2)$ and $MSE(\\hat \\sigma^2)$ and $$ Var\\left[\\sum(X_i - \\bar X)^2\\right] = E\\left[\\left(\\sum(X_i - \\bar X)^2\\right)^2\\right] - (n-1)^2\\sigma^4. $$ Let $Z_i = X_i - \\mu$, $\\mu = E(X_i)$, $\\sigma^2 = Var(X_i)$ and the kurtosis $\\kappa = E Z_i^4/\\sigma^4$,\n$$ \\begin{align*} E\\left[\\left(\\sum(X_i - \\bar X)^2\\right)^2\\right] \u0026amp;= E\\left[\\left( \\sum (X_i-\\mu)^2 - n (\\bar X - \\mu)^2 \\right)^2\\right] \\\\ \u0026amp;= E\\left[\\left( \\sum Z_i^2 - n (\\bar Z)^2 \\right)^2\\right] \\\\ \u0026amp;= E\\left[(\\sum Z_i^2)^2\\right] - \\frac{2}{n} E \\left[ (\\sum Z_i^2) (\\sum Z_i)^2\\right] + \\frac{1}{n^2} E\\left[(\\sum Z_i)^4 \\right]. \\end{align*} $$ The first term\n$$ E\\left[(\\sum Z_i^2)^2\\right] = E\\left[\\sum Z_i^4 + \\sum_{i\\ne j} Z_i^2 Z_j^2 \\right] = n \\kappa \\sigma^4 + n(n-1) \\sigma^4. $$ The second term,\n$$ \\begin{align*} E \\left[ (\\sum Z_i^2) (\\sum Z_i)^2\\right] \u0026amp;= E \\left[ (\\sum Z_i^2) (\\sum Z_i^2 + \\sum_{i\\ne j} Z_i Z _j)\\right] \\\\ \u0026amp;=E \\left[\\sum Z_i^4 + \\sum_{i\\ne j} Z_i^2 Z_j^2 + \\sum_k \\sum_{i\\ne j} Z_k^2 Z_i Z_j \\right] \\\\ \u0026amp;= n \\kappa \\sigma^4 + n(n-1) \\sigma^4. \\end{align*} $$ The third term,\n$$ \\begin{align*} E\\left[(\\sum Z_i)^4 \\right] \u0026amp;= E[\\sum Z_i^4 + 4 \\sum_{i \\ne j} Z_i^3 Z_j + 3 \\sum_{i \\ne j} Z_i^2 Z_j^2 + 6 \\sum_{i \\ne j \\ne k} Z_i^2 Z_j Z_k + \\sum_{i \\ne j \\ne k \\ne l} Z_i Z_j Z_k Z_l ] \\\\ \u0026amp;= E[\\sum Z_i^4 + 3 \\sum_{i \\ne j} Z_i^2 Z_j^2] \\\\ \u0026amp;= n \\kappa \\sigma^4 + 3n(n-1) \\sigma^4. \\end{align*} $$ Therefore,\n$$ \\begin{align*} E\\left[\\left(\\sum(X_i - \\bar X)^2\\right)^2\\right] \u0026amp;= \\frac{\\sigma^4}{n} \\left[(n^2 - 2n + 1) \\kappa + n^2(n-1) - 2 n(n-1) + 3(n-1) \\right] \\\\ \u0026amp;= \\frac{(n-1)\\sigma^4}{n} \\left[(n-1) \\kappa + n^2 - 2n + 3)\\right] \\end{align*} $$ and $$ \\begin{align*} Var\\left[\\left(\\sum(X_i - \\bar X)^2\\right)^2\\right] \u0026amp;= \\frac{(n-1)\\sigma^4}{n} \\left[(n-1) \\kappa + n^2 - 2n + 3)\\right] - (n-1)^2 \\sigma^4 \\\\ \u0026amp;= \\frac{(n-1)\\sigma^4}{n} \\left[(n-1) \\kappa - n + 3 \\right]. \\end{align*} $$ Finally,\n$$ MSE(S^2) = \\frac{\\sigma^4}{n(n-1)} \\left[(n-1) \\kappa - n + 3 \\right] = \\sigma^4\\left[\\frac{\\kappa}{n} - \\frac{n-3}{n(n - 1)}\\right] $$ and $$ MSE(\\hat \\sigma^2) = \\frac{(n-1)\\sigma^4}{n^3} \\left[(n-1) \\kappa - n + 3 \\right] + \\left[\\frac{-\\sigma^2}{n}\\right]^2 = \\sigma^4\\left[ \\frac{(n-1)^2}{n^3} \\kappa - \\frac{n^2-5n + 3}{n^3}\\right]. $$ Some algebra shows that $MSE(\\hat \\sigma^2) \u0026lt; MSE(S^2)$ for all $n$ when $\\kappa \u0026gt; 1.5$. Most common distributions have kurtosises greater than 1.5. A noticeable exception is the Bernoulli distribution with $1/3 \u0026lt; p \u0026lt; 2/3$. Nevertheless, in practice, we don\u0026rsquo;t directly estimate the variance of the Bernoulli distribution but to estimate $p$ directly.\nConclusion In terms of MSE, $\\hat \\sigma^2$ is almost always better than $S$. But wait, one may ask why we should use MSE as a loss function for $\\sigma^2$. We shall explore other options later.\n","permalink":"https://randylai.info/post/2020/mse_of_sample_variance/","summary":"\u003cp\u003eIn introductory statistics courses, we often focus on unbiased estimators such as the sample variance,\n\u003c/p\u003e\n$$\nS^2 = \\frac{\\sum(X_i - \\bar X)^2}{n - 1}\n$$\u003cp\u003e\nrather than a more initiative estimator\n\u003c/p\u003e\n$$\n\\hat \\sigma^2 = \\frac{\\sum(X_i - \\bar X)^2}{n}.\n$$\u003cp\u003eThe common argument is that $S^2$ is unbiased whereas $\\hat \\sigma^2$ is biased for estimating $\\sigma^2$. While it is true, this statement is a bit over-stressed. A more statistical way to evaluate performance of estimators is to compare the mean squared error (MSE) of the estimators.\u003c/p\u003e","title":"MSE of sample variance"},{"content":"It is well known that $\\bar X^2$ is a biased estimator for $\\mu^2$, but how about its variance?\nBefore we start working on $Var(\\bar X^2)$, let\u0026rsquo;s recall that $\\bar X$ is an unbiased estimator for $\\mu$, i.e. $E(\\bar X) = \\mu$. Moreover, $Var(\\bar X) = \\sigma^2/n$ and $E(\\bar X^2) = Var(\\bar X) + E(\\bar X)^2 = \\frac{\\sigma^2}{n} + \\mu^2$.\nLet $Z_i = X_i - \\mu$,\n$$ \\begin{align*} E\\left[(\\sum X_i)^4\\right] \u0026amp;= E\\left[(n \\mu + \\sum Z_i)^4\\right]\\\\ \u0026amp;= E\\left[[n^4 \\mu^4 + 4 n^3 \\mu^3 \\sum Z_i+ 6 n^2 \\mu^2 (\\sum Z_i)^2 + 4 n \\mu (\\sum Z_i)^3 + (\\sum Z_i)^4 \\right]. \\end{align*} $$ By the fact that,\n$$ \\begin{align*} E\\left[\\sum Z_i\\right] \u0026amp;= 0\\\\ E\\left[(\\sum Z_i)^2\\right] \u0026amp;= E(\\sum Z_i^2 + \\sum_{i\\ne j} Z_i Z_j) = E(\\sum Z_i^2) = n\\sigma^2\\\\ E\\left[(\\sum Z_i)^3\\right] \u0026amp;= E(\\sum Z_i^3 + 3 \\sum_{i\\ne j} Z_i^2 Z_j + \\sum_{i\\ne j \\ne k} Z_i Z_j Z_k) = n\\gamma \\sigma^3\\\\ E\\left[(\\sum Z_i)^4 \\right] \u0026amp;= E[\\sum Z_i^4 + 4 \\sum_{i \\ne j} Z_i^3 Z_j + 3 \\sum_{i \\ne j} Z_i^2 Z_j^2 + 6 \\sum_{i \\ne j \\ne k} Z_i^2 Z_j Z_k + \\sum_{i \\ne j \\ne k \\ne l} Z_i Z_j Z_k Z_l ] \\\\ \u0026amp;= E[\\sum Z_i^4 + 3 \\sum_{i \\ne j} Z_i^2 Z_j^2] \\\\ \u0026amp;= n \\kappa \\sigma^4 + 3n(n-1) \\sigma^4. \\end{align*} $$ We have, $$ E\\left[(\\sum X_i)^4\\right] = n^4 \\mu^4 + 6n^3 \\sigma^2 \\mu^2 + 4n^2 \\gamma \\sigma^2 \\mu + n(3n + \\kappa - 3) \\sigma^4. $$$$ Var[\\bar X^2] = \\frac{E[(\\sum X_i)^4]}{n^4} - [E(\\bar X^2)]^2 = \\frac{4}{n} \\sigma^2 \\mu^2 + \\frac{4\\gamma}{n^2} \\sigma^3 \\mu + \\frac{(2n+\\kappa-3)}{n^3} \\sigma^4. $$","permalink":"https://randylai.info/post/2020/variance_of_x_bar_squared/","summary":"\u003cp\u003eIt is well known that $\\bar X^2$ is a biased estimator for $\\mu^2$, but how about its variance?\u003c/p\u003e\n\u003cp\u003eBefore we start working on $Var(\\bar X^2)$, let\u0026rsquo;s recall that $\\bar X$ is an unbiased estimator for $\\mu$, i.e. $E(\\bar X) = \\mu$. Moreover, $Var(\\bar X) = \\sigma^2/n$ and $E(\\bar X^2) = Var(\\bar X) + E(\\bar X)^2 = \\frac{\\sigma^2}{n} + \\mu^2$.\u003c/p\u003e\n\u003cp\u003eLet $Z_i = X_i - \\mu$,\u003c/p\u003e\n$$\n\\begin{align*}\nE\\left[(\\sum X_i)^4\\right] \u0026amp;= E\\left[(n \\mu + \\sum Z_i)^4\\right]\\\\\n\u0026amp;= E\\left[[n^4 \\mu^4 + 4 n^3 \\mu^3 \\sum Z_i+ 6 n^2 \\mu^2 (\\sum Z_i)^2 + 4 n \\mu (\\sum Z_i)^3  + (\\sum Z_i)^4 \\right].\n\\end{align*}\n$$\u003cp\u003eBy the fact that,\u003c/p\u003e","title":"Variance of sample mean squared"},{"content":"https://stats.stackexchange.com/questions/93830/expected-number-of-ratio-of-girls-vs-boys-birth\nThe Zorganian Republic has some very strange customs. Couples only wish to have female children as only females can inherit the family\u0026rsquo;s wealth, so if they have a male child they keep having more children until they have a girl. If they have a girl, they stop having children. What is the ratio of girls to boys in Zorgania?\nLet $X$ be the number of children in a family, it is geometric random variable with $p=0.5$, i.e., $$ P(X = x) = 0.5^x, x=1,2,3... $$ which implies $E(X) = 2$\nSuppose there are $N$ families in the country, the girl ratio is $$ \\frac{N}{ \\sum X_i} $$ Since $\\sum X_i /N \\rightarrow E(X) = 2$ (law of large number), the ratio coverages to 1/2 if $N \\rightarrow \\infty$.\nIf there are only finite families, let $T$ be the total number of children of the country: $T = \\sum X_i$, then $T$ has a negative binomial distribution with pmf\n$$ P(T=t) = C^{t-1}_{N-1} 0.5^t, t = N, N+1... $$ It implies $$ E\\left[ \\frac{N}{\\sum X_i} \\right] = E\\left[ \\frac{N}{T} \\right] = \\sum_{t=N}^{\\infty} \\frac{N}{t} C^{t-1}_{N-1} 0.5^t = {_2F_1} (N, 1, N+1, -1) $$ where $_2F_1$ is the hypergeometric function.\nTherefore the expected girl ratio is ${_2F_1} (N, 1, N+1, -1)$.\n","permalink":"https://randylai.info/post/2017/girl-ratio/","summary":"\u003cp\u003e\u003ca href=\"https://stats.stackexchange.com/questions/93830/expected-number-of-ratio-of-girls-vs-boys-birth\"\u003ehttps://stats.stackexchange.com/questions/93830/expected-number-of-ratio-of-girls-vs-boys-birth\u003c/a\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eThe Zorganian Republic has some very strange customs. Couples only wish to have female children as only females can inherit the family\u0026rsquo;s wealth, so if they have a male child they keep having more children until they have a girl. If they have a girl, they stop having children. What is the ratio of girls to boys in Zorgania?\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eLet $X$ be the number of children in a family, it is geometric random variable with $p=0.5$, i.e.,\n\u003c/p\u003e","title":"Girl ratio"},{"content":"Just discover two cool things about dice, Suppose we have two (fair) 6-sided dice, let $Y$ be the reminder of the product of the two numbers divided by 7, i.e., $Y = X1 \\times X2 (\\text{ mod } 7)$, guess what the distribution of $Y$ is? Uniform 1,\u0026hellip;, 6\nIf it doesn\u0026rsquo;t surprise you, now consider two n-sided dice and repeat the above with $Y = X_1 \\times X_2 (\\text{ mod } n+1)$. Guess what values of $n$ will yield a uniform distribution? It turns out that the values of $n$ are prime numbers minus 1. (As 6 = 7-1 and 7 is a prime number, a regular 6-sided dice has this property).\n","permalink":"https://randylai.info/post/2016/dice/","summary":"\u003cp\u003eJust discover two cool things about dice,\nSuppose we have two (fair) 6-sided dice, let $Y$ be the reminder of the product of the two numbers divided by 7, i.e., $Y = X1 \\times X2 (\\text{ mod } 7)$, guess what the distribution of $Y$ is? Uniform 1,\u0026hellip;, 6\u003c/p\u003e\n\u003cp\u003eIf it doesn\u0026rsquo;t surprise you, now consider two n-sided dice and repeat the above with $Y = X_1 \\times X_2 (\\text{ mod } n+1)$. Guess what values of $n$ will yield a uniform distribution? It turns out that the values of $n$ are prime numbers minus 1. (As 6 = 7-1 and 7 is a prime number, a regular 6-sided dice has this property).\u003c/p\u003e","title":"Two cool things about dice!"}]