Maximum Likelihood Estimation

As before, we begin with a sample of $X=(X_1,...,X_n)$ random variables chosen according to one of a family of probabilities $P_\theta$ . In addition, $f(x|\theta )$ , $x=(x_1,...,x_n)$ will be used to denote the density function for the data when $\theta$ is the true state of nature.

Definition 1. The likelihood function is the density function regarded as a function of $\theta$

The MLE is, $\hat{\theta } (x)=arg\max_{\theta }L(\theta |x)$ .

Note that if $\hat{\theta } (x)$ is a maximum likelihood estimator for $\theta$ , then $g(\hat{\theta } (x))$ is a maximum likelihood estimator for $g(\theta )$ . For example, if $\theta$ is a parameter for the variance and $\hat{\theta }$ is the maximum likelihood estimator, then $\sqrt{\hat{\theta } }$ is the maximum likelihood estimator for the standard deviation. This flexibility in the estimation criterion seen here is not available in the case of unbiased estimators. Typically, maximizing the score function $lnL(\theta |x)$ will be easier.

Bernoulli Trials. If the experiment consists of n Bernoulli trial with a success probability $\theta$ , then

$L(\theta |x)=\theta ^{x_1}(1-\theta )^{(1-x_1)}...\theta ^{x_n}(1-\theta )^{(1-x_n)}=\theta ^{(x_1+...+x_n)}(1-\theta )^{n-(x_1+...+x_n)}$ ,

$lnL(\theta |x)=ln\theta (\sum_{i=1}^n x_i)+ln(1-\theta )(n-\sum_{i=1}^n x_i )=n\tilde{x} ln\theta +n(1-\tilde{x} )ln(1-\theta )$ ,

$\frac{\partial}{\partial \theta } lnL(\theta |x)=n(\frac{\tilde{x} }{\theta } -\frac{1-\tilde{x} }{1-\theta } )$ .

This equals zero when $\theta =\tilde{x}$ . Check that this is a maximum. Thus, $\hat{\theta } (x)=\tilde{x}$ .

Normal Data. Maximum likelihood estimation can be applied to a vector-valued parameter. For a simple random sample of n normal random variables,

$L(\mu , \sigma ^2|x)=(\frac{1}{\sqrt{2\pi \sigma ^2} } exp\left\{ \frac{-{(x_1-\mu )}^2}{2\sigma ^2} \right\} )...(\frac{1}{\sqrt{2\pi \sigma ^2} } exp\left\{ \frac{-{(x_n-\mu )}^2}{2\sigma ^2} \right\})=\frac{1}{(\sqrt{2\pi \sigma ^2}) ^n} exp {-\frac{1}{2\sigma ^2} \sum_{i=1}^n (x_i-\mu )^2 }.$

$lnL(\mu ,\sigma ^2|x)=-\frac{n}{2} 2\pi \sigma ^2 - \frac{1}{2\sigma ^2} \sum_{i=1}^n (x_i-\mu )^2$ .

$\frac{\partial}{\partial \mu } lnL(\mu ,\sigma ^2|x)=\frac{1}{\sigma ^2} \sum_{i=1}^n (x_i-\mu ) =\frac{1}{\sigma ^2} n(\tilde{x} -\mu )$ .

Because the second partial derivative with respect to $\mu$ is negative, $\hat{\mu } (x)=\tilde{x}$ is the maximum likelihood estimator.

$\frac{\partial}{\partial \sigma ^2} lnL(\mu ,\sigma ^2|x)=-\frac{n}{\sigma ^2} +\frac{1}{(\sigma ^2)^2} \sum_{i=1}^n {(x_i-\mu )}^2 =\frac{n}{(\sigma ^2)^2} (\sigma ^2-\frac{1}{n}\sum_{i=1}^n (x_i-\mu )^2 )$ .

Recalling that $\hat{\mu } (x)=\tilde{x}$ , we obtain $\hat{\sigma } ^2(x)=\frac{1}{n} \sum_{i=1}^n (x_i-\hat{x} )^2$ . Note that the MLE is a biased estimator.

Linear Regression. Our data is n observations with one explanatory variable and one response variable. The model is that $y_i=\alpha + \beta x_i+\epsilon _i$ , where the $\epsilon _i$ are independent mean 0 normal random variable. The unknown variance is $\sigma ^2$ . The likelihood function

$L(\alpha ,\beta ,\sigma ^2|y,x)=\frac{1}{\sqrt{(2\pi \sigma ^2)^n} } exp{-\frac{1}{2\sigma ^2}} \sum_{i=1}^n (y_i-(\alpha +\beta x_i))^2$ .

$lnL(\alpha ,\beta ,\sigma ^2|y,x)=-\frac{n}{2} ln2\pi \sigma ^2-\frac{1}{2\sigma ^2} \sum_{i=1}^n (y_i-(\alpha +\beta x_i))^2$ .

This is the maximum likelihood estimators $\hat{\alpha }$ and $\hat{\beta }$ also the least square estimator. The predicted value for the response variable $\hat{y}_i = \hat{\alpha } +\hat{\beta } x_i$ . The MLE for variance is $\hat{\sigma } ^2_{MLE}=\frac{1}{n} \sum_{k=1}^n (y_i-\hat{y} _i)^2$ . The unbiased estimator is $\hat{\sigma } ^2_U=\frac{1}{n-2} \sum_{k=1}^n (y_i-\hat{y} _i)^2$ .

Asymptotic Properties

Much of the attraction of maximum likelihood estimators is based on their properties for a large sample size.

Consistency. If $\theta _0$ is the state of nature, then $L(\theta _0|X) > L(\theta |X)$ , if and only if

$\frac{1}{n} \sum_{i=1}^n ln\frac{f(X_i|\theta _0)}{f(X_i|\theta)} >0$ .

By the strong law of large numbers, this sum converges to $E_{\theta _0}[ln\frac{f(X_1|\theta _0)}{f(X_1|\theta )} ]$ , which is greater than 0. From this, we obtain $\hat{\theta } (X) \rightarrow \theta _0$ as $n \rightarrow ∞$ . We call this property of the estimator consistency.

Asymptotic Normality and Efficiency. Under some assumptions that are meant to insure some regularity, a central limit theorem holds. Here we have $\sqrt{n} (\hat{\theta }(X)-\theta _0 )$ converged in distribution as $n \rightarrow ∞$ to a normal random variable with mean 0 and variance $\frac{1}{I(\theta _0)}$ , the Fisher information for one observation. Thus

$Var_{\theta _0}(\hat{\theta } (X))\approx \frac{1}{nI(\theta _0)}$ ,

the lowest possible under the Cramer-Rao lower bound. This property is called asymptotic efficiency.

Properties of the log-likelihood surface. For the large sample size, the variance of an MLE of a single unknown parameter is approximately the negative of the reciprocal of the Fisher information

$I(\theta )=-E[\frac{\partial^2}{\partial \theta ^2} lnL(\theta |X)]$ .

Thus, the estimate of the variance given data $x$ ,

$\hat{\sigma } ^2=-1/\frac{\partial^2}{\partial \theta ^2} lnL(\hat{\theta } |x)$ ,

the negative reciprocal of the second derivative, also known as the curvature, of the log-likelihood function evaluated at the MLE.

If the curvature is small, then the likelihood surface is flat around its maximum value. If the curvature is large and thus the variance is small, the likelihood is strongly curved at the maximum. For a multidimensional parameter space $\theta =(\theta _1,\theta _2,...,\theta _n)$ . Fisher information $I(\theta )$ is a matrix, the ij-th entry is

$I(\theta _i,\theta _j)=E_\theta [\frac{\partial}{\partial\theta _i} lnL(\theta |X)\frac{\partial}{\partial\theta _j} lnL(\theta |X)]=-E_\theta [\frac{\partial^2}{\partial\theta _i \partial\theta _j} lnL(\theta |X)]$ .

温莎日记 26

温莎日记 26

Maximum Likelihood Estimation

Asymptotic Properties

推荐阅读更多精彩内容