Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
MLE
Maximum likelihood estimation:
multidimensional case
Shota Gugushvili
Leiden University
Leiden, 20 October 2014
Outline
Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
MLE
1 Multivariate normal
2 MLE
Multivariate normal: particular case
Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
MLE
Definition
Let


Z1
 
Z =  ... 
Zk
where Z1 , . . . , Zk ∼ N(0, 1) are independent. Then Z is said to
have multivariate normal distribution with mean 0 (k × 1
vector of zeros) and covariance matrix I (k × k identity
matrix). The PDF of Z is given by


k
X
1
1
1
1
2
T
f (z) =
exp −
zj  =
exp − z z .
2
2
(2π)k/2
(2π)k/2
j=1
Multivariate normal: general case
Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
MLE
Definition
A vector


X1
 
X =  ... 
Xk
is said to have multivariate normal distribution with mean µ
(vector of length k) and covariance matrix Σ (k × k symmetric,
positive definite matrix), which we denote by X ∼ N(µ, Σ), if
its PDF is given by
1
1
T −1
f (x; µ, Σ) =
exp − (x − µ) Σ (x − µ) .
2
(2π)k/2 |Σ|1/2
Properties of a normal vector
Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
Theorem
If Z ∼ N(0, I ) and X = µ + Σ1/2 Z , then X ∼ N(µ, Σ).
Conversely, if X ∼ N(µ, Σ), then Σ−1/2 (X − µ) ∼ N(0, I ).
MLE
Theorem
Let X ∼ N(µ, Σ). Then Xj ∼ N(µj , Σjj ). Furthermore,
aT X ∼ N(aT µ, aT Σa),
and
V = (X − µ)T Σ−1 (X − µ) ∼ χ2k .
Asymptotic normality
Maximum
likelihood
Theorem
Shota
Gugushvili
Let X1 , . . . , Xn be IID random vectors, where
 
X1i
 .. 
X = . 
Xki
Multivariate
normal
MLE
with mean
 

µ1
E[X1i ]
  

µ =  ...  =  ... 

µk
E[Xki ]
and covariance
P matrix Σ. Let X be a vector with components
X j = n−1 ni=1 Xji . Then
√
n(X − µ)
N(0, Σ).
MLE and Fisher information
Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
MLE
Let θ = (θ1 , . . . , θk ), and let θˆ = (θˆ1 , . . . , θˆk ) be the MLE.
P
Let `n = ni=1 log f (Xi ; θ) and
Hjk =
∂ 2 `n
.
∂θj ∂θk
The k × k matrix In (θ) = −(E[Hjk ]) is called the Fisher
information matrix.
Let Jn (θ) = In−1 (θ) be the inverse of In (θ).
Asymptotic normality
Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
MLE
Theorem
Under mild assumptions,
√
n(θˆ − θ) ≈ N(0, Jn ).
Also,
θˆj − θj
se
ˆj
N(0, 1),
where se
ˆ 2j is the jth diagonal element of Jn . The approximate
covariance of θˆj and θˆk is Jn (j, k).
Multiparameter delta method
Maximum
likelihood
Shota
Gugushvili
Theorem
Let τ = g (θ1 , . . . , θk ) and let
Multivariate
normal
 ∂g 
∂θ1
 
∇g =  ... 
MLE
∂g
∂θ1
ˆ Then
. Let τˆ = g (θ).
τˆ − τ
se(ˆ
ˆ τ)
where se(ˆ
ˆ τ) =
ˆ
ˆ = ∇g (θ).
∇g
N(0, 1),
q
ˆ and
ˆ )T Jˆn (∇g
ˆ ), with Jˆn = Jn (θ)
(∇g
Example
Maximum
likelihood
Shota
Gugushvili
Multivariate
normal
MLE
Example
Let X1 , . . . , Xn be IID N(µ, σ 2 ). Suppose we want to estimate
τ = g (µ, σ) = σ/µ. It can be shown that
1 σ2 0
Jn (θ) =
2
n 0 σ2
and ∇g = (−σ/µ2 , 1/µ)T . Thus
q
ˆ )T Jˆn (∇g
ˆ ) = √1
se
ˆ = (∇g
n
s
1
σ
ˆ2
+
.
µ
ˆ4 2ˆ
µ2

Erdinger 2 bonnen Bier - wij n en frisdrank 1 bon Broodje hotdog 1

slides - Tohoku University

PowerPoint プレゼンテーション