Linear Discriminant Analysis

📘 Linear Discriminant Analysis (LDA)

🎯 Complete step‑by‑step derivation (no omission)
📐 From Gaussian Bayes → Equal variance assumption → Quadratic expansion → Linear boundary
🧠 Includes comparison with QDA and interpretation

---

1. Start from Gaussian Bayes Discriminant (1D)

From Gaussian discriminant analysis:

\[f^*(x) = \text{sign}\left( -\frac12\log\frac{\sigma_1^2}{\sigma_{-1}^2} -\frac{(x-\mu_1)^2}{2\sigma_1^2} +\frac{(x-\mu_{-1})^2}{2\sigma_{-1}^2} + \log\frac{\alpha}{1-\alpha} \right)\]

---

2. LDA Assumption (Equal Variances)

LDA assumes equal variances across classes:

\[\sigma_1^2 = \sigma_{-1}^2 = \sigma^2\]

Substitute:

\[f^*(x) = \text{sign}\left( -\frac12\log\frac{\sigma^2}{\sigma^2} -\frac{(x-\mu_1)^2}{2\sigma^2} +\frac{(x-\mu_{-1})^2}{2\sigma^2} + \log\frac{\alpha}{1-\alpha} \right)\]

Since:

\[\log\left(\frac{\sigma^2}{\sigma^2}\right)=\log(1)=0\]

we get:

\[f^*(x) = \text{sign}\left( -\frac{(x-\mu_1)^2}{2\sigma^2} +\frac{(x-\mu_{-1})^2}{2\sigma^2} + \log\frac{\alpha}{1-\alpha} \right)\]

Factor:

\[f^*(x) = \text{sign}\left( \frac{1}{2\sigma^2}\big[(x-\mu_{-1})^2-(x-\mu_1)^2\big] + \log\frac{\alpha}{1-\alpha} \right)\]

---

3. Expand Quadratic Terms

\[(x-\mu_1)^2 = x^2 - 2\mu_1 x + \mu_1^2\] \[(x-\mu_{-1})^2 = x^2 - 2\mu_{-1} x + \mu_{-1}^2\]

Compute difference:

\[(x-\mu_{-1})^2-(x-\mu_1)^2 = (x^2 - 2\mu_{-1}x + \mu_{-1}^2) - (x^2 - 2\mu_1x + \mu_1^2)\]

Cancel \(x^2\):

\[= -2\mu_{-1}x + \mu_{-1}^2 + 2\mu_1x - \mu_1^2\]

Group terms:

\[= 2(\mu_1-\mu_{-1})x + (\mu_{-1}^2-\mu_1^2)\]

---

4. Substitute Back

\[f^*(x) = \text{sign}\left( \frac{1}{2\sigma^2}\left[2(\mu_1-\mu_{-1})x + (\mu_{-1}^2-\mu_1^2)\right] + \log\frac{\alpha}{1-\alpha} \right)\]

Distribute:

\[= \text{sign}\left( \frac{\mu_1-\mu_{-1}}{\sigma^2}x + \frac{\mu_{-1}^2-\mu_1^2}{2\sigma^2} + \log\frac{\alpha}{1-\alpha} \right)\]

Rearrange:

\[= \text{sign}\left( \frac{\mu_1-\mu_{-1}}{\sigma^2}x - \frac{\mu_1^2-\mu_{-1}^2}{2\sigma^2} + \log\frac{\alpha}{1-\alpha} \right)\]

---

5. Final Linear Form

Define:

\[w = \frac{\mu_1-\mu_{-1}}{\sigma^2}\] \[b = -\frac{\mu_1^2-\mu_{-1}^2}{2\sigma^2} + \log\frac{\alpha}{1-\alpha}\]

Classifier:

\[f^*(x) = \text{sign}(wx + b)\]

👉 Decision boundary is linear.

---

6. LDA vs QDA

QDA (Different Variances)

If:

\[\sigma_1^2 \neq \sigma_{-1}^2\]

Quadratic terms remain → decision boundary is quadratic.

---

LDA (Equal Variances)

If:

\[\sigma_1^2 = \sigma_{-1}^2\]

Quadratic terms cancel → boundary is linear:

\[f^*(x) = \text{sign}(wx+b)\]

---

7. Interpretation

• LDA assumes shared variance/covariance
• Produces linear decision boundary
• More stable with small datasets
• QDA more flexible but higher variance
• LDA ≈ Gaussian generative model with shared covariance