8) Bayesian Statistical Inference

8.1 Bayesian Inference and the Posterior Distribution

• Bayesian Inference involves treating unknown parameters as random variables with known prior distributions. The posterior distribution of the parameter, given the observed data, provides the updated beliefs after observing the data.
• Posterior Distribution: Once data $X=x$ is observed, the posterior distribution of a parameter $\theta$ is derived from Bayes’ rule: $$f_{\theta |x}(\theta |x)=\frac{f_{\theta }(\theta )f_{X|\theta }(x|\theta )}{\int f_{\theta }({\theta }^{\prime })f_{X|\theta }(x|{\theta }^{\prime })d{\theta }^{\prime }}$$ This posterior can be used for parameter estimation, hypothesis testing, and predictive analysis.

8.2 Point Estimation, Hypothesis Testing, and the MAP Rule

• Maximum A Posteriori (MAP) Estimation: The MAP rule selects the parameter value with the highest posterior probability: $${\hat{\theta }}_{MAP}=\arg \underset{\theta }{\max }{f}_{\theta |x}(\theta |x)$$ This is useful in both point estimation and hypothesis testing.

8.3 Bayesian Least Mean Squares Estimation

• Least Mean Squares (LMS) Estimation: Selects an estimator that minimizes the mean squared error: $${\hat{\theta }}_{LMS}=E[\theta |X=x]$$ The LMS estimator is the posterior mean of $\theta$ given the data.

8.4 Bayesian Linear Least Mean Squares Estimation

• Linear LMS Estimation: In some cases, the estimator is restricted to be a linear function of the observations. It minimizes the mean squared error subject to this linearity constraint: $${\hat{\theta }}_{LLMS}=a^{T}X+b$$ where the coefficients $a$ and $b$ are chosen to minimize the mean squared error.

8.5 Summary and Discussion

• Bayesian inference provides a systematic framework to update beliefs based on observed data. It requires specifying a prior distribution and a likelihood function. While the exact posterior distribution may be difficult to compute in practice, approximations and numerical methods can be used.
• Key Advantages:
  • Explicit use of prior information.
  • The posterior distribution provides a complete description of uncertainty.