Definition: a parametric probabilistic model or, in short, parametric model is a family of probability distributions that can be described using a finite number of parameters. The set of all allowable values for the parameter, or the parameter space, is denoted as \( \Theta \subseteq \mathbb{R}^k \), and the parametric model is denoted as

\[ \mathcal{P} = \{ P_{\boldsymbol \theta}\ |\ \boldsymbol \theta \in \Theta \} \]

Categories of Parametric Models

Exponential Family

Definition: A parametric model is called an exponential family, if it has the form

\[ f( \mathbf{x} \ | \ \boldsymbol \theta ) = c(\boldsymbol \theta) h(\mathbf{x}) \exp \left\{ \sum_{i=1}^l w_i(\boldsymbol \theta) t_i(\mathbf{x}) \right\} \]

Here \( c(\cdot), h(\cdot) \) are nonnegative, and all functions are real-valued.

In addition,

  1. If k < l, the parametric model is called a curved exponential family.
  2. If k = l, it is called a full exponential family.

Examples of exponential families:

  1. Discrete distributions: Binomial, Poisson, negative binomial;
  2. Continuous distributions: Normal, Gamma, Beta.

Theorem: (Natural Parameterization) For an exponential family, denote natural parameters \( \eta_i = w_i(\boldsymbol \theta) \), then the parametric model can be rewritten as

\[ f( \mathbf{x} \ | \ \boldsymbol \eta ) = c^{*}(\boldsymbol \eta) h(\mathbf{x}) \exp \left\{ \sum_{i=1}^l \eta_i t_i(\mathbf{x}) \right\} \]

And the natural parameter space defined as \( \left\{ \boldsymbol \eta \ \big| \ \int_{\mathbb{R}} h(\mathbf{x}) \exp \left\{ \sum_{i=1}^l \eta_i t_i(\mathbf{x}) \right\} \text{d}x < \infty \right\} \) is convex.

Location-Scale Family

Definition: the location-scale family with standard PDF \( f(x) \) is the parametric model

\[ \mathcal{P} = \left\{ f_{(\mu,\sigma)} = \frac{1}{\sigma} f(\frac{x-\mu}{\sigma}) \ \bigg| \ (\mu,\sigma) \in \mathbb{R}\times\mathbb{R}^{+} \right\} \]

Here, \( \mu \) is called the location parameter, and \( \sigma \) is called the scale parameter.

Theorem:

\[ X \sim f_{(\mu,\sigma)} \Leftrightarrow \exists Z \sim f(z), \text{ s.t. } X = \sigma Z + \mu \]


🏷 Category=Statistics