For any r.v. X with finite expectation and any convex function φ(⋅),
Eφ(X)≥φ(EX)
Note:
Corollary 1: Arithmetic-geometric Mean Inequality
If p1,⋯,pn≥0 and p1+⋯+pn=1, then
n∑i=1piai≥n∏i=1apii
A special case of arithmetic-geometric mean inequality is Young's inequality:
x,y≥0,p,q>0,1p+1q=1⇒xy≤xpp+yqq
Corollary 2: Likelihood Inequality
If X∼p(x) and q(x) is another density function, then
Elogp(X)≥Elogq(X))
Equality holds iff p(x)=q(x).
For all X,g(⋅),h(⋅) s.t. Eg(X),Eh(X),Eg(X)h(X) exist:
If p,q>0,1p+1q=1, then for all r.v. X and Y, given the expectations exist,
|EXY|≤(E|X|p)1p(E|Y|q)1q
Derivation: Using Young's inequality, |X|(E|X|p)1p|Y|(E|Y|q)1q≤|X|ppE|X|p+|Y|qqE|Y|q. Holder's inequality can be derived by taking expectation on both sides.
Corollary: Cauchy-Schwarz Inequality
For all r.v. X and Y, given the expectations exist,
|EXY|≤(E|X|2)12(E|Y|2)12
For any positive r.v. X and r>0,
P(X≥r)≤EXr
Corollary:
Bernstein inequality (exponential version for bounded r.v.'s): Suppose that |X|≤M almost surely, then for all ϵ>0
P(|ˉX−μ|≥ϵ)≤2exp(−nϵ2/2σ2+Mϵ/3)
Concentration inequalities:
Some properties of Gamma, Chi-squared, Poisson and negative binomial distribution.