Lecture 11 - 2025 / 4 / 29

1. Define $C:= \max_{P(X)} I(X; Y)$
2. Channel Coding Theorem
   If $Rate < C, error \to 0$
   If $Rate > C, error \ge \varepsilon_0$

Asymptotic Equipartition Property (AEP)

1. The law of large numbers
   $X_1, \cdots, X_n$ i.i.d., $\frac{1}{n}\sum_{i=1}^{n} X_i\to X$ i.e.
   $$P\left( \left| \frac{1}{n} \sum_{i=1}^{n} X_i - E(X) \right| \ge \varepsilon \right) \xrightarrow{n \to \infty} 0$$

2. $X_1, \cdots, X_n$ i.i.d., let $g$ be a function
   $$P\left( \left| \frac{1}{n} \sum_{i=1}^{n} g(X_i) - E(g(X)) \right| \ge \varepsilon \right) \to 0$$

3. r.v. $X$ discrete, pmf of $X$ is $p(x) := \Pr(X = x)$, let $g(x) := \log \frac{1}{p(x)}$
   $$P\left( \left| \frac{1}{n} \sum_{i=1}^{n} \log \frac{1}{p(X_i)} - E\left(\log \frac{1}{p(X)} \right) \right| \ge \varepsilon \right) \to 0$$

w.p. $\approx 1$, $P(X_1, \cdots, X_n) \in 2^{-n (H(X) \pm \varepsilon)}$

Which implies $A \sube \{0, 1\}^n, A:= \{ (x_1, \cdots, x_n) : P(x_1, \cdots, x_n) \in 2^{-n (H(X) \pm \varepsilon)}\}$, $A^c = \{0, 1\}^n \backslash A$, $P(A) \approx 1, P(A^c) \approx 0$

1. Only need to care about $A$
2. $P(x_1, \cdots, x_n) \approx 2^{-nH(X)}$ in $A$, so $|A| \approx 2^{nH(X)}$

Joint Typical Sequence

$(X, Y), (X_1, Y_1), \cdots, (X_n, Y_n)$ i.i.d. $\sim p(x, y)$

Jointly Typical Set

$$A := \left\{ (x_1,y_1, \cdots, x_n, y_n) : \begin{aligned} P_{XY}(x_1, y_1, \cdots, x_n, y_n) &\in 2^{-n(H(X, Y) \pm \varepsilon)} \\ P_X(x_1, \cdots, x_n) &\in 2^{-n(H(X) \pm \varepsilon)} \\ P_Y(y_1, \cdots, y_n) &\in 2^{-n(H(Y) \pm \varepsilon)} \end{aligned} \right\}$$

1. $P(A) \approx 1$
2. $\forall (x_1, \cdots, y_n) \in A$, $P(x_1, \cdots, y_n) \approx 2^{-nH(X, y)}$, $|A| \approx 2^{nH(X, Y)}$

How to generate $X_1, Y_1, \cdots, X_n, Y_n$ ?

1. $(X_i, Y_i) \sim P(X, Y)$
2. $X_1, \cdots, X_n \sim P(X)$,
   $Y_1, \cdots, Y_n \sim P(Y | X)$

For a fixed typical sequence $x_1, x_2, \cdots, x_n$, the number of corresponding typical sequences of $y_1, \cdots, y_n$ is $2^{n H(Y | X)}$.

If we consider typical sequences of $X$ and $Y$ independently, then the probability that they form a joint typical sequence is $\dfrac{2^{nH(X, Y)}}{2^{n( H(X) + H(Y))}} = 2^{-n I(X;Y)}$