Lecture 12 - 2025 / 5 / 13

Shannon Channel Coding Theorem

$C :=$ Channel Compacity $= \max_{P(X)} I(X; Y)$

• $R < C$, $\exists$ error correcting code, $err \to 0$
• $R > C$, $\forall$ error correcting code, $err \ge \varepsilon_0 > 0$

Proof. (Assume $R < C$)

1. Assumption on raw message, message $= \{m_1, m_2, \cdots, m_{2^{nR}}\}$, uniform distribution.

2. Design codebook $CB = (c_1, \cdots, c_{2^{nR}})$
   $$\begin{aligned} m_1 &\to C_1 = (c_{11}, c_{12}, \cdots, c_{1n})\\ m_2 &\to C_2 = (c_{21}, c_{22}, \cdots, c_{2n})\\ &\vdots\\ m_{2^{nR}} &\to C_{2^{nR}} = (c_{2^{nR}1}, c_{2^{nR}2}, \cdots, c_{2^{nR}n}) \end{aligned}$$

Idea: random coding

$c_{ij} \sim$ i.i.d. $P(X)$ for $i \in [2^{nR}], j \in [n]$

$P(X) = \argmax_{P(X)} I(X; Y)$

3. Decoding algorithm / mode
   Upon reveriving $(y_1, \cdots, y_n) \xrightarrow{?} c_i$

Rule: $c_i = (x_1, \cdots, x_n)$ s.t. $(x_i, \cdots, x_n, y_1, \cdots, y_n)$ is a joint typical sequence w.r.t. $P(X)P(Y|X)$

• (a) If no such $(x_1, \cdots, x_n)$, report fail (implicitly, if $(y_1, \cdots, y_n)$ isn't typical sequence, report fail).
• (b) If there are more than $1$ codewords forming jointly typical sequence of $(y_1, \cdots, y_n)$, report fail.
• (c) If there is a unique $c_i = (x_1, \cdots, x_n)$ forming jointly typical sequence $(y_1, \cdots, y_n)$, ok.

$$P(b) \le 2^{nR} \cdot 2^{-nI(X; Y)} = 2^{n(R-C)}\to 0$$