Lecture 2 - 2025 / 2 / 25

1. Infinite communication
2. A probability distribution over the message

Goal: Minimize average code length

Prefix-Free Codes

(1) Prefi-free is a sufficient condition

(2) Is prefix-free a necessary condition? No

$A$: prefix-free codes

$B$: uniquely decodable codes

$A \subsetneq B$

$$\min_{C \in B} E ( \ell (C)) = \min_{C \in A} E ( \ell (C))$$

But why? (Uniquely decodable means $\forall L, S(L) < 2^L$ where Where $S(L)$ represents the number of strings of length $L$. By calculating $S(L)$, Kraft inequality must be satisfied.)

$\forall C^* \in B$ such that $E(\ell(C^*))$ achieves the minimum, $\exists \tilde C^* \in A , E(\ell(C^*)) = E(\ell(\tilde C^*))$

Kraft Inequality for Prefix-free Codes

Theorem. Assume $C = (c_1, \cdots, c_n)$ is prefix-free, Let $\ell_1, \cdots, \ell_n$ be the length (numbers of bits) of $c_1, \cdots, c_n$. Then
$$\sum_{i=1}^{n} 2^{-\ell_i} \le 1$$

Minimal Average Code Length

Setting: Message $M = \{m_1, \cdots, m_n\}$, probability distribution $P = \{p_1, \cdots, p_n\}$.

Goal: Find $C = \{c_1, \cdots, c_n\}$ with length $\ell_1, \cdots, \ell_n$, $C$ is prefix-free.

$$\min_{\ell_1, \cdots, \ell_n} \sum_{i=1}^{n} p_i\ell_i \quad \text{s.t. } \sum_{i=1}^{n} 2^{-\ell_i} \le 1, \ell_i \ge 0$$

Note that $\ell_i$ may not in $\N$.

WLOG we assume that $\sum_{i=1}^{n} 2^{-\ell_i} = 1$. Let $q_i = 2^{-\ell_i}$, then $q_1, \cdots, q_n$ is PMF,
$$\max_{q_1, \cdots, q_n} \sum_{i=1}^{n} p_i \log_2 q_i$$

Therefore, $\ell_i = - \log_2 p_i$.

Definition (Entropy) Given a random source $X$ (random variable), with PMF $(p_1, \cdots, p_n)$. The entropy of $X$ is
$$H(X) := \sum_{i=1}^{n} p_i \log_2 \frac{1}{p_i}$$

1. Minimal code length (description length)
2. Quantify information
3. Uniform distribution $H(X)$ maximum. Deterministic $H(X) = 0$.
4. $H$ measures uncertainty of $X$