Computability

A decision problem $L \sube \{0, 1\}^*$

an instance of the decision problem $x \in \{0, 1\}^*$, decide if $x \in L$

Kolmogrov Complexity

The $k$-complexity of a string $x \in \{0, 1\}^*$ w.r.t. a universal Turing Machine $U$ is defined as $K(x) := \min\limits_{U(p) = x} |p|$.

Theorem. Let $U, U^*$ be 2 universal TMs. Then $\exists c \in \N$ s.t. for $\forall x \in \{0, 1\}^*, K_U(x) \le K_{U'}(x) + c$, where $c$ doesn't depend on $x$.

There is no algorithm such that given program $p$ and input $x$, the algorithm can decide if $p$ with input $x$ halts in finite stops.

Theorem. $K_U(x)$ is not computable.

Proof. Assume for the sale of contradiction, $\exists p_0$ can compute $K_U(x)$ for all $x \in \{0, 1\}^*$, WLOG assume $|p_0| = O(10^3)$.

Construct $p_0'$ with length $O(10^3)$, but output a string $s$ with $K_U(s) = O(10^8):$

for $s_i \in \{0, 1\}^*$ in shortlex order:
$\quad$ compute $K_U(s_i)$ using $p_0$.
$\quad$ if $K_U(s_i) \ge 10^8$:
$\qquad$ output $U(s_i)$, return.

Probability Density Estimation

Given partial information about the pdf $f(x)$

$\int g_i(x) f(x) \text d x = r _i$, $i = 1, 2, \cdots, m$

$\max _f : -\int f(x) \ln f(x) \text d x$

Continuous r.v. $X$, pdf $f(x)$, partial information

(1) $E(X) = 0$ (2) $Var(x) = \sigma^2$

MaxEnt distribution ?