Skip to main content

Cross-entropy, feedback and variational inference

What the functional selects​

Fix a density matrix Γ\Gamma. For an output ρ\rho supported on its support, the von Neumann entropy and the quantum relative entropy satisfy

S(ρ)+D(ρ∥Γ)=−Tr(ρlog⁡Γ).S(\rho)+D(\rho\Vert\Gamma)=-\mathrm{Tr}(\rho\log\Gamma).

The right side is linear in the output. Diagonalizing Γ\Gamma shows that its minimum is −log⁡λmax⁡(Γ)-\log\lambda_{\max}(\Gamma). Equality holds precisely when the output is supported on the maximal eigenspace. The bound is attainable over all CPTP maps: discard the input and prepare any state on that space. This optimization cannot single out a feedback channel; even fixing the output for one input generally leaves its action on others undetermined. To select a channel one needs a task with independently specified observations, allowed resources and a rule for scoring different channels. Resource cost of feedback

Two different optimization problems​

For diagonal input pp and output qq, the functional becomes the cross-entropy −∑iqilog⁡pi-\sum_i q_i\log p_i. Its minimum lies on the most probable input symbol. Variational inference instead starts with a joint generative law p(x,z)p(x,z), observed xx with p(x)>0p(x)>0, and a trial posterior q(z)q(z). Its objective is

F(q;x)=D(q∥p(⋅∣x))−log⁡p(x).F(q;x)=D(q\Vert p(\cdot\mid x))-\log p(x).

With an unrestricted posterior family it is minimized by the posterior itself. For p=(0.7,0.2,0.1)p=(0.7,0.2,0.1) the first objective is about 0.3570.357 at a point mass and 0.8020.802 at q=pq=p, in nats. Removing off-diagonal entries neither supplies a sensory observation nor selects the posterior. A bridge to active inference needs a generative law, interventions and a learning rule that can be compared on data.

Spectral entropy on the state domain​

For a positive operator of unit trace, all eigenvalues are nonnegative, so the sum −∑i∣λi∣log⁡∣λi∣-\sum_i |\lambda_i|\log|\lambda_i| with 0log⁡0=00\log0=0 equals the von Neumann entropy. For a general operator it is not a state entropy. This identity gives no extremal characterization of the self-model.