Physics 6311: Statistical Mechanics - Homework 4

Problem 1: Additivity of Shannon entropy (8 points)

Consider two discrete, jointly distributed random variables $X$ and $Y$ with values $x_i$ and $y_j$, respectively. The joint probability of $X$ having the value $x_i$ and $Y$ having the value $y_j$ is $p_{\mathit{ij}}$.

a)

Show that if $X$ and $Y$ are statistically independent, then the Shannon entropy $S_s$ of the joint distribution is the sum of the Shannon entropies of the reduced distributions of $X$ and $Y$

b)

Generalize the derivation to the case on $M$ jointly distributed variables $X^{(m)}$ with $m=1\dots M$.

Problem 2: Random $N$-Letter words (8 points)

Your printer malfunctions and prints random words, each consisting of $N$ lower-case (English) letters. Find the Shannon entropy for this distribution of words

a)

if all letters of the alphabet occur with equal probability as each of the $N$ letters (positions) in the word;

b)

if every second letter in the word is one of the vowels (a, e, i, o, u) while every other letter in the word is not a vowel

Hint: Use the result of problem 1, generalized to $N$ random variables.

Problem 3: Maxima of Shannon entropy (8 points)

Consider the entropy of a discrete probability distribution given in terms of the probabilities $p_i$ ($i=1...N$). Determine which $p_i$ lead to the maximum entropy under the following constraints (Hint: Use Lagrange multipliers to enforce the constraints.):

a)

Normalization ${\sum <!--\; FUNCTION\; APPLICATION\; -->}_i{p}_i=1$

b)

Normalization ${\sum <!--\; FUNCTION\; APPLICATION\; -->}_i{p}_i=1$ and fixed average $⟨a⟩={\sum <!--\; FUNCTION\; APPLICATION\; -->}_i{p}_i{a}_i$ of a quantity $A$ with values $a_i$.

Problem 4: 3-level atoms in microcanonical ensemble (16 points)

A system consists of $N$ non-interacting, distinguishable three-level atoms. Each atom can be in one of three states with energies $E_1=E_2=0$ and $E_3=𝜖$. .

a)

Work out the number of available states $\mathrm{\Omega }$ at fixed $N_3$ (number of atoms in level 3).

b)

Calculate the entropy as a function of the total energy $E$ (use Stirling’s formula).

c)

Calculate the temperature.

d)

Express the energy as a function of temperature.