Shannon’s Theorems: From Data Compression to Reliable Ham Radio

Topic Snapshot

What Shannon Established in 1948

In A Mathematical Theory of Communication, published in 1948, Shannon developed mathematical limits for representing information and communicating it through noise. His framework measures uncertainty and distinguishable messages. It does not measure a message’s importance, truth, or usefulness.^[1]

These are mathematical limits under stated assumptions. They do not promise that a particular modem achieves capacity, or that a finite transmission has literally zero errors.^[1]

Entropy: Measuring Average Information

Why a Logarithm Appears

Let p be an event’s probability. An information measure should assign zero surprise to a certain event and more surprise to a rarer one. For independent events, probabilities multiply, but their information should add.

With continuity and this additive rule, the measure has logarithmic form. Choosing base 2 makes a one-in-two outcome worth one bit:

i(p) = −log₂(p) = log₂(1/p)

Thus, i(pq) = i(p) + i(q). Averaging over all outcomes x gives Shannon entropy:^[2]

H(X) = −Σ_x p(x) log₂ p(x)

Here, X is a random variable, p(x) is the probability of outcome x, and Σ means “add over all outcomes.” A zero-probability term contributes zero. Logarithms to base 2 ask what power of 2 produces the number inside the logarithm.

A Four-Symbol Example

Imagine a source that independently produces four symbols with the following probabilities. The code shown has no complete codeword at the beginning of another, so a decoder can separate consecutive symbols without extra separators.

For this example:

H(X) = (1/2 × 1) + (1/4 × 2) + (1/8 × 3) + (1/8 × 3) = 1.75 bits per symbol.

A fixed two-bit representation uses 2 bits for every symbol. This variable-length code averages 1.75 bits, a 12.5% reduction before any file headers or codebook overhead. For example, ABCD becomes 010110111. That particular sequence takes nine bits; the savings apply to the probability-weighted average, not every individual message.

As another check, a fair binary source has entropy 1 bit per symbol. A source producing 0 with probability 0.9 and 1 with probability 0.1 has entropy about 0.469 bits per symbol. Its predictability offers more room for compression.

Why Lossless Compression Has a Limit

For a binary prefix code, let ℓ(x) be a codeword’s length and L its average length. The Kraft inequality requires Σ2^−ℓ(x) ≤ 1. This constraint, together with the nonnegativity of relative entropy, gives L ≥ H(X).

To see why the bound is approachable, choose ℓ(x) = ⌈−log₂p(x)⌉, rounding each ideal length upward. These lengths satisfy the Kraft inequality and give:^[3]

H(X) ≤ L < H(X) + 1.

Now encode blocks of n independent source symbols. Their entropy is nH(X), so a suitable block prefix code satisfies:

H(X) ≤ L_n/n < H(X) + 1/n.

As n grows, the overhead per symbol can shrink toward zero. Exact recovery remains possible. For sources with memory, such as text, the relevant long-run limit is the entropy rate, which accounts for dependencies between symbols.^[3]

Lossy compression addresses a different question: how small can a representation become when some reconstruction error is allowed? Shannon’s rate-distortion framework relates the required rate to an explicitly chosen distortion measure.^[4]

Mutual Information: What the Receiver Learns

Next, let X represent the transmitted channel input and Y the observed output. Conditional entropy, H(X|Y), is the average uncertainty about X remaining after Y is known. Their mutual information is:

I(X;Y) = H(X) − H(X|Y).

Using p(x,y) = p(y)p(x|y), the same quantity becomes:

I(X;Y) = Σ_x,y p(x,y) log₂[p(x,y)/(p(x)p(y))].

If input and output are independent, the ratio is 1 and mutual information is zero. If Y identifies X perfectly, the remaining uncertainty is zero and I(X;Y) = H(X). Mutual information is symmetric, even though a radio link has a transmitting end and a receiving end.^[2]

Channel Capacity and Reliable Communication

A discrete memoryless channel is described by p(y|x), the probability of each output given an input. “Memoryless” means each use depends on the current input rather than earlier uses. Its capacity is:

C = max_p(x) I(X;Y) bits per channel use.

The maximization selects the input probabilities that convey the most information through that channel. Multiplying by the number of channel uses per second converts this result to bits per second.^[5]

Example: A Channel That Flips Bits

Suppose each transmitted bit flips independently with probability p. This is a binary symmetric channel. Its binary entropy is:

H₂(p) = −p log₂p − (1 − p) log₂(1 − p).

For this channel, H(Y|X) = H₂(p). Equally likely input bits make H(Y) = 1, its maximum. Therefore:

C = 1 − H₂(p).^[5]

At p = 0.10, C ≈ 1 − 0.469 = 0.531 bits per use. At 1,000 uses per second, the capacity is about 531 information bits per second. Coding overhead occupies part of the transmitted stream; 1,000 transmitted binary symbols do not necessarily represent 1,000 new information bits.

Why Coding Can Approach Capacity

A proof sketch helps explain Shannon’s result. Over n channel uses, a rate-R code has roughly 2^nR possible messages. Randomly chosen long codewords can become distinguishable at the receiver because noise produces statistically predictable patterns.

For an input distribution with mutual information I(X;Y), the probability that an unrelated codeword looks compatible with the received output falls roughly like 2^−nI(X;Y). Comparing it against roughly 2^nR candidates gives the characteristic factor 2^−n(I−R). When R < I, that term decreases with block length. Formal proofs also control atypical events.^[6]

The converse establishes that rates above capacity cannot have error probability tending to zero. For a memoryless channel, Fano’s inequality and the bound I(Xⁿ;Yⁿ) ≤ nC connect reliable recovery of a message to R ≤ C.^[7]

The practical challenge is building good codes with manageable processing and delay. Longer blocks are not free: operators and applications still need timely messages.

Deriving the Shannon–Hartley Formula

Start with One Gaussian Channel Use

Consider Y = X + Z, where Z is independent, zero-mean Gaussian noise with variance σ², and the input satisfies E[X²] ≤ P. Here, E means average or expected value.

For continuous variables, use differential entropy, h. A Gaussian variable of variance v has:

h = (1/2) log₂(2πev).

This follows by inserting the Gaussian probability density into h = −∫f(u)log₂f(u) du and using its variance; π is pi and e is the base of natural logarithms. Among distributions with a fixed variance, the Gaussian has the largest differential entropy.^[8]

Because the noise is independent, h(Y|X) = h(Z). Consequently:

I(X;Y) = h(Y) − h(Z)
≤ (1/2) log₂[2πe(P + σ²)] − (1/2) log₂(2πeσ²)
= (1/2) log₂(1 + P/σ²).

A zero-mean Gaussian input using the full allowed power achieves equality in this model. This is capacity per real channel use.^[9]

Convert Channel Uses into Bits per Second

An ideal real channel of bandwidth B has 2B real signaling dimensions per second. Multiplying the per-dimension capacity by 2B gives:^[10]

C = B log₂(1 + S/N).

Convert first: S/N = 10^SNR_dB/10. Thus, 10 dB means 10, 0 dB means 1, and −10 dB means 0.1. Do not put “−10” directly into the capacity formula.

The model assumes additive white Gaussian noise (AWGN): noise adds to the signal, has a flat power spectrum over the channel, and follows a Gaussian amplitude distribution. The formula describes digital information carried by a continuous waveform. It does not require the original message to be analog.

A 3,000-Hz Worked Example

With B = 3,000 Hz and an in-band SNR of 10 dB:

C = 3,000 log₂(11) ≈ 10,378 bits per second.

The following values are calculations for that idealized model, not measured modem performance.

xychart-beta
    title "Ideal Capacity in a 3000 Hz Channel"
    x-axis "In-band SNR in dB" ["-10", "-5", "0", "5", "10", "15", "20"]
    y-axis "Capacity in bits per second" 0 --> 21000
    line [413, 1189, 3000, 6172, 10378, 15083, 19975]

For another comparison, doubling received signal power from S/N = 10 to 20 raises capacity from about 10,378 to 13,177 bits/s. That is roughly a 27% increase, not a doubling.

Why More Bandwidth Does Not Mean Unlimited Capacity

With fixed received power S and white-noise density N₀, the in-band noise is N = N₀B. Therefore:

C(B) = B log₂[1 + S/(N₀B)].

As B grows without bound, use ln(1 + u) ≈ u for small u:

C → S/(N₀ ln 2).

Capacity approaches a finite limit. Doubling bandwidth doubles capacity only if S/N stays fixed; keeping that ratio fixed in white noise requires more received signal power.^[10]

The same ideal model gives a minimum energy-per-information-bit ratio. At capacity, let η = C/B and E_b = S/C. Then:

E_b/N₀ = (2^η − 1)/η → ln 2 ≈ 0.693, or −1.59 dB, as η → 0.

This is a limiting energy-efficiency result at vanishing spectral efficiency. It is not a universal SNR threshold for an FT8 decoder or any other practical receiver.^[10]

How It Works: Compress, Protect, and Recover

Source coding removes predictable redundancy. Channel coding adds carefully structured redundancy that helps a receiver correct errors. These stages serve different purposes and can work together.

flowchart TD
    A["Message source"] --> B["Source coding"]
    B --> C["Channel coding and modulation"]
    C --> D["Radio channel"]
    N["Noise"] --> D
    D --> E["Demodulation and decoding"]
    E --> F{"Message passes checks?"}
    F -- Yes --> G["Source decoding and delivery"]
    F -- No --> H["Reject or request a repeat"]

Practical Applications

Data Compression: DEFLATE and Huffman Coding

DEFLATE combines LZ77, which represents repeated strings using references to earlier data, with Huffman coding, which assigns variable-length bit patterns to symbols. The gzip format uses DEFLATE compression.^[11]

However, no lossless compressor can shorten every possible input. There are fewer short bit strings than long ones, so some inputs must stay the same size or expand. File headers also matter, especially for small files.

Try it: Compress a text file with many repeated lines. Then compress its compressed output again. Record both sizes, and verify that decompression restores the original bytes. Explain the outcome in terms of remaining predictable structure.

5G: Practical Error-Correcting Codes

Fifth-generation cellular systems use New Radio (NR). The 3rd Generation Partnership Project (3GPP) specification defines low-density parity-check (LDPC) coding for shared data channels and polar coding for important control and broadcast information. Some short control payloads use other coding arrangements.^[12]

These are practical ways to protect information. Actual data rates also depend on assigned radio resources, modulation, overhead, and channel conditions. A phone’s observed download rate is not the capacity of one ideal Gaussian channel.

For a club discussion, ask: when reception gets worse, what could a system change to favor reliability over speed? The Shannon framework helps explain why that tradeoff exists. Check the official specifications for current implementation details.

Ham Radio: Why FT8 Is a Useful Example

FT8 is a digital amateur-radio mode designed for short exchanges under weak-signal conditions. Its message format packs information into 77 bits. A 14-bit cyclic redundancy check (CRC) helps detect errors, and an LDPC code expands the resulting 91 bits into a 174-bit codeword.^[13]

That illustrates both efficient message representation and forward error correction (FEC). The transmitted waveform also includes synchronization information. The 174 coded bits are not 174 independent payload bits.

FT8 occupies approximately 50 Hz, while its reported signal-to-noise ratios use a 2,500-Hz reference bandwidth. Therefore, a negative signal report must be interpreted with its measurement bandwidth.^[14]

For an approximate illustration, assume flat noise and that a 50-Hz measurement captures essentially all the signal power. Converting a −20 dB report gives:

SNR_{50 Hz} ≈ −20 + 10 log₁₀(2,500/50) ≈ −3.01 dB.

That is a linear ratio of approximately 0.5. An ideal 50-Hz AWGN channel at that ratio would have:

C ≈ 50 log₂(1.5) ≈ 29.25 bits/s.

This is our simplified calculation, not an FT8 throughput prediction or decoding threshold. It shows why bandwidth definitions matter. Consult the current WSJT-X documentation for operating and decoder details.

How to Participate: Three Small Experiments

1. Build a source code. Use the A–D table to encode a short message. Trade it with a partner, decode it, and compare the total with a fixed two-bit code. Then discuss why a short sample can differ from the average.
2. Calculate a channel limit. Reproduce the 3,000-Hz example. Change only SNR, then try a bandwidth change while holding signal power and noise density fixed. Keep track of what you are holding constant.
3. Observe a digital mode. If you have a receiving setup, record the mode, signal report, reference bandwidth, and decoding outcome. Compare several observations before drawing conclusions.

A calculator and paper are enough for the first two activities. For the third, bring a receiver and computer if available, or work with a member who already has a station. Use the exercise to ask questions, rather than treat a few observations as a performance benchmark.

For an operating takeaway, consider changes that improve the received signal-to-noise ratio or fit the information rate to the available channel. These follow directly from the model. Real interference, fading, receiver overload, and protocol overhead require further investigation.

Suggested SARC Goals

Give It a Try

You do not need to master every proof before Shannon’s ideas become useful. Start with one question: how much new information is in the message, or how much can the channel reliably carry?

Then calculate one example and share what you learned with another SARC member. For related background, explore The Physics Behind Amateur Radio. A familiar signal on your screen can become an invitation to understand the engineering behind it.

Try one calculation and share what it teaches you about your favorite digital mode.

References

1. Claude E. Shannon.
   A Mathematical Theory of Communication.
   Bell System Technical Journal; author’s paper hosted by Yale University.
   ↩
2. Information, Entropy, and the Motivation for Source Codes, MIT 6.02, Chapter 2; and Muriel Médard.
   Information Theory, Lecture 1: Entropy and Mutual Information.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
3. Muriel Médard.
   Information Theory, Lecture 5: Codes, Kraft Inequality, and Optimal Codes;
   see also Lecture 4 on entropy rates.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
4. Yury Polyanskiy and Yihong Wu.
   Information Theory Lecture Notes: Rate-Distortion Theory, Chapters 23–25.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
5. Muriel Médard.
   Information Theory, Lecture 8: Channel Capacity and Binary Channels.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
6. Muriel Médard.
   Information Theory, Lecture 10: The Channel Coding Theorem.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
7. Muriel Médard.
   Information Theory, Lecture 13: Fano’s Inequality and the Converse to the Coding Theorem.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
8. Muriel Médard.
   Information Theory, Lecture 16: Differential Entropy.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
9. Muriel Médard.
   Information Theory, Lecture 17: Additive Gaussian Noise Channels.
   Massachusetts Institute of Technology, MIT OpenCourseWare.
   ↩
10. Yury Polyanskiy and Yihong Wu.
    Information Theory, Chapter 19: Energy-per-Bit and Continuous-Time Channels.
    Massachusetts Institute of Technology, MIT OpenCourseWare.
    ↩
11. L. Peter Deutsch.
    RFC 1951: DEFLATE Compressed Data Format Specification, Version 1.3.
    RFC Editor.
    ↩
12. 3rd Generation Partnership Project.
    3GPP TS 38.212: NR Multiplexing and Channel Coding, Release 18, Version 18.4.0.
    European Telecommunications Standards Institute (ETSI).
    ↩
13. Steven J. Franke, Bill Somerville, and Joe Taylor.
    The FT4 and FT8 Communication Protocols.
    QEX, American Radio Relay League (ARRL); hosted by the WSJT project.
    ↩
14. WSJT development team.
    WSJT-X User Guide: Protocol Specifications and Signal Reports.
    WSJT project.
    ↩