liveBook · Manning

Variable bit-rate coding

As explained in Math and Architectures of Deep Learning

A coding scheme that assigns shorter codes to more probable symbols and longer codes to less probable ones to reduce average storage cost.

Variable bit-rate coding reduces average storage by assigning short codes to frequent symbols and longer codes to rare symbols. Its benefit depends on the symbol distribution: skewed frequencies compress well, while equally probable symbols may not. section 6.2

A four-color image

For an image containing 12 green, 2 red, 1 blue, and 1 yellow pixels, use G = 0, R = 10, B = 110, and Y = 111. The image requires 22 bits, or 1.375 bits per pixel, instead of 32 bits with fixed two-bit codes. With four colors occurring equally often, the same variable codes require 36 bits, so fixed-length coding performs better. section 6.2

G(reen) = 0
R(ed) = 10
B(lue) = 110
Y(ellow) = 111

Use the code table to see how prefix-free assignments give the most probable color the shortest representation. section 6.2

Table 6.3 Entropy computation for the pair of images in figure 6.1. The right-hand image has lower entropy and can be compressed more.

Left image

Right image

x1 = G, p(x1) = 4/16 = 0.25

x1 = G, p(x1) = 12/16 = 0.75

x2 = R , p(x2) = 4/16 = 0.25

x2 = R , p(x2) = 2/16 = 0.125

x3 = B, p(x3) = 4/16 = 0.25

x3 = B, p(x3) = 1/16 = 0.0625

x4 = Y, p(x4) = 4/16 = 0.25

x4 = Y, p(x4) = 1/16 = 0.0625

ℍ = −(0.25 log(0.25)+0.25 log(0.25) +
+ 0.25 log(0.25) + 0.25 log(0.25)) = 1.386294

ℍ = −(0.75 log(0.75)+0.125 log(0.125) +
+ 0.0625 log(0.0625) + 0.0625 log(0.0625)) = 0.822265

Compare symbol probabilities and entropy: the image dominated by green has lower entropy and supports stronger compression. section 6.2

How Math and Architectures of Deep Learning covers it

How deep it goes
Defines variable-length and prefix coding, works through storage arithmetic, and connects compression to entropy without implementation details.
Examples
Compares two 4×4 images using fixed two-bit codes, frequency-based codes, and entropy calculations.
What it assumes
Assumes basic symbols, probabilities, and logarithmic entropy calculations while introducing prefix coding through concrete pixels.
What you can do after
Evaluate whether a symbol distribution benefits from variable-length coding by calculating total bits and average bits per symbol.
How it connects
Connects variable bit-rate coding to fixed bit-rate coding, prefix coding, Huffman encoding, entropy, and image compression.
Cover of Math and Architectures of Deep Learning
liveBookmanning.com
Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.