Variable-length encoding

As explained in Julia as a Second Language

Variable-length character encoding represents different characters with different numbers of code units or bytes.

Variable-length character encoding stores different characters in different numbers of bytes or code units. UTF-8 saves space by using one byte for frequently used characters while remaining compatible with ASCII. section 11.1.1

Keep these related ideas distinct:

• A code point identifies a Unicode character; code units store that code point in memory. section 11.1.1
• ASCII character A uses one code unit, Æ uses two, and 😏 uses four. section 11.1.1
• String indexes can therefore skip numbers, so Julia uses nextind rather than assuming every character occupies one position. section 11.1.1

Inspecting characters in Julia

codepoint('A') returns 0x00000041 and ncodeunits('A') returns 1; ncodeunits('Æ') returns 2, while ncodeunits('😏') returns 4. Iterating with nextind correctly traverses strings such as "Þrúðvangar". section 11.1.1

Figure 11.1 Unicode code points encoded to UTF-8 code units (not every index is shown)

Use the encoding diagram to see how Unicode code points map to UTF-8 code units and why byte indexes do not always equal character positions. section 11.1.1

How Julia as a Second Language covers it

How deep it goes

Defines code points and code units, demonstrates differing widths, and examines nextind implementation without fully developing encoding algorithms.

Examples

Uses ASCII A, Æ, 😏, Þrúðvangar, Julia REPL queries, and nextind-based string iteration.

What it assumes

Assumes basic characters and strings, while teaching Unicode code points, code units, and variable-width indexing.

What you can do after

Evaluate/understand character widths and traverse Unicode strings without assuming one byte per character.

How it connects

Connects Unicode, UTF-8, ASCII, code points, code units, strings, and string indexing.

liveBookmanning.com

Read Julia as a Second Language on liveBook Get the book at manning.com

Related concepts

• encodingkind of · Encoding is the transformation of information into a representation defined by a scheme for storage, processing, transmission, or interpretation.

This explainer was generated by Manning's knowledge graph from the text of Julia as a Second Language. The section links point to the parts of the book it draws on.