liveBook · Manning

Unicode

As explained in Julia as a Second Language

Unicode is a universal character standard that assigns identifiers and encoding rules to characters across writing systems.

Unicode maps characters from many writing systems to numbers called code points. This makes it possible to represent text beyond A–Z without relying on a different interpretation of character numbers for each language. section 11.0 section 11.1.1

A code point identifies a character; an encoding determines how to represent it. Julia strings use UTF-8, which encodes a character in one to four bytes. ASCII letters such as A use one byte, while Æ uses two. The one-byte UTF-8 characters are compatible with ASCII. section 11.1 section 11.1.1

julia> codepoint('Æ')
0x000000c6
 
julia> ncodeunits('Æ')   #1
2
 
julia> isascii('Æ')      #2
false
 
julia> codepoint('😏')   #3
0x0001f60f
 
julia> ncodeunits('😏')  #4
4
 
julia> isascii('😏')
false

Compare the code points and UTF-8 code-unit counts for Æ and an emoji. section 11.1.1

This distinction matters when indexing Julia strings. `length("Æser")` is 4, but `sizeof("Æser")` is 5 bytes. The first character occupies two bytes, so index 2 is not a valid character index; the next character starts at index 3. section 11.1

julia> "Æser"[1]                                     #1
'Æ': Unicode U+00C6 (category Lu: Letter, uppercase)
 
julia> "Æser"[2]                                     #2
ERROR: StringIndexError("Æser", 2)
 
julia> "Æser"[3]                                     #3
's': ASCII/Unicode U+0073 (category Ll: Letter, lowercase)
 
julia> "Æser"[4]
'e': ASCII/Unicode U+0065 (category Ll: Letter, lowercase)

See the indexing error and the next valid character index in the REPL. section 11.1

To visit characters without guessing their byte positions, iterate directly with `for ch in s`. If you need indices too, use `eachindex(s)`. A manual loop can advance from `firstindex(s)` using `nextind(s, i)` rather than adding one. section 11.1.1

Iterate over valid indices

For `s = "Þrúðvangar"`, `for i in eachindex(s); println((i, s[i])); end` visits the characters at indices 1, 3, 4, 6, 8, and onward. The gaps reflect multibyte characters. section 11.1.1

How Julia as a Second Language covers it

How deep it goes
Explains code points, UTF-8 code units, and string traversal, including an excerpt of the string-specific nextind implementation.
Examples
Compares the lengths and indices of Norwegian words and inspects code points and code-unit counts for letters and an emoji.
What it assumes
Assumes some experience with Julia strings and introduces Unicode and UTF-8 terminology from the ground up.
What you can do after
Diagnose Julia string indexing errors by distinguishing code points from UTF-8 code units and using valid traversal indices.
How it connects
Connects Unicode to ASCII, UTF-8, code points, code units, and string indexing.

Also explained in

Compare how 9 books explain Unicode →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Julia as a Second Language. The section links point to the parts of the book it draws on.