liveBook · Manning

Code point

As explained in Julia as a Second Language

A code point is a numeric value assigned to an abstract character or other element in a coded character set.

The letter `A` is easy to store and read. Text gets trickier when it includes characters from many languages or symbols such as emoji. Unicode assigns each character a number called a code point, so the character has an identity independent of how it is stored. section 11.0 section 11.1.1

A code point is not the same as a code unit. A code point identifies a character; code units store it. Julia strings use UTF-8, which takes a variable number of code units per code point. ASCII characters fit in one byte, while other characters can take more. That distinction matters when examining storage or moving through a string. section 11.1.1 section 11.5

Check a character in Julia

At the REPL, `Int(codepoint('A'))` returns `65`, while `ncodeunits('A')` returns `1`. For `'Æ'`, `codepoint('Æ')` returns `0x000000c6` and `ncodeunits('Æ')` returns `2`. The first call asks which character it is; the second asks how much UTF-8 storage it needs. section 11.1.1

julia> codepoint('Æ')
0x000000c6
 
julia> ncodeunits('Æ')   #1
2
 
julia> isascii('Æ')      #2
false
 
julia> codepoint('😏')   #3
0x0001f60f
 
julia> ncodeunits('😏')  #4
4
 
julia> isascii('😏')
false

Compare `A`, `Æ`, and an emoji: their code points identify them, but their UTF-8 code-unit counts differ. section 11.1.1

Listing 11.1 Iterating over characters in a string using a while loop
julia> s = "Þrúðvangar"
 
julia> i = firstindex(s)
1
 
julia> while i <= lastindex(s)
           println((i, s[i]))
           i = nextind(s, i)
       end
(1, 'Þ')
(3, 'r')      #1
(4, 'ú')
(6, 'ð')      #2
(8, 'v')
(9, 'a')
(10, 'n')
(11, 'g')
(12, 'a')
(13, 'r')

The multilingual string’s printed indices skip positions because moving to the next character can cross multiple code units. section 11.1.1

How Julia as a Second Language covers it

How deep it goes
Works through numeric identifiers, variable-width storage, and string iteration; shows only an excerpt of `nextind` internals.
Examples
Demonstrates `A`, `Æ`, and an emoji in REPL calls, then iterates a multilingual string with skipped indices.
What it assumes
Assumes basic familiarity with Julia strings and REPL calls while introducing Unicode through ASCII examples.
What you can do after
Apply Julia’s `codepoint`, `ncodeunits`, and `nextind` to distinguish character identity from storage and traverse strings.
How it connects
Connects Unicode code points to ASCII, UTF-8, code units, `Char`, and string indexing.

Also explained in

Compare how 6 books explain code point →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Julia as a Second Language. The section links point to the parts of the book it draws on.