Text gets tricky as soon as it extends beyond A–Z. Unicode assigns a number, called a code point, to characters across writing systems. UTF-8 is a way to encode those code points as bytes. Julia strings use Unicode text encoded in UTF-8. section 11.0 section 11.1 section 11.1.1
UTF-8 uses one to four bytes for a character. Familiar ASCII letters take one byte; characters such as Æ can take more. A code point identifies the character, while code units represent it in the encoding. That distinction matters when measuring and indexing strings. section 11.1 section 11.1.1
julia> sizeof("Æser") #1
5
julia> sizeof("Þrúðvangr") #2
12
Compare length with sizeof: "Æser" has four characters but occupies five bytes; "Þrúðvangr" has nine characters but occupies twelve bytes. section 11.1
julia> codepoint('Æ')
0x000000c6
julia> ncodeunits('Æ') #1
2
julia> isascii('Æ') #2
false
julia> codepoint('😏') #3
0x0001f60f
julia> ncodeunits('😏') #4
4
julia> isascii('😏')
false
Inspect a character with codepoint and ncodeunits. The examples show that 'Æ' and '😏' have distinct code points and require two and four UTF-8 code units, respectively. section 11.1.1
String indices need not advance one position per character. In "Æser", index 2 raises a StringIndexError because it falls inside the encoding of Æ; the next character begins at index 3. To visit characters without guessing their indices, iterate directly with for, or use eachindex when the indices matter. section 11.1 section 11.1.1
julia> for i in eachindex(s)
print(i, s[i], " ")
end
1Þ 3r 4ú 6ð 8v 9a 10n 11g 12a 13r
The eachindex loop prints each valid index beside its character, making the gaps after multibyte characters visible. section 11.1.1
