liveBook · Manning

Extended character set

As explained in Modern C, Third Edition

An extended character set is a repertoire that includes characters beyond the limited C basic character set, such as characters from additional scripts and writing systems.

A C string can contain characters beyond the basic character set. Before printing a multibyte string such as "fär ínóff", call `setlocale(LC_ALL, "")` so character handling uses the installed locale. The locale matters: placing extended characters in a string alone is not enough to make their output work reliably. section 14.3

 setlocale(LC_ALL, "");
   /* Multibyte character printing only works after the locale
      has been switched. */
   draw_sep(TOPLEFT " © 2014 j𝜀nz ' g𝜔z,t𝜀t ", TOPRIGHT);

The listing switches the locale before printing text containing extended characters. section 14.3

A multibyte character is a sequence of bytes interpreted as one character of an extended character set. A multibyte string contains such sequences. Its characters contain no null bytes, and the string is null-terminated, so ordinary string operations such as `strcpy` can still work. TAKEAWAY: `strlen` counts bytes, not necessarily the characters that will print; neither it nor the size of a `char` array directly gives the printed length. section 14.3

To parse characters rather than copy bytes, keep conversion state in an `mbstate_t` from `<wchar.h>`. The library provides `mbsrtowcs` for conversion to `wchar_t` characters and `mbrtowc` for conversion of a multibyte character. The example `mbsrlen` uses `mbsrtowcs` to determine a converted length; it is an ad hoc function, not a standard-library function. Parsing must account for conversion status rather than assume that every byte begins a complete character. section 14.3

 size_t mbsrlen(char const*s, mbstate_t const*restrict state) {
   state = state ? state : MBSTATE;
   mbstate_t st = *state;
   size_t mblen = mbsrtowcs(nullptr, &s, 0, &st);
   if (mblen == mbinvalid) errno = 0;
   return mblen;
 }

This implementation copies the parsing state before measuring the multibyte string with `mbsrtowcs`. section 14.3

TAKEAWAY: Prefer multibyte strings for extended-character text input and output, but distinguish byte-oriented string operations from character-oriented parsing and measurement. Unicode (ISO 10646) supplies a framework for character codes; it does not remove the need to handle multibyte conversion and its state. section 14.3 section 14.7

How Modern C, Third Edition covers it

How deep it goes
Defines multibyte characters and strings, then works through length measurement and stateful conversion; does not treat byte counts as character counts.
Examples
Shows locale-dependent printing of extended characters, box-drawing string literals, and an `mbsrtowcs`-based length function.
What it assumes
Assumes familiarity with pointers and ordinary C strings while defining multibyte characters and introducing conversion state.
What you can do after
Apply it via C library locale and conversion functions to print multibyte text and distinguish byte lengths from converted lengths.
How it connects
Connects extended character sets to locales, null-terminated strings, Unicode, wide characters, and conversion state.

Also explained in

Compare how 2 books explain extended character set →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Modern C, Third Edition. The section links point to the parts of the book it draws on.