liveBook · Manning

Encoding

As explained in PostgreSQL Mistakes and How to Avoid Them

A character encoding maps textual characters to coded representations such as bytes for storage and processing.

Character encoding maps textual characters to coded representations such as bytes. During a MySQL or MariaDB migration, assuming standard SQL compatibility can hide incompatible encodings and corrupt imports. section 10.3

Diagnosing an encoding mismatch

A PostgreSQL import can fail with `invalid byte sequence for encoding "UTF8": 0xe9` when a dump retains MySQL's latin1 bytes for é. Convert the dump with `iconv -f latin1 -t utf8` before retrying the import. section 10.3

Encoding problems can continue after byte conversion:

  • MySQL's historical `utf8` supports only three-byte characters, whereas `utf8mb4` supports four-byte Unicode characters, including characters the older implementation cannot encode. section 10.3

How PostgreSQL Mistakes and How to Avoid Them covers it

How deep it goes
Explains byte-level failures, legacy MySQL encodings, conversion, null-byte cleanup, and locale configuration.
Examples
Uses international names, the character é, invalid UTF8 errors, iconv conversion, and null-byte removal commands.
What you can do after
Diagnose import failures, convert latin1 dumps to UTF-8, remove null bytes, and verify PostgreSQL encoding settings.
How it connects
Connects character encoding with Unicode, UTF-8, latin1, utf8mb4, database migration, and PostgreSQL imports.

Also explained in

Compare how 9 books explain encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of PostgreSQL Mistakes and How to Avoid Them. The section links point to the parts of the book it draws on.