liveBook · Manning

Invalid byte sequence

As explained in PostgreSQL Mistakes and How to Avoid Them

An invalid byte sequence is a sequence of encoded bytes that cannot be interpreted as a valid instance of the intended character encoding.

An invalid byte sequence is encoded data that PostgreSQL cannot interpret as valid UTF-8. Migration failures commonly arise when source and target encodings differ or source data contains illegal characters. section 10.3

A migration that fails in PostgreSQL

Loading a MySQL dump containing a latin1 byte such as 0xe9, or an ASCII NUL byte 0x00, can produce an invalid-byte-sequence error. PostgreSQL strictly validates UTF-8 and rejects NUL characters in string constants, even though another database may accept them. section 10.3

Handle the failure without silently damaging the data:

  • Treat unexamined conversion as a data-integrity risk: accepted bytes may later decode incorrectly and become unrecoverable. section 10.3

How PostgreSQL Mistakes and How to Avoid Them covers it

How deep it goes
Explains rejected UTF-8 bytes and ASCII NULs, demonstrates diagnostics and cleanup, but does not detail PostgreSQL decoder internals.
Examples
Uses MySQL dumps, 0xe9 and 0x00 diagnostics, iconv conversion, and shell commands that remove NUL characters.
What it assumes
Assumes database and migration familiarity while establishing the need to identify source and target encodings explicitly.
What you can do after
Diagnose migration errors, convert source encodings, clean illegal bytes, and assess potential data loss before importing.
How it connects
Connects character encoding, UTF-8, data migration, PostgreSQL, MySQL, and data corruption.
Cover of PostgreSQL Mistakes and How to Avoid Them
liveBookmanning.com
Read PostgreSQL Mistakes and How to Avoid Them on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of PostgreSQL Mistakes and How to Avoid Them. The section links point to the parts of the book it draws on.