liveBook · Manning

Character encoding

As explained in The Well-Grounded Rubyist, Fourth Edition

An encoding is a scheme that maps sequences of bytes to textual characters and determines their representation.

Character encoding determines how a sequence of bytes maps to textual characters. Because every Ruby string consists of bytes, its encoding determines how many bytes represent each character and which characters those bytes denote. section 8.1.6

Ruby exposes encoding at several boundaries:

  • Ruby source files default to UTF-8; __ENCODING__ reports the source encoding, while a magic comment such as # encoding: ASCII changes it. section 8.1.7
  • Individual strings report their encoding through encoding and can be transcoded with encode or encode!, provided the conversion is permitted. section 8.1.7
  • force_encoding bypasses permitted-conversion checks and labels the existing bytes with the specified encoding; it does not perform the same conversion as encode. section 8.1.7
  • File input can be read character-wise with getc or byte-wise with getbyte; these operations may return different results when characters occupy multiple bytes. section 12.2.3

Transcoding versus relabeling

A string such as str = "Test string" reports UTF-8 through str.encoding. Calling str.encode("US-ASCII") produces a converted string, while str.encode!("US-ASCII") changes str itself. force_encoding is different: it assigns an encoding to the current byte sequence, which can affect how later characters are interpreted. section 8.1.7

>> str = "Test string"
=> "Test string"
>> str.encoding
=> #<Encoding:UTF-8>

Inspect a Ruby string's encoding directly before experimenting with transcoding or byte-level operations. section 8.1.7

How The Well-Grounded Rubyist, Fourth Edition covers it

How deep it goes
Defines byte-to-character mapping and demonstrates source settings, transcoding, forced labels, and stream-level differences without exploring encoding deeply.
Examples
Uses Ruby source comments, String#encoding, encode, encode!, force_encoding, getc, and getbyte as concrete examples.
What you can do after
Apply Ruby encoding methods to inspect, transcode, relabel, and read strings at character or byte boundaries.
How it connects
Connects strings, bytes, UTF-8, transcoding, source files, and file input operations.

Also explained in

Compare how 5 books explain character encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of The Well-Grounded Rubyist, Fourth Edition. The section links point to the parts of the book it draws on.