liveBook · Manning

Character-based modeling

As explained in LLMs in Production

Character-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.

Character-based modeling represents language as sequences of individual alphabetic characters. It is easy to implement, but character-level inputs often lack enough information to produce statistically significant language-modeling results. section 5.2.1

Character-based modeling is mainly useful as a contrast with broader tokenization strategies:

  • Word-, subword-, or sentence-level tokenization captures a larger context window and more linguistic information than individual characters. section 5.2.1
  • Increasing tuning alone does not solve the fundamental weakness that alphabets and orthography do not intrinsically encode meaning. section 5.2.1
  • Pretraining before exposing a model to its task can improve the approximate representations available for downstream performance. section 5.2.1

Training from scratch

A GPT-style training example assembles embeddings, transformer blocks, normalization, and an output head before running batched epochs. Its character-based language-modeling example illustrates why character sequences are a weak foundation for generating language. section 5.2.1

How LLMs in Production covers it

How deep it goes
Evaluates the representation’s linguistic limitation and contrasts it with word-, subword-, and sentence-level tokenization.
Examples
Uses a from-scratch GPT training architecture and training loop to illustrate character-based modeling’s weak results.
What you can do after
Evaluate character-level representations against broader tokenization choices for language-modeling systems.
How it connects
Connects character-based modeling to language modeling, tokenization, pretraining, and GPT architectures.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.