liveBook · Manning

Causal language model

As explained in Deep Learning with R, Third Edition

A causal language model is a language model that predicts tokens sequentially from preceding context.

A causal language model predicts the next token from the tokens that precede it. Repeating this step generates a sequence, making the model an autocomplete system for text. section 16.3.1

A high-level CausalLM combines a tokenizer, preprocessing, and a pretrained backbone. KerasHub can load a matching architecture and tokenizer, while generate() handles autoregressive text generation. section 16.3.1

Generating text with Gemma

Instantiate Gemma, choose greedy sampling, and continue a prompt: gemma_lm$generate("A piece of advice", max_length = 40L). Prompts can produce plausible continuations, such as brownie instructions, but unrestricted pretraining mainly produces likely text rather than reliably useful answers.

Instruction fine-tuning adapts the same next-token mechanism to prompts and responses. A template marks [instruction], [end], and [response]; training computes loss mainly on response tokens, excluding the fixed prompt and padding. section 16.3.2

preprocessor <- gemma_lm$preprocessor
preprocessor$sequence_length <- 512L
batch <- iter_next(as_iterator(train_ds))
.[x, y, sample_weight] <- preprocessor(batch)
str(x)

List of 2

 $ token_ids   : <jax.Array shape(2, 512), dtype = int32>

 $ padding_mask: <jax.Array shape(2, 512), dtype = bool>

str(y)

<jax.Array shape(2, 512), dtype = int32>

str(sample_weight)

<jax.Array shape(2, 512), dtype = bool>

Inspect the preprocessor outputs to see token IDs, next-token labels, padding masks, and response-focused sample weights. section 16.3.2

How Deep Learning with R, Third Edition covers it

How deep it goes
Explains generation, tokenizer-model packaging, prompt templates, padding masks, labels, and response-weighted loss without detailing model internals.
Examples
Demonstrates Gemma generation with advice, brownie, and factual prompts, plus a 15,000-example instruction dataset and tokenized training batch.
What it assumes
Assumes familiarity with prompts and datasets while teaching tokenization, next-token labels, padding masks, and causal preprocessing explicitly.
What you can do after
Apply it via KerasHub to load Gemma, generate continuations, and prepare response-masked batches for instruction fine-tuning.
How it connects
Connects tokenization, pretrained language models, text generation, sampling strategies, instruction fine-tuning, padding masks, and sequence-to-sequence learning.
Cover of Deep Learning with R, Third Edition
liveBookmanning.com
Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 3 books explain causal language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.