Causal llm

As explained in Causal AI

A causal LLM is a large language model embedded in a causal graphical-model structure so that its conditional text distributions represent causal mechanisms.

A causal LLM embeds language generators in a causal DAG, making each conditional text distribution correspond to a causal mechanism rather than only a predictive association. Parent outputs are concatenated as inputs to downstream models. section 13.3.2

A practical construction uses separate models for each variable:

• Fine-tune a GPT-2 causal language model to generate the King variable. section 13.3.2
• Fine-tune BART sequence-to-sequence models for Prince given King and Kingdom given King plus Prince. section 13.3.2
• Wrap generation and log-probability computation in a Torch distribution, then connect the distributions with Pyro sample sites. section 13.3.2

Generated story

A generated King declares war, a Prince abdicates, and the Kingdom falls into poverty. Observational and interventional distributions can then be compared to test the causal model. section 13.3.2

Figure 13.5 GPT-2 is fine-tuned to represent the distribution of King’s action vignettes.

Use the architecture figure to distinguish the pretrained language model from the causal-variable scaffold. section 13.3.2

How Causal AI covers it

How deep it goes

Descends through tokenization, fine-tuning, generation, log probabilities, PyTorch distributions, and Pyro integration.

Examples

Uses King, Prince, and Kingdom action vignettes, including war, abdication, resource depletion, and generated full stories.

What you can do after

Build it from scratch with Hugging Face Transformers, PyTorch, and Pyro, then generate stories and compare causal distributions.

How it connects

Connects causal DAGs, causal Markov kernels, Transformer models, PyTorch distributions, Pyro, and observational-interventional comparison.

liveBookmanning.com

Read Causal AI on liveBook Get the book at manning.com

Related concepts

• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Causal AI. The section links point to the parts of the book it draws on.