liveBook · Manning

Semantic adaptation

As explained in Rearchitecting LLMs

Semantic adaptation is model adaptation that changes how a model interprets or generates meaning rather than only enforcing a fixed output form.

Semantic adaptation changes what a model means or reasons about, rather than only changing how it expresses an answer. Common-sense reasoning, visual understanding, and multimodal instruction are examples of tasks that require it. section 7.4

DoRA separates magnitude and direction during training, a design that may help with deep semantic adaptation. By contrast, structured extraction mainly changes output form; QLoRA is the faster pragmatic choice for that task, while QDoRA’s semantic-task advantage remains a hypothesis to test. section 7.4 section 7.3.5

Testing for a semantic-task advantage

Replace a clinical extraction dataset with a reasoning dataset such as a subset of databricks-dolly-15k, keep the notebook’s code and hyperparameters unchanged, and compare QLoRA with QDoRA for general-capability retention and training stability. section 7.5

How Rearchitecting LLMs covers it

How deep it goes
Contrasts semantic and structural adaptation and describes DoRA’s proposed benefit, but leaves its semantic-task advantage for testing.
Examples
Uses common-sense reasoning, visual understanding, multimodal instruction, and a proposed Dolly reasoning task as examples.
What you can do after
Evaluate whether a task requires semantic variation when weighing QLoRA against QDoRA.
How it connects
Connects semantic adaptation to DoRA and compares it with LoRA, QLoRA, and QDoRA.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.