liveBook · Manning

Mamba

As explained in LLMs in Production

Mamba is a sequence-modeling architecture based on selective state-space mechanisms and designed to avoid standard attention costs.

Mamba is an attention-free neural architecture that improves state space models (SSMs) for sequence modeling. It targets the quadratic cost of standard attention as context grows. section 12.2.2

Its practical appeal is reduced long-context cost, but performance may depend on specialized hardware optimization. JAMBA extends the idea with hybrid SSM–transformer layers and joint attention. section 12.2.2

JAMBA as a practical hybrid

A 52-billion-parameter JAMBA mixture-of-experts model supports 140K-token contexts on an 80 GB GPU. Training requires multiple GPUs, while inference uses Hugging Face Transformers, quantization, and generation. section 12.2.2

Listing 12.1 Finetuning and inferencing JAMBA 
from trl import SFTTrainer
from peft import LoraConfig
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
)
from transformers import BitsAndBytesConfig
import torch
from datasets import load_dataset

tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1")
model = AutoModelForCausalLM.from_pretrained(
    "ai21labs/Jamba-v0.1", device_map="auto"
)

dataset = load_dataset("Abirate/english_quotes", split="train")

Inspect tokenizer loading, model loading, dataset preparation, and the starting point for fine-tuning a Mamba-style JAMBA model. section 12.2.2

How LLMs in Production covers it

How deep it goes
Defines Mamba as an improved SSM architecture and notes hardware constraints, but defers its internal mechanism.
Examples
Uses JAMBA fine-tuning, 140K-token inference, quantization, and multi-GPU training as concrete examples.
What it assumes
Assumes familiarity with attention and neural sequence models while identifying state space models as the architectural foundation.
What you can do after
Evaluate Mamba-style alternatives against attention models for long-context cost and hardware requirements.
How it connects
Connects Mamba with state space models, transformers, attention, and hybrid architectures.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.