liveBook · Manning

NEFTune

As explained in Building Specialized AI Systems

NEFT is a fine-tuning method that adds controlled noise to input embeddings during training to improve generalization.

NEFTune (Noisy Embeddings Improve Instruction Finetuning) improves instruction fine-tuning by adding controlled uniform noise to input embeddings during training. The perturbation regularizes learning, reducing overfitting and encouraging more robust, generalizable representations without changing the parameter-update strategy used by methods such as LoRA. section 3.5.4 section 3.8

Apply NEFTune through these implementation rules:

  • Add a random noise vector after tokens become embeddings and before the embeddings pass through the transformer. section 3.5.4
  • Scale noise magnitude with α/√(L×d), using the noise alpha, sequence length, and embedding dimension. section 3.5.4
  • Enable noise only during training; leave inference embeddings unperturbed for clean, deterministic outputs. section 3.5.4
Listing 3.33 Configuring the trainer to use NEFT
trainer = SFTTrainer(
    # Other training parameters here
    neftune_noise_alpha=5
)

Configure NEFTune in SFTTrainer with the neftune_noise_alpha parameter. section 3.5.4

Instruction-following improvement

Standard fine-tuning of LLaMA-2-7B on Alpaca reached 29.79% on AlpacaEval; NEFTune reached 64.69%, an improvement of nearly 35 percentage points. section 3.5.4

How Building Specialized AI Systems covers it

How deep it goes
Defines embedding-level noise injection, scaling, and training-only application, but does not detail its implementation beyond configuration.
Examples
Uses LLaMA-2-7B fine-tuned on Alpaca and evaluated with AlpacaEval.
What it assumes
Assumes familiarity with fine-tuning, embeddings, transformer layers, sequence length, and embedding dimensions.
What you can do after
Apply it via SFTTrainer by setting neftune_noise_alpha to configure training-time embedding noise.
How it connects
Contrasts with LoRA, QLoRA, and DoRA while complementing broader fine-tuning and regularization choices.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.