liveBook · Manning

ReCode

As explained in Domain-Specific Small Language Models

An evaluation approach for code generation models that measures robustness under prompt perturbations.

ReCode evaluates code-generation models by measuring robustness under prompt perturbations, using worst-case behavior as the signal for prompt sensitivity and production usefulness. section 7.2.3 section 7.4

It is applied here to Python code generation with HumanEval, alongside syntactic validation and unit tests for functional correctness. section 7.2.3 section 7.4

Key pieces used in the evaluation path are:

  • HumanEval prompts with function signatures, docstrings, and tests. section 7.2.3
  • Generation settings for bounded output length and cached decoding. section 7.2.3
  • Python-package setup for running the evaluation locally. section 7.2.3

Prompt-robustness evaluation

A model is run against HumanEval tasks, then checked for how much performance shifts when prompts are perturbed, so the result reflects worst-case robustness rather than a single prompt outcome. section 7.2.3 section 7.4

How Domain-Specific Small Language Models covers it

How deep it goes
Defines robustness metrics and shows an end-to-end Python evaluation workflow, but does not derive the metric mathematics.
Examples
Uses HumanEval tasks, prompt perturbations, and a Colab notebook workflow with package installation and dataset download.
What it assumes
Assumes Python code-generation evaluation basics and introduces HumanEval and prompt perturbation handling.
What you can do after
Apply it to evaluate code-generation robustness under prompt changes.
How it connects
Connects to HumanEval, code generation, syntactic validation, and unit tests.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.