The generate function runs generation over a fine-tuned language model, extending an encoded prompt with newly generated token IDs. Its returned text combines the original input and model response, so response extraction removes the prompt before comparison or storage. section 7.7
A typical generation flow:
- Format the instruction, tokenize it, move token IDs to the selected device, and call generate with the model. section 7.7
- Bound generation with max_new_tokens, context_size, and eos_id to control length, context capacity, and termination. section 7.7
- Decode the returned token IDs, remove the original input and response marker, then strip whitespace. section 7.7
Generating test responses
For each held-out test entry, call generate with up to 256 new tokens, decode the result, extract the model response, and store it as model_response for later analysis. section 7.7
Listing 7.9 Generating test set responses
from tqdm import tqdm
for i, entry in tqdm(enumerate(test_data), total=len(test_data)):
input_text = format_input(entry)
token_ids = generate(
model=model,
idx=text_to_token_ids(input_text, tokenizer).to(device),
max_new_tokens=256,
context_size=BASE_CONFIG["context_length"],
eos_id=50256
)
generated_text = token_ids_to_text(token_ids, tokenizer)
response_text = (
generated_text[len(input_text):]
.replace("### Response:", "")
.strip()
)
test_data[i]["model_response"] = response_text
with open("instruction-data-with-response.json", "w") as file:
json.dump(test_data, file, indent=4) #1
The complete loop generates responses across the test dataset and saves the augmented records to instruction-data-with-response.json. section 7.7
