A token generation loop produces a response incrementally instead of waiting for the complete sequence. Each pass asks the model for one new token, decodes it, and places it in the requesting client’s result queue. section 17.1.4
A batched implementation repeats these steps:
- Generate with max_new_tokens=1 using the batched tokens and attention mask. section 17.1.4
- Extract and decode the newest token for each active request. section 17.1.4
- Place each decoded token in its request queue under a results lock. section 17.1.4
- Stop when the maximum token count is reached or no requests remain. section 17.1.4
Streaming an answer
A FastAPI /generate endpoint assigns a request ID, queues the prompt, and returns a StreamingResponse. An asynchronous stream_results generator yields tokens as they become available. section 17.1.4
for _ in range(MAX_TOKENS):
if not active_requests:
break # All requests are done
with torch.no_grad():
outputs = model.generate(
batch_tokens,
attention_mask=batch_attention_mask,
max_new_tokens=1,
pad_token_id=tokenizer.pad_token_id,
)
generated_token_ids = [output[-1:] for output in outputs]
for idx in list(active_requests):
request_id, _ = batch[idx]
new_token_id = generated_token_ids[idx]
new_token = tokenizer.decode(new_token_id)
...
with results_lock:
if request_id in results:
results[request_id].put(new_token)
Inspect the loop that generates one token, decodes it, and queues it for the matching request. section 17.1.4
