Hardware acceleration speeds generative Transformer inference by using a GPU whose compute throughput, memory capacity, and memory bandwidth determine the effective bottleneck. section 4.3.1
Evaluate acceleration operationally:
- Compare arithmetic intensity with the GPU’s ops/byte ratio to distinguish memory-bound from compute-bound workloads. section 4.3.1
- Inspect attention layers because next-token weighting over previous tokens can demand substantial computation. section 4.3.1
- Account for prefill and autoregressive sampling, including the key–value cache populated during prompt processing. section 4.3.1
Tesla T4 inference
A Tesla T4 provides 65 TFLOPS, 16 GB of memory, and 300 GB/sec of bandwidth. GPT-Neo requires 12.96 GB in half precision. A cache-enabled generation test averages 10.281 seconds versus 51.978 seconds without caching. section 4.3.1
import time
import numpy as np
prompt = "The story so far: in the beginning, the universe was created."
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
for use_cache in (True, False):
times = []
for _ in range(20):
start = time.time()
generated_ids = model.generate(input_ids,
do_sample=True,
temperature=0.9,
max_length=200,
pad_token_id=50256,
use_cache=use_cache)
times.append(time.time() - start)
print(f"{'Using' if use_cache else 'No'} KV cache: {round(np.mean(times), 3)} +- {round(np.std(times), 3)} seconds")
Benchmark generation with and without KV caching using a notebook-style timing loop. section 4.3.1
