Ops/byte ratio measures floating-point operations performed for each byte transferred from memory. Comparing this ratio with a GPU’s threshold indicates whether inference is memory bound or compute bound. section 4.3.1
Use the ratio to connect hardware characteristics with model behavior:
- A T4 provides 65 TFLOPS and 300 GB/sec bandwidth, producing a threshold of 216.6 ops/byte. section 4.3.1
- Below 216.6 ops/byte, memory movement limits performance; above it, available compute limits performance. section 4.3.1
- Compare this hardware threshold with arithmetic intensity, computed as operations divided by bytes accessed, then inspect demanding layers such as attention. section 4.3.1
Attention arithmetic intensity
For a cached attention computation, explicit memory-movement and compute counts yield 71,303,168 bytes, 2,160,066,560 operations, and 30.29 ops/byte.
