Throughput measures how many queries a system processes per unit of time, often expressed as queries per second. It helps assess whether a RAG system can handle high query volumes, but does not by itself measure retrieval accuracy. section 6.5.1 section 6.5.4
Higher accuracy and relevance can come with lower throughput when components require more computation. Compare throughput with latency, response quality, and resource use rather than optimizing it alone. section 6.5.4
Table 6.7 Evaluation of PharmaBot with various configurations
|
Metrics
|
Configuration 1
|
Configuration 2
|
Configuration 3
|
|
|---|---|---|---|---|
|
Response accuracy
|
Recall
|
0.85
|
0.87
|
0.88
|
|
Precision
|
0.75
|
0.77
|
0.78
|
|
|
F1 score
|
0.8
|
0.82
|
0.83
|
|
|
Relevance
|
Mean reciprocal rank (MMR)
|
0.70
|
0.72
|
0.74
|
|
Average precision
|
0.65
|
0.68
|
0.70
|
|
|
Latency
|
Average latency (ms)
|
50
|
55
|
60
|
|
Throughput
|
Queries/second
|
20
|
18
|
16
|
|
Resource utilization
|
CPU usage (%)
|
70
|
65
|
90
|
|
Memory usage
|
8
|
7
|
12
|
Compare PharmaBot configurations: their throughput ranges from 20 to 16 queries per second as accuracy and relevance rise, with latency and resource use also varying. section 6.5.4
