Pagination retrieves a large result set through multiple bounded API pages. A ratings response exposes the current result page through result, limit, and offset, while total indicates whether more records remain. section 9.1.1
A practical pagination workflow:
- Start at offset 0 and request a bounded batch with limit. section 9.1.1
- Read the returned records, advance the offset by the batch size, and continue until offset reaches total. section 9.1.2
- Yield individual records instead of exposing page boundaries to downstream code. section 9.1.2
- Combine pagination with start_date and end_date filters to load smaller incremental time ranges. section 9.1.1
Lazy pagination helper
def _get_with_pagination(session, url, params, batch_size=100): offset = 0 total = None while total is None or offset < total: response = session.get( url, params={**params, "offset": offset, "limit": batch_size}, ) response.raise_for_status() data = response.json() yield from data["result"] offset += batch_size total = data["total"] This generator fetches the next page only after downstream code exhausts the current records, while the session supplies authentication and configurable API settings. section 9.1.2
Listing 9.3 Helper function for handling pagination
def _get_with_pagination(session, url, params, batch_size=100):
"""
Fetches records using a GET request with given URL/params,
taking pagination into account.
"""
offset = 0 #1
total = None #1
while total is None or offset < total: #2
response = session.get(
url,
params={
**params,
**{"offset": offset, "limit": batch_size} #3
}
)
response.raise_for_status()
response_json = response.json() #4
yield from response_json["result"] #5
offset += batch_size #6
total = response_json["total"] #6
Inspect the complete helper implementation showing offset initialization, bounded requests, response parsing, record yielding, and total-based termination. section 9.1.2
