LM calls often spend time waiting for a response. Parallelism lets independent calls overlap instead of waiting for each to finish before starting the next. In DSPy, this can mean running multiple programs at once or calling multiple submodules or tools within one program. Start with synchronous code, then add parallelism where measured delays justify the extra complexity. section 7.5 section 4.3.2
Two practical routes serve different needs:
- For evaluation, set `num_threads` on `dspy.Evaluate` to run work across examples using its multithreaded framework. section 4.3.2
- For independent module calls, use Python’s `asyncio`: call each module with `acall()` and await the grouped tasks with `asyncio.gather()`. A custom module that supports this route defines an async `aforward()` and awaits its submodule calls; it can also retain a synchronous `forward()`. section 7.5.2
Listing 7.18 Executing Predict multiple times in parallel
import dspy
import asyncio
import os
os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))
predict = dspy.Predict("question->answer")
async def main():
task_list = asyncio.gather(predict.acall(question="What is the capital
of Finland?"),
predict.acall(question="What is the capital
of Norway?"))
answer1, answer2 = await task_list #1
print(answer1)
print(answer2)
asyncio.run(main())
The listing groups two independent `dspy.Predict.acall()` requests for country capitals and awaits both results. section 7.5.2
Split, run, combine
A Skeleton-of-Thought module first splits a question into independent subtasks, starts an `asyncio` task for each answer, awaits the answers, then combines them. It uses `create_task()` because the number of subtasks is not known until the first module responds. section 7.5.4
Concurrency can shorten elapsed time, but it can also hit provider rate limits. The benefit depends on the workload and service: a single-threaded evaluation with deliberate sleeps took longer largely because it lacked parallelization. Async code also complicates debugging and error handling, and may need care in notebook environments. section 7.5.2 section 4.7 section 7.5
