asyncio.create_subprocess_exec asynchronously starts a program directly, without invoking a shell. It returns a Process object that exposes the process identifier, waiting, termination, and related controls. This is useful when an existing command-line application can provide functionality more efficiently than reimplementing it in Python. section 13.1
Use the API with these practical rules:
- Pass the executable and each argument separately, such as create_subprocess_exec('ls', '-l'); prefer this over create_subprocess_shell unless shell functionality is required. section 13.1 section 13.3
- Await process.wait() to obtain the exit status, and use process.terminate() after a timeout when a subprocess does not finish promptly. section 13.1
- Set stdout or stderr to asyncio.subprocess.PIPE when output needs processing; consume piped output because unconsumed output can deadlock the application. section 13.1.1 section 13.3
- Pipe stdin and use communicate(input_bytes) to send data, collect output, and wait for completion. section 13.1.2 section 13.3
- Limit large batches of concurrent subprocesses with an asyncio semaphore to reduce resource contention. section 13.1.2 section 13.3
Run a command and wait for completion
An asyncio program can launch ls -l, print its process ID, await completion, and report the returned status code. On Windows, the equivalent example uses cmd /c dir. section 13.1
Listing 13.3 Using the standard output stream reader
import asyncio
from asyncio import StreamReader
from asyncio.subprocess import Process
async def write_output(prefix: str, stdout: StreamReader):
while line := await stdout.readline():
print(f'[{prefix}]: {line.rstrip().decode()}')
async def main():
program = ['ls', '-la']
process: Process = await asyncio.create_subprocess_exec(*program,
stdout=asyncio
.subprocess.PIPE)
print(f'Process pid is: {process.pid}')
stdout_task = asyncio.create_task(write_output(' '.join(program), process.stdout))
return_code, _ = await asyncio.gather(process.wait(), stdout_task)
print(f'Process returned: {return_code}')
asyncio.run(main())
Shows how to pipe a subprocess's output into a StreamReader, prefix each line, and consume output concurrently with process.wait(). section 13.1.1
Listing 13.6 Using communicate
import asyncio
from asyncio.subprocess import Process
async def main():
program = ['python3', 'listing_13_4.py']
process: Process = await asyncio.create_subprocess_exec(*program,
stdout=asyncio
.subprocess.PIPE)
print(f'Process pid is: {process.pid}')
stdout, stderr = await process.communicate()
print(stdout)
print(stderr)
print(f'Process returned: {process.returncode}')
asyncio.run(main())
Demonstrates communicate() collecting stdout and stderr from a subprocess whose output is piped. section 13.1.1
