liveBook · Manning

TorchScript

As explained in Designing Deep Learning Systems: A guide for software engineers

TorchScript is a serializable and optimizable intermediate representation of PyTorch programs that can execute independently of Python.

TorchScript packages PyTorch programs as a serializable, optimizable intermediate representation that can execute independently of Python, including in standalone C++ applications. Triton requires PyTorch models in this format for serving. section 7.4.3

A practical TorchScript serving workflow is:

  • Place the model in Triton’s model repository alongside config.pbtxt, register it through the management API, and send inference requests. section 7.4.3

Tracing a PyTorch model

After defining a model, call model.eval(), create an example tensor such as torch.rand(1, 3, 224, 224), trace with torch.jit.trace, and save traced_torch_model.pt.

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes
Defines serialization and optimization, explains Python-independent execution, and demonstrates tracing and saving without detailing compiler internals.
Examples
Uses evaluation mode, a random image-shaped input, torch.jit.trace, a saved .pt file, and a Triton config.pbtxt.
What it assumes
Assumes a PyTorch model with a forward() method and teaches serialization through an example input and evaluation-mode workflow.
What you can do after
Build it from scratch by tracing a PyTorch model into a saved TorchScript file for Triton serving.
How it connects
Connects PyTorch model serialization with Triton configuration, model repositories, management APIs, and inference APIs.
Cover of Designing Deep Learning Systems: A guide for software engineers
liveBookmanning.com
Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.