A latent-variable model may have a posterior that cannot be sampled from directly. Stochastic variational inference (SVI) fits a tractable approximating distribution instead. A guide function defines that distribution; training adjusts its parameters to bring it closer to the posterior. When training uses randomly selected batches rather than the full dataset, memory use falls and larger datasets become practical. section 5.2.3 section 11.6.5
The training objective is the evidence lower bound (ELBO). Minimizing negative ELBO indirectly reduces reconstruction error and the KL divergence between the guide distribution and the target posterior. In Pyro, `Trace_ELBO` supplies the loss, while an optimizer updates the guide through `SVI`. section 5.2.3
A practical setup has three parts:
- Define a probabilistic model and a guide that approximates its posterior. An encoder can map observations to the location and scale of a distribution over latent variables. section 5.2.3 section 11.6.5
- Create `SVI(model, guide, optimizer, loss=Trace_ELBO())`, then call `svi.step(...)` during training. section 5.2.3 section 11.6.5
- Track loss on training data and, where available, held-out data; inspect outputs such as reconstructed images or fitted guide parameters. section 5.2.3 section 11.6.5
Choosing SVI for latent confounders
Suppose a causal model estimates unknown parameters from N observations and includes a latent proxy for confounders for each observation. The latent unknowns grow with N, making posterior inference challenging for MCMC on large datasets. SVI is used to fit an approximating posterior; it does not, by itself, establish that a causal query is identified. section 10.9 section 11.6.5
Listing 5.13 Set up the training procedure
from pyro.infer import SVI, Trace_ELBO
from pyro.optim import Adam
pyro.clear_param_store() #1
vae = VAE() #2
train_loader, test_loader = setup_dataloaders(batch_size=256) #3
svi_adam = Adam({"lr": 1.0e-3}) #4
model = vae.training_model #5
guide = vae.training_guide #5
svi = SVI(model, guide, svi_adam, loss=Trace_ELBO()) #5
Shows the compact Pyro setup: a VAE model and guide, batches of 256, an Adam optimizer, and a `Trace_ELBO` loss. section 5.2.3
Listing 11.27 Run the training loop
from pyro.infer import SVI, Trace_ELBO
from pyro.optim import Adam
from pyro import condition
pyro.clear_param_store() #1
adam_params = {"lr": 0.0001, "betas": (0.90, 0.999)} #2
optimizer = Adam(adam_params) #2
training_model = condition(data_model, data) #3
svi = SVI(training_model, guide, optimizer, loss=Trace_ELBO()) #4
elbo_values = [] #5
N = len(data['In-game Purchases']) #5
for step in range(500_000): #5
loss = svi.step(data) / N #5
elbo_values.append(loss) #5
if step % 500 == 0: #5
print(loss) #5
Shows SVI in a causal data-model workflow, including conditioning on observations, optimization steps, and recording normalized loss. section 11.6.5
