How can a causal model generate an image when the effects of its causes are too complex for a simple conditional probability table? A neural decoder supplies a nonlinear mapping from causes to an output. In a variational autoencoder (VAE), it takes a latent encoding Z and observed labels, then produces parameters for an image distribution. It acts as a model of the image’s causal Markov kernel—not as proof that Z captures the image’s actual unobserved causes. section 5.2.2
Encoding runs in the opposite direction: an encoder uses an image and its labels to infer a distribution over Z. Decoding uses a sampled Z and the labels to reconstruct an image or generate a new one. The labels specify modeled causes; Z stands in for variation not explicitly modeled, such as differences among writers. section 5.2.2 section 5.2.3
Listing 5.4 Implement the decoder
from torch import nn
class Decoder(nn.Module): #1
def __init__(self, z_dim, hidden_dim):
super().__init__()
img_dim = 28 * 28 #2
digit_dim = 10 #3
is_handwritten_dim = 1 #4
self.softplus = nn.Softplus() #5
self.sigmoid = nn.Sigmoid() #5
encoding_dim = z_dim + digit_dim + is_handwritten_dim #6
self.fc1 = nn.Linear(encoding_dim, hidden_dim) #6
self.fc2 = nn.Linear(hidden_dim, img_dim) #7
def forward(self, z, digit, is_handwritten): #8
input = torch.cat([z, digit, is_handwritten], dim=1) #9
hidden = self.softplus(self.fc1(input)) #10
img_param = self.sigmoid(self.fc2(hidden)) #11
return img_param
The PyTorch implementation concatenates Z, a ten-component digit label, and a handwritten indicator. A hidden layer and sigmoid output yield 784 image parameters for a 28 × 28 image. section 5.2.2
A decoder’s output has different jobs in the workflow:
- During training, the encoder’s approximate posterior supplies Z; the decoder’s output parameterizes the image likelihood. Stochastic variational inference trains both networks using negative ELBO, rather than reconstruction error alone. section 5.2.3
- During evaluation, compare an original image with its reconstruction, inspect newly generated images, and track test loss. These checks reveal whether image quality improves during training. section 5.2.4
- In a VAE-inspired causal inference model, separate decoders map a latent proxy and other inputs to distribution parameters for engagement and purchases; the decoder need not output an image. section 11.6.4
Changing a shape while retaining inferred variation
For a dSprites image, an encoder infers an image-specific exogenous representation from the image and its causal factors. A pretrained decoder maps that representation plus factors such as shape back toward an image. Counterfactual generation changes the shape factor to a heart and uses the decoder within the image assignment function. This is a proof of concept: the factors are independent and the images simple, so broader counterfactual use requires additional assumptions. section 9.4.1
Listing 9.29 Create an exogenous distribution and assignment function for the image
def p_n_image(n_image_params): #1
n_image_loc, n_image_scale, n_unif_upper = n_image_params #2
n_image_norm = dist.Normal( #3
n_image_loc, n_image_scale #3
).to_event(1).sample() #3
n_image_unif = dist.Uniform(0, n_unif_upper).expand( #4
torch.Size([1, 64*64]) #4
).sample() #4
n_image = n_image_norm, n_image_unif #5
return n_image
def f_image(factor, n_image): #6
n_image_norm, n_image_unif = n_image #7
p_output = decoder.forward( #8
n_image_norm, #8
encode_one_hot(factor) #8
)
sim_img = (n_image_unif <= p_output).int() #9
return sim_img
The assignment function shows a useful distinction: the decoder produces image probabilities, while additional sampled noise turns them into simulated pixel values. section 9.4.1
