A spatio-temporal volume is a three-dimensional tensor representing video across two spatial dimensions and time. Each cross-section is a frame: a discrete height-by-width pixel array sampled at one point in the evolving scene. Visualize the video as a cuboid whose axes are time, height, and width. section 10.4
The volume matters because:
- Analyzing frames independently discards motion information; successive frames reveal changes that a single image cannot determine, such as whether a half-open door is opening or closing. section 10.4
- Three-dimensional convolution extracts local patterns across space and time. A kernel acts like a small brick sliding through the input volume, producing one output value at each slide stop and an output spatio-temporal volume overall. section 10.4
Detecting motion with a 3D kernel
A 2 × 3 × 3 kernel compares two successive frames by averaging each 3 × 3 spatial neighborhood and subtracting one average from the other. The response is high where motion changes pixel colors and low where the scene is static. For a uniformly colored moving ball, the detector responds mainly at the boundary, not the indistinguishable center. section 10.4.1
