Stateful training carries an RNN’s hidden state from one batch or document to the next, allowing earlier text to provide context for later predictions. section 8.3.1
Use it only when document order remains consistent across batches and epochs. Otherwise, reset the hidden state to zeros before each new document so predictions do not inherit unrelated context. section 8.3.1
Keeping context reproducible
Arrange documents in a reproducible order, preserve that order across batches, and warm up the model with context documents before each real-world or test-set prediction. Shuffled training examples suit models expected to predict cold, without priming. section 8.3.1
