A multiword state makes two or more consecutive words the current state of a Markov text generator. Conditioning on this larger context gives the next-word distribution more information than a single-word state. section 16.2
The technique addresses implausible sentences caused by memoryless generation. Instead of using all words following his, a generator can use the narrower distribution following phrases such as of his or of France. section 16.2
Increasing the state size creates a clear trade-off:
- Pairs or triples preserve more local context and can make word choices more plausible. section 16.2
- Using an entire session as the state supplies still more context, but repeatedly analyzing longer text can make generation quadratic in document length. section 16.2
A narrower next-word distribution
When the generated ending is of his, prefer words observed after of his rather than words observed after his alone. section 16.2
Listing 16.2 A simple words-and-sentences parser
public static class Extensions
{
static readonly char[] punct = "*()[]#:;_\"".ToCharArray();
public static IEnumerable<List<string>> Sentences(
this IEnumerable<string> lines)
{
List<string> current = [];
foreach (var line in lines)
{
foreach (var word in line.Split())
{
var trimmed = word.Trim(punct);
if (trimmed.Length > 0)
{
current.Add(trimmed);
char c = trimmed[trimmed.Length - 1];
if (c == '.' || c == '!' || c == '?')
{
yield return current;
current = [];
}
}
}
}
}
public static string Space<T>(this IEnumerable<T> ts) =>
string.Join(" ", ts);
[...]
}
Inspect the sentence parser used to turn source text into ordered word sequences before constructing Markov states. section 16.2
