A character-level language model generates text one character at a time. In Andrej Karpathy’s 2015 char-rnn experiment, a recurrent neural network learned from roughly one million characters of Paul Graham’s essays, then sampled each new character from a probability distribution. A temperature setting controlled the trade-off: lower values made output more predictable and repetitive; higher values made it more varied but less coherent. section 4.1
Startup advice, one character at a time
The generated passage mentions investors, fundraising rounds, and acquirers, echoing its training essays. Yet its sentences fail to add up to useful advice: recognizable subject matter is not the same as a coherent argument. The experiment made RNN text generation vivid without suggesting that plausible prose meant understanding. section 4.1
Figure 4.1 This image, from Chris Olah’s work (discussed in the next section), shows a chain-like structure representing an RNN. It illustrates how RNNs process data sequentially, with the output of one step feeding into the next. The recurrent, linked modules reflect the RNN’s ability to carry information forward, making it well suited to learning patterns that unfold over time, such as time series, text, or audio. Used with permission granted by the author (Chris Olah).
The linked RNN modules show how information passes from one step to the next during sequential processing. section 4.1
