A token is one piece of input produced by a scanner. In Go, bufio.Scanner uses a split function to determine token boundaries; by default, it returns lines without their trailing newlines. Scan advances to the next token, which Text makes available as a string. section 2.8.1
Listing 2.36 Output of go doc bufio Scanner
type Scanner struct {
// Has unexported fields.
}
Scanner provides a convenient interface for reading data such as a file
of newline-delimited lines of text. Successive calls to the Scan method
will step through the 'tokens' of a file, skipping the bytes between the
tokens. The specification of a token is defined by a split function of
type SplitFunc; the default split function breaks the input into lines
with line termination stripped. Split functions are defined in this
package for scanning a file into lines, bytes, UTF-8-encoded runes, and
space-delimited words. The client may instead provide a custom split
function.
Scanning stops unrecoverably at EOF, the first I/O error, or a token too
large to fit in the buffer. When a scan stops, the reader may have
advanced arbitrarily far past the last token. Programs that need more
control over error handling or large tokens, or must run sequential
scans on a reader, should use bufio.Reader instead.
func NewScanner(r io.Reader) *Scanner
func (s *Scanner) Buffer(buf []byte, max int)
func (s *Scanner) Bytes() []byte
func (s *Scanner) Err() error
func (s *Scanner) Scan() bool
func (s *Scanner) Split(split SplitFunc)
func (s *Scanner) Text() string
The Scanner documentation describes default and alternative split rules, including lines, bytes, runes, words, and custom splitting. section 2.8.1
Count words in scanned lines
Scan yields each line by default; strings.Fields then separates that line into words, whose count can be added to a running total. section 2.8.1
