Streaming processes data as it arrives or in successive pieces instead of loading it all before doing any work. For an event stream, those pieces are discrete records: a log can contain one JSON record per line, and a network logging service can receive messages continuously. File copying and uploads also use streaming, though their pieces are bytes or multipart parts rather than necessarily events. section 7.1.1 section 7.2.1 section 10.2.6
Choose the unit of work to match the input. For a large newline-separated log, open the file and scan one line at a time, then decode each line as JSON. This avoids the memory cost of reading and splitting the whole file. Repeatedly appending tiny byte reads until they happen to form valid JSON is brittle, especially when input is malformed. section 7.1.1
Listing 7.4 Using bufio to read line by line
package main
import (
"bufio"
"encoding/json"
"log"
"os"
)
func main() {
file, err := os.Open("structured.log")
if err != nil {
panic(err)
}
defer file.Close()
scan := bufio.NewScanner(file) #1
scan.Split(bufio.ScanLines) #2
lineJSON := make(map[string]interface{})
for scan.Scan() { #3
if err := json.Unmarshal([]byte(scan.Text()), &lineJSON); err != n
il {
log.Println(err)
} else {
log.Println(lineJSON["level"])
}
}
}
The scanner supplies each log line to json.Unmarshal, so records can be handled without first loading the entire file. section 7.1.1
When both ends expose Go’s Reader and Writer interfaces, io.Copy moves data through a buffer without a hand-written read/write loop. For an upload, use Request.MultipartReader to receive parts while the request is in progress and save them toward their destination. Unlike ParseMultipartForm, this need not stage the whole upload in the API server’s temporary-file area—a useful distinction for large or concurrent uploads. section 7.1.2 section 10.2.6
Listing 7.6 Copying buffered data from Reader to Writer
package main
import (
"io"
"os"
)
func main() {
src, err := os.Open("test.txt") #1
if err != nil {
panic(err)
}
defer src.Close()
dest, err := os.Create("test2.txt")
if err != nil {
panic(err)
}
defer dest.Close()
_, err = io.Copy(dest, src) #2
if err != nil {
panic(err)
}
}
A source file and destination file are connected with io.Copy, illustrating buffered transfer through Reader and Writer interfaces. section 7.1.2
Streaming changes failure handling as well as resource use. Network log streams can lose messages when sending fails; channels and buffering can improve reliability. Writing a template directly to an HTTP response can deliver content sooner, but an execution error may leave users with a partial page. Buffer the rendered response first when detecting an error before display matters more than immediate output. section 7.2.1 section 9.1.4
