The Go bufio package wraps Readers and Writers with buffering for efficient, incremental stream processing. It is especially useful when large files should be processed without loading everything into memory. section 7.1.1
For newline-delimited JSON, bufio.Scanner combined with bufio.ScanLines reads one record at a time. Each line can be unmarshaled independently, allowing invalid entries to be logged and skipped while processing continues. section 7.1.1
Listing 7.4 Using bufio to read line by line
package main
import (
"bufio"
"encoding/json"
"log"
"os"
)
func main() {
file, err := os.Open("structured.log")
if err != nil {
panic(err)
}
defer file.Close()
scan := bufio.NewScanner(file) #1
scan.Split(bufio.ScanLines) #2
lineJSON := make(map[string]interface{})
for scan.Scan() { #3
if err := json.Unmarshal([]byte(scan.Text()), &lineJSON); err != n
il {
log.Println(err)
} else {
log.Println(lineJSON["level"])
}
}
}
Shows a complete Go program that opens structured.log, configures line scanning, unmarshals each record, and prints its level. section 7.1.1
