A line-delimited record is an independently represented data item separated by a newline. This layout supports streaming: read one line, parse it, and continue without loading the entire file. section 7.1.1
Line-delimited records are useful when:
- Each record is JSON, CSV, or another format that can be processed independently. section 7.1.1
- Buffered I/O avoids scalability problems associated with reading a whole file at once. section 7.1.1
- An invalid record can be logged or ignored while processing continues with later lines. section 7.1.1
Listing 7.3 Finding JSON needles in a haystack
lineJSON := make(map[string]interface{}) #1
var bChunk []byte
for {
b := make([]byte, 2)
_, err := file.Read(b)
if err != nil {
break
}
bChunk = append(bChunk, b[0:]...) #2
if err := json.Unmarshal(bChunk, &lineJSON);
err == nil { #3
log.Println(lineJSON)
bChunk = []byte{}
}
}
The implementation opens a structured log, uses bufio.Scanner with bufio.ScanLines, unmarshals each line as JSON, and reports parsing errors without stopping the scan. section 7.1.1
