Concurrent file compression assigns each input file to its own goroutine instead of compressing files serially. The approach can improve use of available disk I/O bandwidth when processing a few hundred files or fewer. section 5.2.1
A practical implementation follows this structure:
- Keep compression in a reusable function that opens the input, creates a .gz output, copies through gzip.NewWriter, and closes resources. section 5.2.1
- Create a sync.WaitGroup, add one count per input file, and launch a goroutine for each compression task. section 5.2.1
- Call Done when each worker finishes, then wait in main before reporting the number of compressed files. section 5.2.1
Listing 5.3 Compressing files in parallel with a wait group
package main
import (
"compress/gzip"
"fmt"
"io"
"os"
"sync"
)
func main() {
var wg sync.WaitGroup #1
for i, file := range os.Args[1:] {
wg.Add(1) #2
go func(filename string) { #3
compress(filename) #3
wg.Done() #3
}(file) #4
}
wg.Wait() #5
fmt.Printf("Compressed %d files\n", len(os.Args[1:]))
}
func compress(filename string) error {
// Unchanged from above
}
Shows a complete Go implementation that launches one compression goroutine per command-line file and waits for all workers. section 5.2.1
