Introduction
A worker pool is a concurrency pattern where a fixed number of goroutines ("workers") pull tasks from a shared jobs channel, process them, and send results to a results channel. This bounds the amount of concurrent work happening at once — useful when you have many tasks but want to limit resource usage (CPU, memory, open connections) rather than launching an unbounded goroutine per task. The pattern is a classic example of Go's fan-out/fan-in style: fan-out distributes jobs across multiple workers, and fan-in collects their results back into a single channel.
Cricket analogy: A fixed number of net bowlers at practice pull batting turns from a shared queue rather than every batsman getting a personal bowler, bounding resource use just as a worker pool limits concurrent goroutines, mirroring fan-out/fan-in across the nets.
Syntax
jobs := make(chan int, 100)
results := make(chan int, 100)
// fan-out: launch N worker goroutines
for w := 1; w <= numWorkers; w++ {
go worker(w, jobs, results)
}
// send jobs, then close so workers know when to stop ranging
for _, j := range jobList {
jobs <- j
}
close(jobs)
// fan-in: collect results
for i := 0; i < len(jobList); i++ {
<-results
}Explanation
Each worker goroutine typically runs a loop like for job := range jobs { results <- process(job) }, consuming jobs until the jobs channel is closed and drained. Because multiple workers range over the same channel, Go automatically distributes jobs among them — each job is delivered to exactly one worker. Closing the jobs channel after all jobs are sent is essential: it's the signal that lets every worker's range loop exit cleanly, rather than blocking forever waiting for more work. A sync.WaitGroup is often used alongside this to know when all workers have finished, especially if you want to safely close the results channel afterward for a downstream consumer to range over.
Cricket analogy: Net bowlers each run the same loop of grabbing the next batsman from the shared queue and bowling to them until the queue is empty, and once the coach closes the queue, every bowler's loop exits cleanly instead of waiting forever.
Example
package main
import "fmt"
func worker(id int, jobs <-chan int, results chan<- int) {
for j := range jobs {
results <- j * j
}
}
func main() {
const numJobs = 5
const numWorkers = 3
jobs := make(chan int, numJobs)
results := make(chan int, numJobs)
for w := 1; w <= numWorkers; w++ {
go worker(w, jobs, results)
}
for i := 1; i <= numJobs; i++ {
jobs <- i
}
close(jobs)
sum := 0
for i := 0; i < numJobs; i++ {
sum += <-results
}
fmt.Println("Sum of squares:", sum)
}Output
Sum of squares: 55 The 3 workers process the 5 jobs (1² through 5²) concurrently in some interleaved order, but since the results are summed rather than printed individually, the output is deterministic: 1+4+9+16+25 = 55.
Cricket analogy: Three net bowlers delivering to five batsmen in overlapping, unpredictable order still produces the same final scoreboard total, 55 runs, since summing doesn't care which bowler served which batsman, only the total.
Key Takeaways
- A worker pool launches a fixed number of goroutines that consume tasks from a shared jobs channel.
- Closing the jobs channel after all jobs are sent lets worker range loops exit cleanly.
- Multiple workers ranging over the same channel automatically load-balances jobs across them.
- This is Go's classic fan-out/fan-in pattern for bounded, controlled concurrency.
- Combine with sync.WaitGroup to know when all workers are done before closing the results channel.
Practice what you learned
1. What is the main benefit of a worker pool over launching one goroutine per task?
2. Why is closing the jobs channel important in a worker pool?
3. What term describes distributing jobs across multiple worker goroutines from a single channel?
4. What happens if multiple worker goroutines range over the same jobs channel?
Was this page helpful?
You May Also Like
Goroutines in Go
Goroutines are lightweight, runtime-managed threads that let Go programs run functions concurrently with minimal overhead.
Channels in Go
Channels are typed conduits that let goroutines safely send and receive values, embodying Go's share-memory-by-communicating philosophy.
The sync Package in Go
The sync package provides primitives like WaitGroup, Mutex, and Once for coordinating goroutines and protecting shared state.
The select Statement in Go
The select statement lets a goroutine wait on multiple channel operations at once, proceeding with whichever is ready first.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics