Different ways of handling concurrent job in GoLang

GoLang provides very good support of concurrency and the way to make some code to run concurrent is pretty simple and straightforward. Adding a simple go before a normal function call will make the function call asynchronous.

In real cases normally people would concurrently run some jobs to improve the speed and efficiency. One important part of running jobs concurrently is about aggregating results so that the consequent function call would be able to proceed. There are multiple ways handling this scenarios.

One of them is using WaitGroup from sync package. The pattern is like:

var wg sync.WaitGroup
for rows.Next() {

	go func() {
        wg.Add(1)
		defer wg.Done()

		// DO SOME WORK
	}()
}

wg.Wait()

First a WaitGroup is created, thereafter when a new task goroutine is spawned, the task will be added to the wait group and it will call wg.Done() to signal the task is completed with a defer statement. After all tasks are launched, the wait group will just wait until all tasks have called wg.Done(). 

This is frequently used when there are different tasks happening and they are not repeating. Given that the wait group will wait for all tasks to complete before proceeding, there is a pitfall where some resource would be wasted if one task would take much longer time than other tasks. This is especially bad when used in cases where there are repeated new tasks coming into the queue.

To avoid above long running task blocking issue, some other mechanism provided by GoLang can be leveraged -- channel. Basically some task queue using buffered channel can be created. Whenever there is a new task, it will be put in the buffered channel and the task will be waiting to be queued if the buffered channel is full. At the same time, there could be multiple goroutines taking task from the same buffered channel and do its job and take the next available task from the buffered channel. Other goroutines can still take task even if one goroutine is running a time consuming task. This would improve efficiency and performance compared to the WaitGroup approach.

for rows.Next() {
	job := &UpdateJob{
	}

	// add job to queue
	pool.Queue <- job
}

// worker 1
go func() {
	job := <- pool.Queue

	// DO SOMETHING 
}

// worker 2
go func() {
	job := <- pool.Queue

	// DO SOMETHING 
}

To understand this buffered channel approach much better, can refer to this example about worker pool in GoLang which utilizes buffered channel.

These different approaches have its own use scenarios and adopt it wisely.