Concurrent HTTP requests in Go

Go is a programming language known for its ease of use in managing concurrency. One of the common operations that often requires concurrency is sending HTTP requests to multiple endpoints at the same time. In this article, we'll explore how to make concurrent HTTP requests in Go using standard libraries and goroutines to maximize efficiency.

To make HTTP requests in Go, we will use the net/http package. Make sure you import it into your source file:

import (
     "fmt"
     "net/http"
)

Go offers goroutines, which are lightweight and managed by the runtime. We can use goroutines to execute HTTP requests concurrently and make best use of system resources. Here's an example of how to do it:

package main

import (
     "fmt"
     "net/http"
     "sync"
)

func fetchURL(url string, wg *sync.WaitGroup) {
     defer wg.Done()

     resp, err := http.Get(url)
     if err != nil {
         fmt.Printf("Error requesting %s: %v\n", url, err)
         return
     }
     defer resp.Body.Close()

     fmt.Printf("URL: %s, Status: %s\n", url, resp.Status)
}

func main() {
     urls := []string{
         "https://www.example.com",
         "https://www.example.org",
         "https://www.example.net",
     }

     var wg sync.WaitGroup

     for _, url := range urls {
         wg.Add(1)
         go fetchURL(url, &wg)
     }

     wg.Wait()
}

In this example, we create a fetchURL function that makes an HTTP request to the provided URL and prints the result. We use a sync.WaitGroup to wait until all goroutines have completed requests.

It is important to handle errors when making concurrent HTTP requests. In the code above, we've included simple error handling that prints a message if the request is unsuccessful. You can customize error handling to suit your needs.

In some cases, you may want to limit the number of goroutines running at the same time to avoid overloading the system. You can do this by using a semaphore or channel to control the number of active goroutines.

package main

import (
     "fmt"
     "net/http"
     "sync"
)

func fetchURL(url string, wg *sync.WaitGroup, sem chan struct{}) {
     defer func() {
         sem <- struct{}{}
         wg.Done()
     }()

     resp, err := http.Get(url)
     if err != nil {
         fmt.Printf("Error requesting %s: %v\n", url, err)
         return
     }
     defer resp.Body.Close()

     fmt.Printf("URL: %s, Status: %s\n", url, resp.Status)
}

func main() {
     urls := []string{
         "https://www.example.com",
         "https://www.example.org",
         "https://www.example.net",
     }

     var wg sync.WaitGroup
     sem := make(chan struct{}, 5) // Limit the number of active goroutines to 5

     for _, url := range urls {
         wg.Add(1)
         sem <- struct{}{} // Acquire a semaphore slot
         go fetchURL(url, &wg, sem)
     }

     wg.Wait()
     close(sem)
}

In this example, we created a sem channel to limit the number of active goroutines to 5. Each goroutine acquires a semaphore slot before it starts executing the request and releases it once it completes.

Conclusions

Making concurrent HTTP requests in Go is an effective way to improve the performance of your applications. Using goroutines and synchronization structures such as sync.WaitGroup or channels, you can easily handle numerous requests at once efficiently and safely. Make sure you handle errors appropriately and tailor the number of active goroutines to your specific needs. With these techniques, you will be able to make the most of the concurrency potential that Go offers in your web applications.