Concurrent HTTP requests in Python

  • Python
  • Gabriele Romanato

Concurrent HTTP requests are a key aspect of developing modern applications that require retrieving data from web services or interacting with remote APIs. Making HTTP requests sequentially can be inefficient and slow down program execution. Fortunately, Python offers several libraries and tools that allow you to make HTTP requests concurrently, improving the efficiency and performance of your application.

In this article, we will explore some of the most common libraries for making concurrent HTTP requests in Python: asyncio and requests. We will see how to use these libraries to execute HTTP requests in a parallel manner, thus improving the performance of our code.

Before starting, let's make sure we have the following libraries installed:

  • aiohttp: A library for making asynchronous HTTP requests efficiently. You can install it using pip.

  • requests: A widely used library for making synchronous HTTP requests. If you haven't already installed it, you can do it with pip.

Let's start with an example of how to make concurrent HTTP requests using asyncio and aiohttp. Suppose we want to make requests to three different URLs and retrieve the contents of each.


import asyncio
import aiohttp

async def fetch_url(url):
     async with aiohttp.ClientSession() as session:
         async with session.get(url) as response:
             return await response.text()

async def main():
     urls = ['https://example.com', 'https://google.com', 'https://github.com']

     tasks = [fetch_url(url) for url in urls]
     results = await asyncio.gather(*tasks)

     for url, result in zip(urls, results):
         print(f'Response from {url}: {result[:100]}...')

if __name__ == '__main__':
     asyncio.run(main())

In this example, we define a fetch_url function that makes an asynchronous HTTP request to a specific URL using aiohttp. The main function creates a list of URLs to request, then concurrently starts the requests using asyncio.gather. Finally, the contents of the first 100 characters of each response are printed.

If you prefer to use requests, you can use the concurrent.futures module to execute HTTP requests concurrently. Here's an example:


import concurrent.futures
import requests

def fetch_url(url):
     response = requests.get(url)
     return response.text

if __name__ == '__main__':
     urls = ['https://example.com', 'https://google.com', 'https://github.com']

     with concurrent.futures.ThreadPoolExecutor() as executor:
         results = list(executor.map(fetch_url, urls))

     for url, result in zip(urls, results):
         print(f'Response from {url}: {result[:100]}...')

In this case, we use a ThreadPoolExecutor to execute HTTP requests in separate threads. This allows HTTP requests to be executed concurrently, even though requests is synchronous by nature.

Conclusion

Concurrent HTTP requests can greatly improve the performance of your Python applications, allowing you to retrieve data from web services more efficiently. Choose the library that best suits your needs and the requirements of your project. Libraries like asyncio and aiohttp are particularly useful for asynchronous applications, while requests with concurrent.futures can be useful when working with synchronous legacy code.

Back to top