Fixing Memory Leaks When Scraping Millions of URLs with aiohttp in Python

by

When running an async web scraper across 500,000 product pages, my server crashed after 45 minutes due to an Out Of Memory (OOM) error. Python RAM usage ballooned from 150 MB to over 7 GB.

The Culprit: Unclosed ClientSessions and Unlimited Concurrency

Creating a new ClientSession() inside every async task prevents connection reuse and leaks socket file descriptors. Furthermore, firing 50,000 coroutines without a semaphore overwhelms Python’s event loop.

The Optimized Pattern

import asyncio
import aiohttp

SEMAPHORE = asyncio.Semaphore(50)  # Max 50 concurrent requests

async def fetch_url(session, url):
    async with SEMAPHORE:
        try:
            async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:
                return await response.text()
        except Exception as e:
            return None

async def main(urls):
    connector = aiohttp.TCPConnector(limit=100, limit_per_host=10)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [fetch_url(session, u) for u in urls]
        results = await asyncio.gather(*tasks)
        print(f"Fetched {len([r for r in results if r])} pages successfully.")

if __name__ == "__main__":
    urls = [f"https://example.com/item/{i}" for i in range(1000)]
    asyncio.run(main(urls))

By reusing a single connection pool and throttling concurrency with a semaphore, memory stayed flat at 85 MB throughout the entire crawl.

Leave a Comment