When running an async web scraper across 500,000 product pages, my server crashed after 45 minutes due to an Out Of Memory (OOM) error. Python RAM usage ballooned from 150 MB to over 7 GB.<\/p>\n
Creating a new By reusing a single connection pool and throttling concurrency with a semaphore, memory stayed flat at 85 MB throughout the entire crawl.<\/p>\n","protected":false},"excerpt":{"rendered":" When running an async web scraper across 500,000 product pages, my server crashed after 45 minutes due to an Out Of Memory (OOM) error. Python RAM usage ballooned from 150 MB to over 7 GB. The Culprit: Unclosed ClientSessions and Unlimited Concurrency Creating a new ClientSession() inside every async task prevents connection reuse and leaks … Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[32,33,20,9],"class_list":["post-110","post","type-post","status-publish","format-standard","hentry","category-python","tag-aiohttp","tag-asyncio","tag-performance","tag-python"],"_links":{"self":[{"href":"https:\/\/top.fadah.store\/index.php?rest_route=\/wp\/v2\/posts\/110","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/top.fadah.store\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/top.fadah.store\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/top.fadah.store\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/top.fadah.store\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=110"}],"version-history":[{"count":0,"href":"https:\/\/top.fadah.store\/index.php?rest_route=\/wp\/v2\/posts\/110\/revisions"}],"wp:attachment":[{"href":"https:\/\/top.fadah.store\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=110"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/top.fadah.store\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=110"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/top.fadah.store\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=110"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}ClientSession()<\/code> inside every async task prevents connection reuse and leaks socket file descriptors. Furthermore, firing 50,000 coroutines without a semaphore overwhelms Python’s event loop.<\/p>\nThe Optimized Pattern<\/h2>\n
import asyncio\nimport aiohttp\n\nSEMAPHORE = asyncio.Semaphore(50) # Max 50 concurrent requests\n\nasync def fetch_url(session, url):\n async with SEMAPHORE:\n try:\n async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:\n return await response.text()\n except Exception as e:\n return None\n\nasync def main(urls):\n connector = aiohttp.TCPConnector(limit=100, limit_per_host=10)\n async with aiohttp.ClientSession(connector=connector) as session:\n tasks = [fetch_url(session, u) for u in urls]\n results = await asyncio.gather(*tasks)\n print(f\"Fetched {len([r for r in results if r])} pages successfully.\")\n\nif __name__ == \"__main__\":\n urls = [f\"https:\/\/example.com\/item\/{i}\" for i in range(1000)]\n asyncio.run(main(urls))<\/code><\/pre>\n