Bypassing Cloudflare 403 While Scraping Real Estate Data with Python and Playwright

I was building an automated Python scraper to track local real estate price drops. The target website was protected by Cloudflare Turnstile, and simple requests or vanilla Playwright runs immediately resulted in a 403 Forbidden block.

Why Standard Headless Browsers Fail

Cloudflare checks navigator properties, WebGL fingerprints, and browser automation flags like navigator.webdriver = true. When you launch standard headless Chromium, these flags give you away instantly.

The Solution: playwright-stealth and Realistic Viewports

The fix involved using playwright-stealth along with persistent browser contexts and human-like user agent headers.

import asyncio
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

async def scrape_listings():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
            viewport={"width": 1920, "height": 1080}
        )
        page = await context.new_page()
        await stealth_async(page)
        
        await page.goto("https://target-realestate-site.com/properties", wait_until="networkidle")
        print("Page title:", await page.title())
        await browser.close()

asyncio.run(scrape_listings())

The Result

The stealth patch removed the automated webdriver signatures, allowing the script to pass Cloudflare challenges seamlessly on every hourly run.