When running an async web scraper across 500,000 product pages, my server crashed after 45 minutes due to an Out Of Memory (OOM) error. Python RAM usage ballooned from 150 MB to over 7 GB. The Culprit: Unclosed ClientSessions and Unlimited Concurrency Creating a new ClientSession() inside every async task prevents connection reuse and leaks … Read more
Like most developers, my Downloads folder quickly turns into an unrecognizable wasteland of installers, PDFs, ZIP archives, and random screenshots. I wrote a small background daemon in Python to automatically sort files into neat subdirectories based on extension. The Python Cleaner Script import os import shutil from pathlib import Path DOWNLOADS_DIR = Path.home() / “Downloads” … Read more
Extracting financial tables from scanned and vector PDFs is notoriously frustrating. Libraries like tabula-py often miss borders or misalign columns. Here is the exact routine I built using pdfplumber and pandas. Implementation Code import pdfplumber import pandas as pd def extract_clean_tables(pdf_path): all_rows = [] with pdfplumber.open(pdf_path) as pdf: for page in pdf.pages: table = page.extract_table() … Read more
Relying on manual database dumps is a recipe for disaster. I wrote a lightweight, self-contained Python script to dump a remote Postgres database, compress it with gzip, and upload it directly to an Amazon S3 bucket with date versioning. The Script import os import subprocess from datetime import datetime import boto3 DB_NAME = “production_db” DB_USER … Read more
I was building an automated Python scraper to track local real estate price drops. The target website was protected by Cloudflare Turnstile, and simple requests or vanilla Playwright runs immediately resulted in a 403 Forbidden block. Why Standard Headless Browsers Fail Cloudflare checks navigator properties, WebGL fingerprints, and browser automation flags like navigator.webdriver = true. … Read more