[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog:post:en:adaptive-proxy-rotation-adjusting-proxies-based-on-response-codes-and-latency":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":14,"thumbnail_url":15,"translations":16,"body":17,"asset_base":18},"adaptive-proxy-rotation-adjusting-proxies-based-on-response-codes-and-latency","en","Adaptive Proxy Rotation: Adjusting Proxies Based on Response Codes and Latency","Learn how to monitor request outcomes and adjust your proxy pool dynamically to improve success rates and reduce bans.","2026-07-17",[10,11,12,13],"proxy","rotation","web-scraping","performance",[10,11,12,13],"https://blog-api.ro-proxy.com/api/blog/posts/adaptive-proxy-rotation-adjusting-proxies-based-on-response-codes-and-latency/thumbnail.svg?lang=en",[5],"## Why Adaptive Proxy Rotation Matters\n\nWhen you scrape or automate tasks at scale, a static rotation schedule can waste good proxies and keep using bad ones. By watching each request's outcome — HTTP status code, response time, and occasional error messages — you can decide in real time whether to keep using a proxy, retire it, or move it to a cool‑down pool.\n\n## Building the Health Score\n\nA simple scoring algorithm works well for many use cases:\n\n- Start each proxy with a score of 100.\n- On a successful request (status 2xx) and latency under a target (e.g., 800 ms), add 2 points.\n- On a retry‑worthy status (429, 502‑504) subtract 10 points.\n- On a hard failure (timeout, connection error) subtract 20 points.\n- If latency exceeds the target, subtract 1 point per 100 ms over the limit.\n- Clamp the score between 0 and 200.\n\nWhen the score falls below 30, move the proxy to a cool‑down list for a configurable period (e.g., 5 minutes). After the cool‑down, reset its score to 50 and return it to the active pool.\n\n## Implementing in Python\n\nBelow is a self‑contained example that uses the requests library and a round‑robin iterator over a list of proxies. The code shows how to update scores, enforce cool‑downs, and pick the next healthy proxy.\n\n```python\nimport time\nimport requests\nfrom collections import deque\n\n# ----- configuration -----\nTARGET_LATENCY_MS = 800\nSUCCESS_BONUS = 2\nRETRY_PENALTY = 10\nHARD_PENALTY = 20\nLATENCY_PENALTY_PER_100MS = 1\nCOOLDOWN_SECONDS = 300\nMIN_SCORE_TO_USE = 30\nMAX_SCORE = 200\nSTART_SCORE = 100\nRESET_AFTER_COOLDOWN = 50\n\n# Example proxy list – replace with your RoProxy credentials\nPROXIES = [\n    'http://user:pass@proxy1.roproxy.com:3128',\n    'http://user:pass@proxy2.roproxy.com:3128',\n    'http://user:pass@proxy3.roproxy.com:3128',\n]\n\n# ----- state -----\nscores = {p: START_SCORE for p in PROXIES}\ncooldown_until = {p: 0 for p in PROXIES}\nproxy_queue = deque(PROXIES)\n\ndef latency_penalty(latency_ms):\n    if latency_ms \u003C= TARGET_LATENCY_MS:\n        return 0\n    excess = latency_ms - TARGET_LATENCY_MS\n    return (excess // 100) * LATENCY_PENALTY_PER_100MS\n\ndef update_score(proxy, status_code, latency_ms, error=None):\n    now = time.time()\n    if error:\n        scores[proxy] = max(0, scores[proxy] - HARD_PENALTY)\n        return\n    if 200 \u003C= status_code \u003C 300:\n        scores[proxy] = min(MAX_SCORE, scores[proxy] + SUCCESS_BONUS)\n    elif status_code in (429, 502, 503, 504):\n        scores[proxy] = max(0, scores[proxy] - RETRY_PENALTY)\n    else:\n        # other 4xx/5xx treat as mild penalty\n        scores[proxy] = max(0, scores[proxy] - 5)\n\n    scores[proxy] -= latency_penalty(latency_ms)\n    scores[proxy] = max(0, min(MAX_SCORE, scores[proxy]))\n\n    if scores[proxy] \u003C MIN_SCORE_TO_USE:\n        cooldown_until[proxy] = now + COOLDOWN_SECONDS\n\ndef is_available(proxy):\n    return time.time() >= cooldown_until[proxy]\n\ndef get_next_proxy():\n    while proxy_queue:\n        p = proxy_queue.popleft()\n        if is_available(p):\n            proxy_queue.append(p)  # put back at end for round‑robin\n            return p\n        else:\n            # still cooling, put at end and try next\n            proxy_queue.append(p)\n    # fallback: return the proxy with highest score regardless of cooldown\n    return max(scores, key=scores.get)\n\ndef fetch(url):\n    start = time.time()\n    proxy = get_next_proxy()\n    proxies = {'http': proxy, 'https': proxy}\n    try:\n        resp = requests.get(url, proxies=proxies, timeout=10)\n        latency_ms = (time.time() - start) * 1000\n        update_score(proxy, resp.status_code, latency_ms)\n        return resp\n    except requests.RequestException as exc:\n        latency_ms = (time.time() - start) * 1000\n        update_score(proxy, 0, latency_ms, error=True)\n        # optional: retry with another proxy\n        return None\n\n# ----- usage example -----\nif __name__ == '__main__':\n    for i in range(20):\n        r = fetch('https://httpbin.org/ip')\n        if r:\n            print('Request', i, ':', r.status_code, 'via', r.headers.get('Via', 'unknown'))\n        else:\n            print('Request', i, ': failed')\n        time.sleep(0.5)\n```\n\n### How the Script Works\n\n1. **Score initialization** – each proxy starts with a neutral score.\n2. **Request execution** – we pick the first available proxy from a rotating queue.\n3. **Result handling** – based on the HTTP status, latency, and any exceptions we adjust the score.\n4. **Cool‑down enforcement** – proxies that dip below the minimum score are shelved for a set period.\n5. **Fallback** – if every proxy is cooling, we fall back to the one with the highest score to avoid stalling.\n\n## Tuning the Parameters\n\n- **Target latency** – adjust TARGET_LATENCY_MS to match your network expectations. For residential proxies you might allow 1.2 seconds; for datacenter you could tighten to 400 ms.\n- **Reward/penalty values** – increase SUCCESS_BONUS if you want fast recovery of good proxies, or raise HARD_PENALTY to aggressively ban flaky nodes.\n- **Cool‑down duration** – longer cool‑downs reduce churn but may leave you with fewer active proxies; short cool‑downs keep the pool fluid but risk re‑using a bad proxy too soon.\n- **Score limits** – the MAX_SCORE prevents a single proxy from dominating forever; the MIN_SCORE_TO_USE defines the threshold for removal.\n\n## Real‑World Scenario: Price Monitoring Across Regions\n\nImagine you need to check product prices on an e‑commerce site every five minutes from three different countries. You purchase a mix of residential proxies from RoProxy targeting US, DE, and JP. Without adaptation, a single overloaded proxy might start returning 429 responses, causing you to miss price updates. With adaptive rotation:\n\n- The first 429 triggers a ‑10 penalty; after a few occurrences the score drops below 30 and the proxy goes into cool‑down.\n- While that proxy rests, the scheduler picks the next healthy proxy, preserving data quality.\n- After the cool‑down expires, the proxy’s score is reset to a moderate value (50) giving it another chance, which often succeeds once the remote server’s rate limit window has passed.\n\nThis loop keeps your scraper running smoothly without manual intervention.\n\n## Extending to Other Languages\n\nThe same logic can be ported to Node.js, Go, or any language that lets you make HTTP requests and measure latency. Key steps remain:\n\n1. Maintain a map of proxy identifiers to scores and cooldown timestamps.\n2. Before each request, select a proxy whose cooldown has expired.\n3. After the request, update the score based on status code, latency, and errors.\n4. Apply cool‑down when score falls below threshold.\n5. Optionally, implement a jittered selection to avoid predictable patterns.\n\n## Best Practices & Gotchas\n\n- **Validate proxy format** – ensure username:password@host:port is correctly URL‑encoded if special characters appear.\n- **Handle HTTPS** – when using HTTP proxies for HTTPS traffic, the library will establish a CONNECT tunnel; most HTTP clients support this out of the box.\n- **Avoid over‑penalizing transient spikes** – a single latency spike shouldn’t instantly ban a proxy; the linear latency penalty smooths out occasional delays.\n- **Log score changes** – keep a simple CSV or time‑series store (e.g., Prometheus) of each proxy’s score to detect systemic issues (e.g., a particular subnet consistently getting blocked).\n- **Respect target site’s terms** – adaptive rotation improves efficiency, but you should still obey rate limits and copyright rules; use proxies responsibly.\n\n## Conclusion\n\nAdaptive proxy rotation turns a static list of IPs into a self‑healing pool that reacts to real‑world feedback. By scoring each proxy on success, failure, and latency, you automatically sideline troubled nodes and give healthy ones more work. The Python example above provides a ready‑to‑start foundation; you can adapt the scoring rules, cool‑down times, and selection strategy to match your specific workload—whether it’s SERP monitoring, ad verification, or large‑scale web scraping.\n\nImplementing this pattern reduces manual proxy management, cuts down on bans, and makes your scraping jobs more reliable and cost‑effective.\n","https://blog-api.ro-proxy.com/api/blog/posts/adaptive-proxy-rotation-adjusting-proxies-based-on-response-codes-and-latency/assets"]