[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog:post:en:proxy-failover-circuit-breaker-python":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":14,"thumbnail_url":15,"translations":16,"body":17,"asset_base":18},"proxy-failover-circuit-breaker-python","en","Implementing Proxy Failover with Circuit Breakers in Python","Build a resilient proxy failover system with circuit breakers in Python to ensure uninterrupted scraping.","2026-08-26",[10,11,12,13],"proxy","failover","circuit","python",[10,11,12,13],"https://blog-api.ro-proxy.com/api/blog/posts/proxy-failover-circuit-breaker-python/thumbnail.svg?lang=en",[5],"## Introduction\n\nIn the world of web scraping and automated data collection, a single proxy failure can bring your entire pipeline to a halt. Whether you are monitoring prices, collecting search results, or aggregating social media feeds, uninterrupted access is essential. A robust failover strategy ensures that when one proxy becomes unavailable, another takes over without noticeable delay. This article explains how to implement proxy failover using the circuit breaker pattern in Python, providing concrete code and real‑world examples.\n\n## Why Proxy Failover Matters\n\nProxies can fail for many reasons: IP bans, network partitions, rate limiting, or hardware issues. In each case, the symptom is usually an exception—connection timeout, HTTP 429, or a 5xx error. If your script does not handle these gracefully, it will stop processing and you may lose data or violate service level agreements.\n\nA failover mechanism addresses this by:\n\n- Detecting failures quickly.\n- Switching to a healthy proxy automatically.\n- Recovering the original proxy once it becomes available again.\n\n## Understanding Circuit Breakers\n\nThe circuit breaker is a design pattern inspired by electrical circuit breakers. It wraps a remote call and tracks consecutive failures. When the failure count exceeds a configurable threshold, the breaker opens, preventing further calls to the faulty endpoint. After a cool‑down period, the breaker moves to a half‑open state, allowing a limited number of test requests. If those succeed, the breaker closes; otherwise it reopens.\n\n### Key Parameters\n\n- **Failure threshold** – number of consecutive errors before opening.\n- **Timeout** – duration the breaker stays open before attempting recovery.\n- **Cool‑down period** – time between opening and half‑open transition.\n\n## Implementing a Basic Circuit Breaker in Python\n\nBelow is a self‑contained implementation using Python’s standard library. It is intentionally simple; you can extend it with logging, metrics, or asynchronous support.\n\n```python\nimport time\nimport threading\nfrom enum import Enum\n\nclass CircuitState(Enum):\n    CLOSED = 'closed'\n    OPEN = 'open'\n    HALF_OPEN = 'half-open'\n\nclass CircuitBreaker:\n    def __init__(self, failure_threshold=5, timeout=30):\n        self.failure_threshold = failure_threshold\n        self.timeout = timeout\n        self.failure_count = 0\n        self.last_attempt_time = None\n        self.state = CircuitState.CLOSED\n        self.lock = threading.Lock()\n\n    def call(self, func, *args, **kwargs):\n        with self.lock:\n            if self.state == CircuitState.OPEN:\n                if time.time() - self.last_attempt_time > self.timeout:\n                    self.state = CircuitState.HALF_OPEN\n                else:\n                    raise Exception('Circuit breaker is OPEN')\n            try:\n                result = func(*args, **kwargs)\n                self._on_success()\n                return result\n            except Exception as e:\n                self._on_failure()\n                raise e\n\n    def _on_success(self):\n        self.failure_count = 0\n        self.state = CircuitState.CLOSED\n\n    def _on_failure(self):\n        self.failure_count += 1\n        self.last_attempt_time = time.time()\n        if self.failure_count >= self.failure_threshold:\n            self.state = CircuitState.OPEN\n```\n\n**Explanation**\n\n- The `call` method wraps any function.\n- If the breaker is open and the timeout has not elapsed, it raises immediately.\n- After a successful call, the breaker resets.\n- After a failure, the counter increments; reaching the threshold opens the breaker.\n\n## Integrating with a Proxy Pool\n\nA proxy pool is a collection of proxy addresses. The circuit breaker can be combined with a round‑robin or random selection strategy.\n\n```python\nimport requests\n\nclass ProxyPool:\n    def __init__(self, proxies):\n        self.proxies = proxies\n        self.index = 0\n        self.lock = threading.Lock()\n\n    def get_next(self):\n        with self.lock:\n            proxy = self.proxies[self.index]\n            self.index = (self.index + 1) % len(self.proxies)\n            return proxy\n\ndef fetch_url(url, pool, breaker):\n    proxy = pool.get_next()\n    proxies = {'http': proxy, 'https': proxy}\n    try:\n        response = requests.get(url, proxies=proxies, timeout=10)\n        response.raise_for_status()\n        return response.text\n    except Exception:\n        # Re‑raise to let the breaker handle it\n        raise\n```\n\n**Usage**\n\n```python\npool = ProxyPool([\n    'http://proxy1.example.com:8080',\n    'http://proxy2.example.com:8080',\n    'http://proxy3.example.com:8080'\n])\nbreaker = CircuitBreaker(failure_threshold=3, timeout=60)\n\nurls = ['https://example.com/page1', 'https://example.com/page2']\nfor url in urls:\n    try:\n        html = breaker.call(fetch_url, url, pool, breaker)\n        # Process html\n    except Exception as e:\n        print(f'Failed to fetch {url}: {e}')\n```\n\nIn this snippet, each request is routed through the next proxy. If a proxy fails three times in a row, the breaker opens for that proxy, and subsequent calls are rejected until the timeout expires.\n\n## Handling Different Failure Modes\n\nNot all errors should trip the breaker equally. For instance:\n\n- **Connection errors** (DNS resolution failure, TCP reset) are often transient.\n- **HTTP 429** indicates rate limiting; you may want to back off rather than failover immediately.\n- **5xx** errors suggest server problems; a retry with exponential back‑off might succeed.\n- **401/407** point to authentication issues, which are unlikely to resolve by switching proxies.\n\nYou can customize the breaker by inspecting the exception type or response status code:\n\n```python\ndef should_trip(exception):\n    if isinstance(exception, requests.exceptions.ConnectionError):\n        return True\n    if isinstance(exception, requests.exceptions.HTTPError):\n        status = exception.response.status_code\n        return status in (429, 500, 502, 503, 504)\n    return False\n```\n\nIntegrate this logic into the `_on_failure` method to decide whether to increment the failure count.\n\n## Advanced: Dynamic Thresholds\n\nStatic thresholds may be too rigid. A more sophisticated approach adjusts the threshold based on recent success rate. For example, if the last ten calls succeeded, increase the threshold; if many failures occur, lower it. This adaptive behavior reduces unnecessary proxy churn while still protecting against prolonged outages.\n\n## Monitoring and Logging\n\nVisibility into breaker state is crucial for debugging. You can expose metrics using Prometheus or simply log state transitions:\n\n```python\nimport logging\nlogging.basicConfig(level=logging.INFO)\n\nclass LoggingCircuitBreaker(CircuitBreaker):\n    def _on_success(self):\n        super()._on_success()\n        logging.info('Breaker closed')\n    def _on_failure(self):\n        super()._on_failure()\n        if self.state == CircuitState.OPEN:\n            logging.warning('Breaker opened')\n```\n\n## Real‑World Example: E‑commerce Price Monitoring\n\nImagine you need to scrape product pages from an online store every minute. Using a single proxy risks being blocked after a few requests. By rotating through a pool of residential proxies and applying a circuit breaker, you can:\n\n1. **Detect** when a proxy returns repeated 403 or 429 responses.\n2. **Open** the circuit for that proxy.\n3. **Redirect** traffic to the next proxy in the pool.\n4. **Recover** after a cool‑down, allowing the original proxy to be retried.\n\nThis approach reduces downtime and improves data freshness.\n\n## Best Practices\n\n- **Set sensible thresholds** – too low causes unnecessary switching; too high delays failover.\n- **Log state changes** – helps debug flaky proxies.\n- **Combine with retries** – a retry with exponential back‑off can complement the breaker.\n- **Monitor health** – expose metrics (e.g., via Prometheus) for each proxy.\n- **Test with mock failures** – simulate errors to verify breaker behavior before deploying.\n\n## Conclusion\n\nImplementing proxy failover with circuit breakers transforms fragile scraping scripts into resilient data pipelines. By wrapping each request in a stateful breaker and rotating through a pool, you gain automatic recovery from transient failures while maintaining high throughput. With the patterns and code provided, you can build a robust foundation for any large‑scale data collection effort.\n","https://blog-api.ro-proxy.com/api/blog/posts/proxy-failover-circuit-breaker-python/assets"]