RoProxy
Back to all posts
Implementing Proxy Failover with Circuit Breakers in Python
proxy failover circuit python

Implementing Proxy Failover with Circuit Breakers in Python

August 26, 2026

Introduction

In the world of web scraping and automated data collection, a single proxy failure can bring your entire pipeline to a halt. Whether you are monitoring prices, collecting search results, or aggregating social media feeds, uninterrupted access is essential. A robust failover strategy ensures that when one proxy becomes unavailable, another takes over without noticeable delay. This article explains how to implement proxy failover using the circuit breaker pattern in Python, providing concrete code and real‑world examples.

Why Proxy Failover Matters

Proxies can fail for many reasons: IP bans, network partitions, rate limiting, or hardware issues. In each case, the symptom is usually an exception—connection timeout, HTTP 429, or a 5xx error. If your script does not handle these gracefully, it will stop processing and you may lose data or violate service level agreements.

A failover mechanism addresses this by:

  • Detecting failures quickly.
  • Switching to a healthy proxy automatically.
  • Recovering the original proxy once it becomes available again.

Understanding Circuit Breakers

The circuit breaker is a design pattern inspired by electrical circuit breakers. It wraps a remote call and tracks consecutive failures. When the failure count exceeds a configurable threshold, the breaker opens, preventing further calls to the faulty endpoint. After a cool‑down period, the breaker moves to a half‑open state, allowing a limited number of test requests. If those succeed, the breaker closes; otherwise it reopens.

Key Parameters

  • Failure threshold – number of consecutive errors before opening.
  • Timeout – duration the breaker stays open before attempting recovery.
  • Cool‑down period – time between opening and half‑open transition.

Implementing a Basic Circuit Breaker in Python

Below is a self‑contained implementation using Python’s standard library. It is intentionally simple; you can extend it with logging, metrics, or asynchronous support.

import time
import threading
from enum import Enum

class CircuitState(Enum):
    CLOSED = 'closed'
    OPEN = 'open'
    HALF_OPEN = 'half-open'

class CircuitBreaker:
    def __init__(self, failure_threshold=5, timeout=30):
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.failure_count = 0
        self.last_attempt_time = None
        self.state = CircuitState.CLOSED
        self.lock = threading.Lock()

    def call(self, func, *args, **kwargs):
        with self.lock:
            if self.state == CircuitState.OPEN:
                if time.time() - self.last_attempt_time > self.timeout:
                    self.state = CircuitState.HALF_OPEN
                else:
                    raise Exception('Circuit breaker is OPEN')
            try:
                result = func(*args, **kwargs)
                self._on_success()
                return result
            except Exception as e:
                self._on_failure()
                raise e

    def _on_success(self):
        self.failure_count = 0
        self.state = CircuitState.CLOSED

    def _on_failure(self):
        self.failure_count += 1
        self.last_attempt_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN

Explanation

  • The call method wraps any function.
  • If the breaker is open and the timeout has not elapsed, it raises immediately.
  • After a successful call, the breaker resets.
  • After a failure, the counter increments; reaching the threshold opens the breaker.

Integrating with a Proxy Pool

A proxy pool is a collection of proxy addresses. The circuit breaker can be combined with a round‑robin or random selection strategy.

import requests

class ProxyPool:
    def __init__(self, proxies):
        self.proxies = proxies
        self.index = 0
        self.lock = threading.Lock()

    def get_next(self):
        with self.lock:
            proxy = self.proxies[self.index]
            self.index = (self.index + 1) % len(self.proxies)
            return proxy

def fetch_url(url, pool, breaker):
    proxy = pool.get_next()
    proxies = {'http': proxy, 'https': proxy}
    try:
        response = requests.get(url, proxies=proxies, timeout=10)
        response.raise_for_status()
        return response.text
    except Exception:
        # Re‑raise to let the breaker handle it
        raise

Usage

pool = ProxyPool([
    'http://proxy1.example.com:8080',
    'http://proxy2.example.com:8080',
    'http://proxy3.example.com:8080'
])
breaker = CircuitBreaker(failure_threshold=3, timeout=60)

urls = ['https://example.com/page1', 'https://example.com/page2']
for url in urls:
    try:
        html = breaker.call(fetch_url, url, pool, breaker)
        # Process html
    except Exception as e:
        print(f'Failed to fetch {url}: {e}')

In this snippet, each request is routed through the next proxy. If a proxy fails three times in a row, the breaker opens for that proxy, and subsequent calls are rejected until the timeout expires.

Handling Different Failure Modes

Not all errors should trip the breaker equally. For instance:

  • Connection errors (DNS resolution failure, TCP reset) are often transient.
  • HTTP 429 indicates rate limiting; you may want to back off rather than failover immediately.
  • 5xx errors suggest server problems; a retry with exponential back‑off might succeed.
  • 401/407 point to authentication issues, which are unlikely to resolve by switching proxies.

You can customize the breaker by inspecting the exception type or response status code:

def should_trip(exception):
    if isinstance(exception, requests.exceptions.ConnectionError):
        return True
    if isinstance(exception, requests.exceptions.HTTPError):
        status = exception.response.status_code
        return status in (429, 500, 502, 503, 504)
    return False

Integrate this logic into the _on_failure method to decide whether to increment the failure count.

Advanced: Dynamic Thresholds

Static thresholds may be too rigid. A more sophisticated approach adjusts the threshold based on recent success rate. For example, if the last ten calls succeeded, increase the threshold; if many failures occur, lower it. This adaptive behavior reduces unnecessary proxy churn while still protecting against prolonged outages.

Monitoring and Logging

Visibility into breaker state is crucial for debugging. You can expose metrics using Prometheus or simply log state transitions:

import logging
logging.basicConfig(level=logging.INFO)

class LoggingCircuitBreaker(CircuitBreaker):
    def _on_success(self):
        super()._on_success()
        logging.info('Breaker closed')
    def _on_failure(self):
        super()._on_failure()
        if self.state == CircuitState.OPEN:
            logging.warning('Breaker opened')

Real‑World Example: E‑commerce Price Monitoring

Imagine you need to scrape product pages from an online store every minute. Using a single proxy risks being blocked after a few requests. By rotating through a pool of residential proxies and applying a circuit breaker, you can:

  1. Detect when a proxy returns repeated 403 or 429 responses.
  2. Open the circuit for that proxy.
  3. Redirect traffic to the next proxy in the pool.
  4. Recover after a cool‑down, allowing the original proxy to be retried.

This approach reduces downtime and improves data freshness.

Best Practices

  • Set sensible thresholds – too low causes unnecessary switching; too high delays failover.
  • Log state changes – helps debug flaky proxies.
  • Combine with retries – a retry with exponential back‑off can complement the breaker.
  • Monitor health – expose metrics (e.g., via Prometheus) for each proxy.
  • Test with mock failures – simulate errors to verify breaker behavior before deploying.

Conclusion

Implementing proxy failover with circuit breakers transforms fragile scraping scripts into resilient data pipelines. By wrapping each request in a stateful breaker and rotating through a pool, you gain automatic recovery from transient failures while maintaining high throughput. With the patterns and code provided, you can build a robust foundation for any large‑scale data collection effort.