[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog:post:en:efficient-proxy-rotation-go-high-throughput-api-scraping":3},{"slug":4,"lang":5,"title":6,"summary":7,"date":8,"tags":9,"tag_slugs":15,"thumbnail_url":16,"translations":17,"body":18,"asset_base":19},"efficient-proxy-rotation-go-high-throughput-api-scraping","en","Efficient Proxy Rotation in Go for High‑Throughput API Scraping","Learn how to build a resilient proxy‑rotating client in Go, complete with concurrency, error handling, and persistence for API scraping at scale.","2026-07-24",[10,11,12,13,14],"go","proxy","scraping","automation","api",[10,11,12,13,14],"https://blog-api.ro-proxy.com/api/blog/posts/efficient-proxy-rotation-go-high-throughput-api-scraping/thumbnail.svg?lang=en",[5],"## Introduction\nWhen you’re pulling data from public or private APIs at scale, the bottleneck is rarely the API itself – it’s the IP limits and rate‑limiting mechanisms that throttle or block repeated requests. A thoughtful proxy‑rotation strategy lets you keep a steady flow of requests while staying under the radar of those limits.\n\nIn this post we walk through:\n\n1. Why Go is a great fit for concurrent proxy rotation.\n2. Designing a simple, yet robust, rotation engine.\n3. Integrating error handling, back‑off, and persistence.\n4. Real‑world patterns for >(10k req/min) performance.\n5. A quick demo using RoProxy’s public API.\n\nBy the end you’ll have a reusable Go module you can drop into any scraping or monitoring project.\n\n## Why Go?\n\n* **Lightweight goroutines** – thousands of lightweight threads (goroutines) can be spawned with minimal overhead.\n* **Excellent standard library** – `net/http` supports custom transports, timeouts, and connection pooling out of the box.\n* **Static binaries** – deployable as a single compiled artifact.\n* **Strong typing & concurrency primitives** – channels, mutexes, and context make thread‑safe rotation trivial.\n\nThese traits combine to give you predictable latency and memory usage, which is essential when you’re trying to hit 10k+ requests per minute.\n\n## Core Concepts\n\n| Concept | What it solves | How it’s implemented in Go |\n|---------|----------------|-----------------------------|\n| **Proxy pool** | A pool of valid IP:port pairs | A slice of structs with metadata, protected by a `sync.RWMutex` |\n| **Round‑Robin / Weighted selection** | Even distribution or priority handling | Simple index counter with modulo arithmetic, or a weighted slice |\n| **Failure tracking** | Avoid repeatedly using a bad proxy | Map of proxy to failure count, reset after success |\n| **Back‑off** | Reduce load on a failing API | `time.Sleep` with exponential back‑off, capped at a max |\n| **Persistence** | Resume from last state on crash | File or KV store (e.g., BoltDB or Redis) that serialises the pool |\n| **Health‑check** | Keep only live proxies | Periodic `HEAD` requests to a lightweight endpoint |\n\n## Building the Rotation Engine\n\nBelow is a minimal yet extensible implementation. We’ll build a `ProxyRotator` type that exposes a `Do(req)` method. It hides all the rotation logic and lets the caller focus on request creation.\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"crypto/tls\"\n    \"fmt\"\n    \"io\"\n    \"net\"\n    \"net/http\"\n    \"sync\"\n    \"time\"\n)\n\n// Proxy represents a single HTTP proxy.\ntype Proxy struct {\n    Address string // e.g., \"http://1.2.3.4:8080\"\n    // You can add fields like Weight, LastUsed, Failures, etc.\n}\n\n// ProxyRotator manages a rotating pool.\ntype ProxyRotator struct {\n    pool       []Proxy\n    mu         sync.RWMutex\n    idx        int\n    httpClient *http.Client\n}\n\n// NewProxyRotator builds a client with a proxy pool.\nfunc NewProxyRotator(proxies []Proxy) *ProxyRotator {\n    tr := &http.Transport{}\n    // Disable HTTP2 for simplicity; keep TCP keep‑alive\n    tr.DisableCompression = true\n    tr.DialContext = (&net.Dialer{Timeout: 10 * time.Second}).DialContext\n\n    // Create a single client; transport will be swapped per request\n    client := &http.Client{Transport: tr, Timeout: 30 * time.Second}\n\n    return &ProxyRotator{pool: proxies, httpClient: client}\n}\n\n// rotate selects the next proxy in a thread‑safe way.\nfunc (pr *ProxyRotator) rotate() Proxy {\n    pr.mu.Lock()\n    defer pr.mu.Unlock()\n    proxy := pr.pool[pr.idx%len(pr.pool)]\n    pr.idx++\n    return proxy\n}\n\n// Do sends a request via a rotated proxy.\nfunc (pr *ProxyRotator) Do(req *http.Request) (*http.Response, error) {\n    proxy := pr.rotate()\n    // Configure transport for this request\n    transport := &http.Transport{\n        Proxy: http.ProxyURL(proxyURL(proxy.Address)),\n        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},\n    }\n    pr.httpClient.Transport = transport\n    return pr.httpClient.Do(req)\n}\n\nfunc proxyURL(addr string) *url.URL {\n    u, _ := url.Parse(addr)\n    return u\n}\n```\n\n> **Tip:** The `Transport` is swapped per request to avoid state leaking between goroutines. In production you might pool transports or keep one per proxy.\n\n## Adding Resilience\n\nThe simple engine above will fail fast if a proxy is down. Below are a few tweaks.\n\n### Failure Tracking & Back‑off\n\n```go\n// failureMap tracks consecutive failures.\nvar failureMap = struct{ mu sync.Mutex; m map[string]int }{m: make(map[string]int)}\n\nfunc recordFailure(addr string) {\n    failureMap.mu.Lock()\n    defer failureMap.mu.Unlock()\n    failureMap.m[addr]++\n}\n\nfunc recordSuccess(addr string) {\n    failureMap.mu.Lock()\n    defer failureMap.mu.Unlock()\n    delete(failureMap.m, addr)\n}\n```\n\nIn the `Do` method, wrap the call with retry logic:\n\n```go\nmaxRetries := 3\nvar lastErr error\nfor i := 0; i \u003C maxRetries; i++ {\n    resp, err := pr.httpClient.Do(req)\n    if err == nil {\n        recordSuccess(proxy.Address)\n        return resp, nil\n    }\n    lastErr = err\n    recordFailure(proxy.Address)\n    // Exponential back‑off\n    time.Sleep(time.Duration(1\u003C\u003Ci) * time.Second)\n}\nreturn nil, fmt.Errorf(\"all retries failed: %w\", lastErr)\n```\n\n### Health‑Check Routine\n\n```go\nfunc (pr *ProxyRotator) StartHealthCheck(ctx context.Context) {\n    go func() {\n        ticker := time.NewTicker(5 * time.Minute)\n        defer ticker.Stop()\n        for {\n            select {\n            case \u003C-ctx.Done():\n                return\n            case \u003C-ticker.C:\n                pr.checkAll()\n            }\n        }\n    }()\n}\n\nfunc (pr *ProxyRotator) checkAll() {\n    for i, p := range pr.pool {\n        if !pr.isAlive(p) {\n            pr.mu.Lock()\n            pr.pool = append(pr.pool[:i], pr.pool[i+1:]...)\n            pr.mu.Unlock()\n        }\n    }\n}\n\nfunc (pr *ProxyRotator) isAlive(p Proxy) bool {\n    testReq, _ := http.NewRequest(\"GET\", \"https://httpbin.org/ip\", nil)\n    transport := &http.Transport{Proxy: http.ProxyURL(proxyURL(p.Address))}\n    client := &http.Client{Transport: transport, Timeout: 5 * time.Second}\n    resp, err := client.Do(testReq)\n    if err != nil { return false }\n    resp.Body.Close()\n    return resp.StatusCode == 200\n}\n```\n\n## Scaling to 10k+ Requests/Minute\n\n1. ** Goroutine Pool** – Use a worker pool to limit concurrent requests and avoid exhausting system resources.\n\n   ```go\n   workers := 200\n   jobs := make(chan *http.Request, workers*2)\n   var wg sync.WaitGroup\n   for i := 0; i \u003C workers; i++ {\n       wg.Add(1)\n       go func() {\n           defer wg.Done()\n           for req := range jobs {\n               resp, err := rot.Do(req)\n               // handle resp / err\n           }\n       }()\n   }\n   // enqueue jobs\n   for _, url := range urls {\n       jobs \u003C- &http.Request{Method: \"GET\", URL: url}\n   }\n   close(jobs)\n   wg.Wait()\n   ```\n\n2. ** Connection Reuse** – Reuse TLS sessions by sharing the same `http.Transport` for a given proxy. The trick is to store a map[proxy]Transport.\n\n3. ** Don’t Forget DNS** – Use `net.Resolver` with `PreferGo: true` to avoid hitting system DNS cache, which can become stale under high churn.\n\n4. ** Monitorvaard** – Export metrics (requests per proxy, failure rate, latency) to Prometheus. A simple counter per proxy gives you insights into uneven wear.\n\n## Using RoProxy in the Code\n\nRoProxy offers a vetted pool of residential and datacenter proxies with a public API for health‑checks and rotation. Here’s how to pull a fresh pool and feed it into our `ProxyRotator`:\n\n```go\nfunc fetchRoProxyPool() ([]Proxy, error) {\n    // Example endpoint; replace with real RoProxy URL\n    resp, err := http.Get(\"https://apiFlows.roProxy.com/proxies?limit=100\")\n    if err != nil { return nil, err }\n    defer resp.Body.Close()\n    var raw []struct{ IP string; Port string }\n    if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { return nil, err }\n    proxies := make([]Proxy, len(raw))\n    for i, p := range raw {\n        proxies[i] = Proxy{Address: fmt.Sprintf(\"http://%s:%s\", p.IP, p.Port)}\n    }\n    return proxies, nil\n}\n```\n\nThen initialize:\n\n```go\nproxies, _ := fetchRoProxyPool()\nrot := NewProxyRotator(proxies)\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\nrot.StartHealthCheck(ctx)\n```\n\nFrom here the rest of the code is the same. RoProxy’s health‑check endpoint ensures that you receive only live proxies, so your failure map stays small.\n\n## Common Pitfalls\n\n| Pitfall | Fix |\n|---------|-----|\n| **Over‑rotating** – rotating too fast causes the same IP to be reused before the API resets its counters. | Use a time‑based throttler: `time.AfterFunc(time.Minute/requestsPerMinute, ...)` |\n| **Ignoring TLS errors** – many proxies use self‑signed certs. | Either trust the cert authority or skip verification (`InsecureSkipVerify:true`). |\n| **Blocking on a slow proxy** – a single bad proxy can stall a goroutine. | Set per‑request timeouts and use a retry counter, as shown above. |\n| **Memory leak** – unbounded `http.Client` pool. | Reuse transports per proxy or limit the number of concurrent clients. |\n\n## Wrap‑Up\n\nEfficient proxy rotation in Go boils down to a small set of patterns:\n\n1. **Thread‑safe pool** – indexed round‑robin with a mutex.\n2. **Per‑request transport** – swap the proxy per request.\n3. **ви Failure handling** – retry with back‑off, track failures, and prune bad proxies.\n4. **Health checks** – keep the pool lean and healthy.\n5. **Worker pool + metrics** – scale without blowing the system.\n\nWhen you combine these with a reliable provider like RoProxy, you can push past API rate limits, stay under the radar of anti‑scraping measures, and keep your data pipeline humming. Happy scraping!\n","https://blog-api.ro-proxy.com/api/blog/posts/efficient-proxy-rotation-go-high-throughput-api-scraping/assets"]