How to Rotate Proxies in Python: Requests, Selenium and Scrapy (2026)

What does it mean to rotate proxies?

Proxy rotation is changing the IP address behind your requests as you go: per request, per batch, or per browser session.

Most residential plans give you a gateway; most static plans give you a list. Check which one you bought before writing code.

The samples below use https://api.ipify.org as the target because it returns only the caller’s IP, so you can see each rotation. Replace the words in capitals with your own proxy details.

How to rotate proxies in Python with requests

1. Install requestsRun pip install requests. You should see Successfully installed, or Requirement already satisfied.
2. Paste your proxy listOne full URL per proxy: scheme, login, host and port.
3. Run the loopEach request takes the next proxy. With 3 proxies, the 6 lines of output repeat in a pattern of 3 IPs.

import itertools
import requests

PROXIES = [
    "http://USERNAME:PASSWORD@HOST1:PORT",
    "http://USERNAME:PASSWORD@HOST2:PORT",
    "http://USERNAME:PASSWORD@HOST3:PORT",
]
pool = itertools.cycle(PROXIES)

for _ in range(6):
    proxy = next(pool)
    r = requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy}, timeout=10)
    print(r.status_code, r.text)

itertools.cycle walks the list in order and starts again at the end. Use random.choice(PROXIES) instead when you don’t want a fixed order; it can pick the same proxy twice in a row.

What should your code do when a proxy fails?

Move to the next proxy and try again, up to a limit. A failed proxy raises an exception; a refused login can come back as a 407 status instead. The function below handles both.

import itertools
import requests

PROXIES = [
    "http://USERNAME:PASSWORD@HOST1:PORT",
    "http://USERNAME:PASSWORD@HOST2:PORT",
    "http://USERNAME:PASSWORD@HOST3:PORT",
]
pool = itertools.cycle(PROXIES)


def fetch(url, tries=3):
    for _ in range(tries):
        proxy = next(pool)
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=10)
            if r.status_code == 200:
                return r
            print("status", r.status_code, "from", proxy.split("@")[1])
        except requests.RequestException as e:
            print("failed", proxy.split("@")[1], type(e).__name__)
    raise RuntimeError(f"{tries} proxies in a row failed for {url}")


print(fetch("https://api.ipify.org").text)

When it works, you see one IP. When the first proxy is down, you see a failed line naming it, then the IP from the next one. If 3 proxies in a row fail, the error stops the script instead of looping forever.

Retrying on any status other than 200 is a starting point. A 404 means the page doesn’t exist, so a new IP won’t help; a 403 or 429 usually means the IP was blocked or rate-limited, which is what rotation fixes.

How do you rotate proxies in Scrapy?

Scrapy’s built-in HttpProxyMiddleware reads a proxy from each request’s meta, so rotation is one line per request. Its docs list both http://some_proxy_server:port and the form with username:password@ before the host.

import random
import scrapy

PROXIES = [
    "http://USERNAME:PASSWORD@HOST1:PORT",
    "http://USERNAME:PASSWORD@HOST2:PORT",
    "http://USERNAME:PASSWORD@HOST3:PORT",
]


class IpSpider(scrapy.Spider):
    name = "ip"

    async def start(self):
        for _ in range(6):
            proxy = random.choice(PROXIES)
            yield scrapy.Request("https://api.ipify.org", meta={"proxy": proxy}, dont_filter=True)

    def parse(self, response):
        yield {"ip": response.text}

Save it as ip_spider.py and run scrapy runspider ip_spider.py -o ips.jsonl. When it works, ips.jsonl holds 6 lines like {“ip”: “…”} with the IPs of your proxies. The async def start() method needs Scrapy 2.13 or later.

A proxy set in meta wins over the http_proxy and https_proxy environment variables, according to the same docs. For a full project, move the random.choice line into a downloader middleware so every spider gets it.

How do you rotate proxies in Selenium?

A browser keeps one proxy for its whole life. Rotating means closing the driver and starting a new one with the next proxy.

from selenium import webdriver

PROXIES = ["HOST1:PORT", "HOST2:PORT", "HOST3:PORT"]

for proxy in PROXIES:
    options = webdriver.ChromeOptions()
    options.add_argument(f"--proxy-server=http://{proxy}")
    driver = webdriver.Chrome(options=options)
    driver.get("https://api.ipify.org")
    print(proxy, driver.find_element("tag name", "body").text)
    driver.quit()

When it works, each line shows a proxy and the IP that ipify saw for it. Selenium 4.6 and later fetch the matching driver for your Chrome, so pip install selenium is the only setup.

Starting a browser per proxy is slow next to an HTTP request. If you only need page HTML, the requests loop above does the same job faster. For a closer look at the two browser tools, see our Playwright vs Selenium comparison.

When should the provider do the rotating?

When you bought residential or mobile traffic. Those plans hand you one address, and the provider swaps the IP behind it.

Oxylabs is a documented example. Its session control page says its residential proxies rotate with every request by default through pr.oxylabs.io:7777. Adding sessid and a random string to the username holds one IP; a session lasts 10 minutes by default, or ends after 60 seconds without requests, and sesstime extends it up to 1,440 minutes.

import requests

ROTATING = "http://customer-USERNAME:PASSWORD@pr.oxylabs.io:7777"
STICKY = "http://customer-USERNAME-sessid-abcd1234-sesstime-30:PASSWORD@pr.oxylabs.io:7777"

for proxy in (ROTATING, ROTATING, STICKY, STICKY):
    r = requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy}, timeout=15)
    print(r.text)

When it works, the first two lines show different IPs and the last two show the same one. Other providers use their own username format for sessions, so copy theirs from the dashboard rather than this one.

Prices for these plans are per GB. Our best residential proxies ranking compares them, and our proxy speed claims article explains what the advertised success rates measure.

What is the most common proxy rotation mistake?

Passing the proxy under the http key only. requests picks the proxy by the scheme of the URL you fetch, so an https:// page with no https key goes out from your own IP. No error appears; the scraper just stops rotating.

import requests

proxy = "http://USERNAME:PASSWORD@HOST:PORT"
wrong = {"http": proxy}                  # https:// URLs skip the proxy
right = {"http": proxy, "https": proxy}  # both schemes use it

print(requests.get("https://api.ipify.org", proxies=wrong, timeout=10).text)
print(requests.get("https://api.ipify.org", proxies=right, timeout=10).text)

The first line prints your real IP, the second the proxy’s. Always fill both keys, as every sample in this guide does.

Which rotation method should you start with?

Free proxies work for trying the code: our free proxy list needs no login. Check which ones are alive with the free proxy checker at GoToProxy.

Most readers here run a requests script against one site. For that case, buy a rotating residential plan, put the gateway URL under both the http and https keys, and wrap calls in the retry function above. The provider handles IP changes, so your code stays at about 20 lines, and the retry covers the requests that still fail. Compare plans in all proxy rankings and independent proxy ratings on ProxyRates before you pick one.

FAQ