526互联

Scrapy 中 Downloader 设置代理

发布时间 2023-06-25 19:38:53作者: jiang_jiayun
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware
class MyProxyMiddleware:
    def process_request(self, request, spider):
        # request.meta['proxy'] ='http://ip:port'
        # request.meta['proxy'] ='http://name:pwd@ip:port'
        request.meta['proxy'] ='http://139.224.211.212:8080'

Scrapy 中 Downloader 设置代理

DOWNLOADER_MIDDLEWARES = {
  'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware':None,
  "scrapy06.middlewares.MyProxyMiddleware": 543,
}