Splash与Scrapy结合
scrapy-splash 教程 — splash中文文档 0.1 文档
https://splash-cn-doc.readthedocs.io/zh_CN/latest/scrapy-splash-toturial.html
安装scrapy-splash库
pip install scrapy-splash==0.8.0配置splash服务（以下操作全部在settings.py）
使用splash解析，要在配置文件中设置splash服务器地址：
SPLASH_URL = 'http://localhost:8050/' 将splash middleware添加到DOWNLOADER_MIDDLEWARE中
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}Enable SplashDeduplicateArgsMiddleware
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100
}这个中间件需要支持cache_args功能; 它允许通过不在磁盘请求队列中多次存储重复的Splash参数来节省磁盘空间。如果使用Splash 2.1+，则中间件也可以通过不将这些重复的参数多次发送到Splash服务器来节省网络流量
配置消息队列所使用的过滤类
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'配置消息队列需要使用的类
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'样例
方法1
import scrapy
from scrapy_splash import SplashRequest
class TxxxxxSpider(scrapy.Spider):
name = 'tuniu'
allowed_domains = ['tuniu.com']
def start_requests(self):
yield SplashRequest('http://www.xxxxx.com/', args={'wait': 0.5})
def parse(self, response):
print(response.text)方法2
class TuniuSpider(scrapy.Spider):
name = 'xxxxx2'
allowed_domains = ['xxxxxx.com']
def start_requests(self):
url = 'http://www.xxxxx.com/'
lua='''
function main(splash,args)
splash:go(args.url)
splash:wait(1.5)
return splash:html()
end
'''
yield SplashRequest(url,callback=self.parse,args={'lua_source':lua},endpoint='execute')
def parse(self, response):
with open('a2.html','wb') as f:
f.write(response.body)