1、使用BeautifulSoup的 'html5lib' 能像网页工具一样渲染内容。
缺点：运行比较慢
2、安装包
pip install html5lib
3、直接获取网页的所有有效内容
import requests #数据请求模块 第三方模块 pip install requests
from bs4 import BeautifulSoup
heads = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
}
def get_response(html_url):
response = requests.get(url=html_url, headers=heads)
return response
url= 'https://www.python100.com/html/1390.html'
response_new = get_response(html_url=url)
response_new.encoding = 'UTF-8'
#soup = BeautifulSoup(response_new.text, 'html.parser')
soup_new = BeautifulSoup(response_new.text, 'html5lib')
print(soup_new.text)
4、结果展示
