机器学习之静态爬虫（Machine Learning about Static Spider）

三、爬虫自动化——静态爬虫Static Spider1、静态爬虫1.1 爬虫的整体架构一个完整的爬虫需要完成如下五步发送请求requests..get(url, headers, timeout)获取响应获取页面响应码resp.status_code获取页面文本resp.text解析数据BeautifulSoup(resp.text, lxml)存储数据在CSV里面写入数据csv.writer()打开文件进行写入open(file, w)翻页爬取for e in range(len(f))1.2 静态爬虫示例主要包含三个步骤调制器、解析器、下载器Books to Scrape 图书信息爬虫分类页示例 import csv import random import time import requests from bs4 import BeautifulSoup class BookSpider: 图书爬虫下载 - 解析 - 保存 def __init__(self): # 复用 Session保持连接并统一携带请求头 self.session requests.Session() self.session.headers.update({ User-Agent: (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36) }) def download(self, url): 下载器 time.sleep(random.uniform(0.5, 1.5)) # 请求间隔降低被限流概率 try: resp self.session.get(url, timeout10) resp.raise_for_status() # 非 2xx 直接抛异常 return resp.content # 用 content 而非 text页面常不声明 charset # requests 默认按 ISO-8859-1 解码会导致中文/符号乱码 except requests.RequestException as e: print(f下载失败: {e}) return None def parse(self, html): 解析器 if not html: return [] soup BeautifulSoup(html, lxml) # 传入 bytessoup 自动检测页面编码 items [] for book in soup.select(article.product_pod): title_tag book.select_one(h3 a) price_tag book.select_one(.price_color) if title_tag and price_tag: # 任一字段缺失就跳过 items.append({ title: title_tag.get(title, ), price: price_tag.text.strip(), link: title_tag[href], }) return items def save(self, data, filename): 保存 if not data: return with open(filename, w, newline, encodingutf-8-sig) as f: writer csv.DictWriter(f, fieldnamesdata[0].keys()) writer.writeheader() writer.writerows(data) def crawl(self, base_url, pages, filename): 调制器 all_data [] for page in range(1, pages 1): # books.toscrape 分类页分页规则page-2.html / page-3.html if page 1: url base_url else: url base_url.replace(index.html, fpage-{page}.html) print(f正在爬取第 {page}/{pages} 页: {url}) html self.download(url) items self.parse(html) all_data.extend(items) print(f 获取到 {len(items)} 条数据) self.save(all_data, filename) print(f完成共爬取 {len(all_data)} 条数据) return all_data if __name__ __main__: # 爬虫对象 spider BookSpider() # 创建爬虫 spider.crawl( base_urlhttp://books.toscrape.com/catalogue/category/books_1/index.html, pages3, filenameproducts.csv, )1.3 分页爬取方法常见URL规律型对于URL中有常见的规律可以使用循环进行找到全部的URLbase_url https://example.com/list for page in range(1, 11): url f{base_url}?page{page} # 爬取该页 ...API接口型直接使用接口进行获取URL# POST请求获取JSON数据 api_url https://example.com/api/products for page in range(1, 11): resp requests.post(api_url, json{page: page, size: 20}) data resp.json() # 处理数据 ...1.4 数据存储方式CSV存储适合表格数据import csv data [ {name: 商品A, price: 99}, {name: 商品B, price: 199} ] with open(output.csv, w, newline, encodingutf-8-sig) as f: writer csv.DictWriter(f, fieldnames[name, price]) writer.writeheader() writer.writerows(data)newline参数在Windows下是必须的否则会导致行间距加倍1.5 爬虫必备注意事项1合法合规爬取网站允许的数据 robot.txtimport requests # 查看网站的robots.txt resp requests.get(https://www.zhihu.com/robots.txt) print(resp.text)2爬取的时候控制请求频率import time import random # 每次请求后随机等待0.5-2秒 delay random.uniform(0.5, 2.0) time.sleep(delay)3异常值处理try: resp requests.get(url, timeout5) resp.raise_for_status() # HTTP错误会抛出异常 except requests.Timeout: print(请求超时) except requests.HTTPError as e: print(fHTTP错误: {e}) except requests.RequestException as e: print(f请求失败: {e})4不爬取敏感信息个人隐私、商业机密数据、受版权保护的内容