使用Django开发爬虫系统

在本文中，我们将介绍如何使用Django开发一个简单但功能强大的爬虫系统。我们将使用Python编写爬虫，并将爬取到的数据存储到Django模型中，然后通过Django的管理页面管理这些数据。

1. 介绍

爬虫系统用于从互联网上收集信息，常用于数据分析、搜索引擎等领域。我们选择使用Django作为开发框架，因为它提供了强大的数据库管理和Web开发功能，能够帮助我们快速搭建一个稳健的爬虫系统。

2. 环境设置

首先确保您已经安装了Python，并使用pip安装了Django和其他必要的库：

pip install django requests beautifulsoup4

3. 创建Django项目

使用Django命令行工具创建一个新的项目：

django-admin startproject mycrawler
cd mycrawler

4. 定义模型

在models.py文件中定义一个模型来存储爬取到的数据：

from django.db import modelsclass Article(models.Model):title = models.CharField(max_length=100)content = models.TextField()created_at = models.DateTimeField(auto_now_add=True)def __str__(self):return self.title

运行数据库迁移命令以创建数据库表：

python manage.py makemigrations
python manage.py migrate

5.具体代码

from django.shortcuts import render
from django.http import HttpResponse
from django.http import JsonResponse
import json
import bag
import re
from bs4 import BeautifulSoup
from tqdm import tqdm
import time
import randomdef index(request):return render(request, 'index.html')def get_url(city_name, item):session = bag.session.create_session()for cookie in bag.Bag.read_json(r'./static/cookies.json'):session.cookies.set(cookie['name'], cookie['value'])session.headers['Connection'] = 'close'js_data = bag.Bag.read_json('./static/city.json')judge = js_data.get(city_name)if judge:city = judge.split('/')[-1]js_data1 = bag.Bag.read_json('./static/menu.json')judge1 = js_data1.get(item)if judge1:return judge1.format(city), sessiondef get_shops(request):if request.method == 'POST':try:data = json.loads(request.body)if not bool(data.get('city')) and not bool(data.get('shopLink')):return JsonResponse({'message': '请选择城市'}, status=200)elif not bool(data.get('city')) and bool(data.get('shopLink')):  # 处理传入具体链接后续开发passelse:url, session = get_url(data.get('city'), data.get('selectedType'))def get_types():  # 正常传参pattern = re.compile(r'<a.*?href="(.*?)".*?<span>(.*?)</span></a>', re.S)if bool(url):resp = session.get(url)html = BeautifulSoup(resp.text, 'html.parser')soup = html.findAll('div', id='classfy')links = re.findall(pattern, str(soup))return linkselse:return Nonedef get_shop():links = get_types()pattern = re.compile(r'<div class="tit">.*?<a.*?data-shopid="(.*?)".*?href="(.*?)".*?title="(.*?)"'r'(?:.*?<div class="star_icon">.*?<span class="(.*?)"></span>.*?<b>(.*?)</b>)?'r'(?:.*?<b>(.*?)</b>)?'r'(?:.*?<div class="tag-addr">.*?<span class="tag">(.*?)</span>.*?<em class="sep">.*?<span class="tag">(.*?)</span>)?',re.S)number = re.compile(r'data-ga-page="(.*?)"', re.S)result = []print(len(links))for link in links[11:12]:  # 获取第一页resp = session.get(link[0])time.sleep(random.randint(1, 10))page = [int(i) for i in re.findall(number, resp.text)]page_num = sorted(page, reverse=True)[0]html = BeautifulSoup(resp.text, 'html.parser')soup = html.findAll('li', class_='')for i in soup:for j in re.findall(pattern, str(i)):result.append(j)if page_num >= 2:  # 获取第一页往后for count in tqdm(range(page_num)):try:resp1 = session.get(link[0] + 'p{}'.format(count + 1))time.sleep(random.randint(1, 5))# time.sleep(0.5)html1 = BeautifulSoup(resp1.text, 'html.parser')soup1 = html1.findAll('li', class_='')for k in soup1:info = pattern.search(str(k))if info:groups = list(info.groups())for i in range(len(groups)):if not groups[i]:groups[i] = 'null'result.append(tuple(groups))except Exception as e:print(e)continueelse:passreturn resultend = get_shop()bag.Bag.save_excel(end, f'./{data.get("city")}_{data.get("selectedType")}.xlsx')return JsonResponse({'message': 'Data received successfully'}, status=200)except json.JSONDecodeError as e:return JsonResponse({'error': 'Invalid JSON format'}, status=400)else:return JsonResponse({'error': 'Only POST requests are allowed'}, status=405)

index.html

<!DOCTYPE html>
<html>
<head><meta charset="utf-8" /><title>首页</title><link rel="stylesheet" href="../static/css/page.css" /><script type="text/javascript" src="../static/js/jquery.min.js" ></script><script type="text/javascript" src="../static/js/index.js" ></script>
</head>
<body><div class="left"><div class="bigTitle">爬虫管理系统</div><div class="lines"><div onclick="pageClick(this)" data-html="/text.html" data-title="文本爬虫"><img src="../static/img/icon-1.png" />文本爬虫</div><div onclick="pageClick(this)" data-html="/image.html" data-title="图片爬虫"><img src="../static/img/icon-2.png" />图片爬虫</div><div onclick="pageClick(this)" data-html="/music.html" data-title="音乐爬虫"><img src="../static/img/icon-3.png" />音乐爬虫</div><div onclick="pageClick(this)" data-html="/video.html" data-title="视频爬虫"><img src="../static/img/icon-4.png" />视频爬虫</div></div></div><div class="top"><div class="leftTiyle" id="flTitle">基于python实现的网络爬虫</div><div class="thisUser">当前用户：互联网民工</div></div><div class="content" id="contentDiv"></div><script>function pageClick(element) {var clickedElement = $(element);$('#contentDiv').empty();var htmlPath = clickedElement.attr('data-html');var pageTitle = clickedElement.attr('data-title');$.ajax({url: htmlPath,dataType: 'html',success: function (data) {$('#contentDiv').html(data);$('#flTitle').text(pageTitle); // 设置标题},error: function (xhr, status, error) {// 处理加载失败的情况}});}$(document).ready(function() {var defaultContent = "欢迎使用爬虫管理系统！请点击左侧功能以查看详细内容。"$('#contentDiv').html(defaultContent);});</script>
</body>
</html>