PDF解析示例代码学习

以下是结合多种技术实现的PDF解析详细示例（Python实现），涵盖文本、表格和扫描件处理场景：

一、环境准备与依赖安装

# 核心依赖库 pip install pdfplumber tabula-py pytesseract opencv-python mysql-connector-python

二、完整解析流程示例

import pdfplumber import tabula import pytesseract import cv2 import re import mysql.connector from mysql.connector import Error from PIL import Image import hashlib # ========== 1. 通用PDF解析 ========== ‌:ml-citation{ref="6,7" data="citationList"} def parse_pdf(pdf_path): result = {"text": [], "tables": [], "images": []} # 文本提取（含坐标信息） with pdfplumber.open(pdf_path) as pdf: for page in pdf.pages: # 基础文本提取 text = page.extract_text(x_tolerance=1, y_tolerance=1) if text: result["text"].append({ "page": page.page_number, "content": text_clean(text), "bbox": page.bbox }) # 表格识别与提取 ‌:ml-citation{ref="3" data="citationList"} tables = page.find_tables() if tables: result["tables"].extend([ {"table_data": table.extract(), "position": table.bbox} for table in tables ]) # 扫描件处理模块 ‌:ml-citation{ref="6" data="citationList"} if not result["text"]: result.update(process_scanned_pdf(pdf_path)) return result # ========== 2. 扫描件OCR处理 ========== ‌:ml-citation{ref="6" data="citationList"} def process_scanned_pdf(pdf_path, dpi=300): ocr_results = [] images = convert_pdf_to_images(pdf_path, dpi) # 使用pdf2image库转换 for img in images: processed_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) text = pytesseract.image_to_string(processed_img, lang='chi_sim+eng') ocr_results.append(text_clean(text)) return {"ocr_text": ocr_results} # ========== 3. 数据标准化处理 ========== ‌:ml-citation{ref="7" data="citationList"} def text_clean(raw_text): # 多级清洗流程 text = re.sub(r'\s{2,}', ' ', raw_text) # 压缩空白字符 text = re.sub(r'[\x00-\x1F\x7F-\x9F]', '', text) # 删除控制字符 return text.strip() # ========== 4. 结构化字段提取 ========== ‌:ml-citation{ref="4,7" data="citationList"} def extract_structured_data(texts): patterns = { "date": r'\d{4}年\d{1,2}月\d{1,2}日', "amount": r'金额：(\d+\.\d{2})元', "parties": r'甲方：(.*?)\n乙方：(.*?)\n' } structured_data = {} for pattern_name, regex in patterns.items(): matches = [] for text in texts: matches.extend(re.findall(regex, text)) structured_data[pattern_name] = matches return structured_data # ========== 5. 数据库存储 ========== ‌:ml-citation{ref="8" data="citationList"} def save_to_mysql(data, pdf_hash): try: conn = mysql.connector.connect( host='localhost', database='pdf_archive', user='root', password='' ) cursor = conn.cursor() insert_query = """ INSERT INTO documents (file_hash, raw_text, structured_data) VALUES (%s, %s, %s) """ cursor.execute(insert_query, ( pdf_hash, "\n".join(data['text']), json.dumps(data['structured']) )) conn.commit() except Error as e: print(f"数据库错误: {e}") finally: if conn.is_connected(): cursor.close() conn.close() # ========== 6. 主执行流程 ========== if __name__ == "__main__": pdf_file = "sample_contract.pdf" # 生成文件指纹 ‌:ml-citation{ref="6" data="citationList"} with open(pdf_file, "rb") as f: file_hash = hashlib.sha256(f.read()).hexdigest() # 执行解析 parsed_data = parse_pdf(pdf_file) structured = extract_structured_data(parsed_data['text']) parsed_data['structured'] = structured # 数据持久化 save_to_mysql(parsed_data, file_hash)

三、关键处理策略说明

‌混合解析机制‌
- 优先尝试直接文本提取（可编辑PDF）‌6
- 自动降级到OCR处理（扫描件）‌6
- 保留原始坐标信息用于后期验证‌4
‌表格识别增强‌
使用PDF页面的物理布局检测表格边界（page.find_tables()），配合tabula进行精确提取‌3
‌多语言OCR支持‌
通过lang='chi_sim+eng'参数实现中英文混合识别，需提前安装Tesseract中文训练包‌6
‌数据验证机制‌
- 通过file_hash字段实现重复文件过滤‌6
- 同时保存原始文本和结构化数据用于数据追溯‌8

四、数据库表结构设计

CREATE TABLE `documents` ( `id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, `file_hash` CHAR(64) NOT NULL UNIQUE, `raw_text` LONGTEXT, `structured_data` JSON, `ocr_flag` TINYINT(1) DEFAULT 0, `create_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

五、异常处理建议

‌编码兼容性‌
添加数据库连接参数charset='utf8mb4'支持生僻字存储‌8
‌大文件分块处理‌
使用生成器逐页处理超过50页的PDF文档：

def batch_process(pdf_path, batch_size=10): with pdfplumber.open(pdf_path) as pdf: total_pages = len(pdf.pages) for i in range(0, total_pages, batch_size): yield pdf.pages[i:i+batch_size]
‌容错机制‌
在解析循环中添加异常捕获：

try: text = page.extract_text() except PDFSyntaxError as e: logging.warning(f"Page {page_num} parse failed: {str(e)}") continue