需求

1. 在进行渗透测试过程中，有多个字典，内容有重叠部分

2. 现需将所有字典内容去重后的内容写入一个字典中

代码

#! /usr/bin/env python
# -*- coding: utf-8 -*-def remove_duplicates(input_file, output_file):# 读取输入文件内容并去重with open(input_file, 'r', encoding='utf-8') as file:lines = file.readlines()lines = list(set(lines))# 将去重后的内容写入输出文件with open(output_file, 'w', encoding='utf-8') as file:file.writelines(lines)if __name__ == "__main__":# 调用函数来处理文件#input_file_path = 'path/to/your/input_file.txt'#output_file_path = 'path/to/your/output_file.txt'input_file_path = 'payload_wordlists/list1.txt'output_file_path = 'payload_wordlists/list2.txt'remove_duplicates(input_file_path, output_file_path)